Remotion Video Composition

# Remotion Video Composition Create React-based video compositions rendered frame-by-frame with Remotion. Full documentation available via the `remotion-documentation` MCP server. ## When to Use - Creating programmatic videos with React components - Building motion graphics, explainer videos, or animated content - Rendering videos from data (captions, segments, overlays) - Building vertical (9:16) short-form videos with storyboard-driven segments - Creating animated data graphics (tables, process steps, elimination lists) - Capturing screenshot B-roll from web pages for video overlays - Working with the video-editor pipeline (see `references/video-editor-integration.md`) ## Core Concepts ### Project Structure The Root file (`src/Root.tsx`) defines compositions — each is a renderable video: ```tsx import {Composition} from 'remotion'; import {MyComp} from './MyComp'; export const Root: React.FC = () => { return ( <Composition id="MyComp" component={MyComp} durationInFrames={120} width={1920} height={1080} fps={30} defaultProps={{}} /> ); }; ``` Defaults: 30 fps, 1920x1080, "MyComp" ID. Use `calculateMetadata` for dynamic duration/dimensions. Define props with Zod schemas for visual UI controls. ### Frame-Based Animation Access the current frame (starts at 0) and video config: ```tsx import {useCurrentFrame, useVideoConfig, interpolate, spring} from 'remotion'; export const MyComp: React.FC = () => { const frame = useCurrentFrame(); const {fps, durationInFrames, height, width} = useVideoConfig(); // Linear interpolation const opacity = interpolate(frame, [0, 30], [0, 1], { extrapolateLeft: 'clamp', extrapolateRight: 'clamp', }); // Spring animation (preferred for natural motion) const scale = spring({fps, frame, config: {damping: 200}}); return ( <div style={{opacity, transform: `scale(${scale})`}}> Hello World </div> ); }; ``` ### Critical Rules - **Frame-based only** — all animations driven by `useCurrentFrame()`, never CSS animations/transitions - **Deterministic** — no `Math.random()`, use `random('seed')` from remotion - **No interactivity** — no `onClick`, `onHover`, `useState` for UI state, no event handlers - **No side effects** — avoid `useEffect`, keep components pure - **Use Remotion components** — `<Img>` not `<img>`, `<OffthreadVideo>` not `<video>`, `<Audio>` not `<audio>` ## Component Reference ### Media Components ```tsx import {OffthreadVideo, Img, Audio, staticFile} from 'remotion'; import {Gif} from '@remotion/gif'; // Video — use OffthreadVideo (NOT HTML video) <OffthreadVideo src={staticFile('video.mp4')} startFrom={30} endAt={120} volume={0.8} /> // Image — use Img (NOT HTML img, NOT Next.js Image) <Img src={staticFile('photo.png')} style={{width: '100%'}} /> // Audio <Audio src={staticFile('music.mp3')} startFrom={0} endAt={150} volume={0.5} /> // GIF (requires @remotion/gif) <Gif src={staticFile('animation.gif')} style={{width: '100%'}} /> ``` Properties: `startFrom` (trim frames from start), `endAt` (limit duration), `volume` (0-1). Use `staticFile()` for `public/` folder assets. ### Layout & Timing ```tsx import {AbsoluteFill, Sequence, Series} from 'remotion'; // Layering — later children render in front <AbsoluteFill> <AbsoluteFill style={{background: 'blue'}} /> {/* back */} <AbsoluteFill /> {/* front */} </AbsoluteFill> // Delayed appearance — child's useCurrentFrame() resets to 0 <Sequence from={30} durationInFrames={60}> <MyComponent /> {/* appears at frame 30, sees frame 0 internally */} </Sequence> // Sequential playback <Series> <Series.Sequence durationInFrames={30}><SceneA /></Series.Sequence> <Series.Sequence durationInFrames={45}><SceneB /></Series.Sequence> <Series.Sequence durationInFrames={30} offset={-10}><SceneC /></Series.Sequence> </Series> ``` ### Transitions ```tsx import {TransitionSeries, springTiming, linearTiming} from '@remotion/transitions'; import {fade} from '@remotion/transitions/fade'; import {wipe} from '@remotion/transitions/wipe'; <TransitionSeries> <TransitionSeries.Sequence durationInFrames={60}><SceneA /></TransitionSeries.Sequence> <TransitionSeries.Transition timing={springTiming({config: {damping: 200}})} presentation={fade()} /> <TransitionSeries.Sequence durationInFrames={60}><SceneB /></TransitionSeries.Sequence> <TransitionSeries.Transition timing={linearTiming({durationInFrames: 30})} presentation={wipe()} /> <TransitionSeries.Sequence durationInFrames={60}><SceneC /></TransitionSeries.Sequence> </TransitionSeries> ``` ## Quick-Start Workflow 1. **Scaffold**: `npx create-video@latest` (select Blank + TailwindCSS + Skills) 2. **Compose**: Build components in `src/`, define in `Root.tsx` 3. **Preview**: `npm run dev` (opens Remotion Studio) 4. **Render**: `npx remotion render MyComp output.mp4` ## Production Video Workflow For storyboard-driven short-form video (9:16 vertical): 1. **Transcribe** — Word-level timestamps via Whisper (medium.en model) 2. **Storyboard** — Build segment JSON: assign display modes (`split`/`full_head`/`full_content`), graphic types, step labels 3. **Capture B-roll** — Screenshot web pages via Playwright (see `references/screenshot-pipeline.md`), save to `public/` 4. **Build graphics** — Create data graphics (DataTable, ProcessSteps, EliminationList) from storyboard data 5. **Compose** — Wire storyboard into ShortFormEdit: avatar layer + graphics panel + captions + title card 6. **QA** — Verify: text legibility at mobile size, caption sync, avatar framing, mode transitions. Score >= 90/100 7. **Render** — `npx remotion render ShortFormEdit output.mp4` ## Talking Head Short Checklist Before rendering any talking-head short-form video, verify ALL items: ### Pre-render (Mandatory) - [ ] **Silence/filler removal** — NEVER render raw footage. Run `roughcut-*.ts` script first (Silero VAD + detectFillers + buildEdl + applyEdl). Expect 30-40% duration reduction - [ ] **Trim leading silence** — always cut dead air before the first word. Keep 50ms pre-roll max. The roughcut pipeline adds a synthetic cut from `0` to `firstWord.start - 0.05` automatically - [ ] **Don't cut pauses < 0.3s** — use `silenceBuffer: 0.3` (300ms) in buildEdl to preserve natural speech rhythm. Detect silences at `minDuration: 0.5` but the buffer ensures only silences > ~1.1s are actually removed - [ ] **Review transcript after edit** — ALWAYS review the roughcut transcript output. Verify no meaningful words were cut. Re-check any flagged gaps (>0.5s between words). The script prints `[CUT]` lines for removed words and `[GAP]` lines for suspicious gaps - [ ] **Remap word timestamps** — after cuts, offset all subsequent word timestamps by `-cutDuration` so captions stay synced - [ ] **Target duration** — final video should be under 40s for Reels/TikTok (60s absolute max) ### Captions - [ ] **Font size minimum: 108px** at 1080x1920 — 36px is illegible on mobile. `CAPTION_FONT_SIZE` in design-system.ts - [ ] **Pill padding scales with fontSize** — `padding: ${fontSize * 0.33}px ${fontSize * 0.67}px` - [ ] **wordsPerGroup: 2** for pill style — single-line only, never wrap to 2 lines. Max 2 words per pill group. `PILL_DEFAULTS.wordsPerGroup` in design-system.ts - [ ] **Pill flexWrap: nowrap** — caption pill container must use `flexWrap: 'nowrap'` to enforce single-line rendering - [ ] **Caption position** — bottom 10-15% of frame, never overlapping face ### Title Card - [ ] **No full-screen dark backdrop** — KineticTitle renders text with motion blur over the video. Never add `AbsoluteFill` with `backgroundColor: rgba(0,0,0,...)` - [ ] **Accent color: #ED0D51** (brand pink, not default red) ### Audio - [ ] **Background music** from your own royalty-free library (`$MUSIC_DIR`, if you have one) — adopt a naming convention like `yes-` prefixed = pre-approved, `maybe-` = needs approval, `buildup-` = non-talking sections only. If you have no library, skip background music. (See `references/music-library.md`) - [ ] **Music volume: 0.10** (Remotion `volume` prop) — barely perceptible under voice - [ ] **Normalize to -16 LUFS** — `audio.targetLufs: -16` in EDL config - [ ] **Fade music** — 1-2s fade in/out, never hard cut ### QA - [ ] Captions synced with speech throughout (spot-check 3+ points) - [ ] Text legible at mobile size (captions fill ~50-60% of screen width) - [ ] No long silences or "um/uh" remain - [ ] Title card shows video behind text (transparent, no black overlay) - [ ] Music is subtle — voice always dominates ## Detailed References For comprehensive rules on all Remotion features (30+ topics): - `references/rules.md` — animations, media, fonts, captions, charts, 3D, transitions, utilities - `references/video-editor-integration.md` — existing video pipeline, B-roll/music catalogs, critical gotchas - `references/production-patterns.md` — spring presets, layer architecture, storyboard schema, 7 reusable components, data graphic templates - `references/screenshot-pipeline.md` — Playwright screenshot capture recipe for deterministic B-roll - `references/music-library.md` — 25-track catalog, audio level preferences, mood-based selection guide ## Packages | Package | Purpose | |---------|---------| | `remotion` | Core framework | | `@remotion/cli` | CLI tools, Remotion Studio | | `@remotion/renderer` | Programmatic rendering | | `@remotion/player` | In-browser video player | | `@remotion/captions` | Caption rendering + SRT import | | `@remotion/transitions` | Scene transitions (fade, wipe, etc.) | | `@remotion/gif` | Animated GIF support | | `@remotion/google-fonts` | Google Fonts loading | | `@remotion/three` | Three.js 3D integration | | `@remotion/lottie` | Lottie animation support | | `@remotion/shapes` | SVG shape primitives | ## Lessons Learned - [2026-04-05] [CAPTION_MAX_WIDTH]: Caption pills must constrain max-width to 90% and use flexWrap: wrap to prevent text clipping at right edge. Font size 108px with padding demands tight bounds. Add maxWidth style to pill container div. - [2026-04-05] [TITLE_CAPTION_TIMING]: KineticTitle and first caption must not overlap. Increase title.durationSec to 2.0s minimum, and ensure first caption word starts after title animation completes. Use manifest timing to delay captions if needed. - [2026-04-05] [SENTENCE_BOUNDARY]: Pill caption grouping must detect sentence-end punctuation (., !, ?) and force new group immediately after. Don't split sentences across groups. Modify wordGroups builder to check SENTENCE_ENDERS.has(word.text.slice(-1)). - [2026-04-05] [TITLE_INITIAL_FRAME]: KineticTitle animation must have 3-4 static, readable frames before spring animation starts. Add animFrame = Math.max(0, frame - 3) offset to delay all animations. Frames 0-2 will show unblurred text at initial scale. - [2026-04-05] [WORD_HYPHENATION]: Caption pills must use CSS word-break: keep-all and whiteSpace: nowrap on word spans to prevent hyphenation and awkward wrapping of compound words (one-time, end-to-end, etc.). - [2026-04-05] [END_CTA_REQUIRED]: Talking-head videos must end with a CTA caption, not silent footage. Add final transcription word with CTA text (e.g., "Claim now.") at final timestamp, starting 1s before video end. Never have > 1s of silent staring at video end. - [2026-04-05] [LEADING_SILENCE]: Always trim dead air before first spoken word. Roughcut pipeline adds synthetic cut from 0 to firstWord.start - 0.05s. Even 0.1s of leading silence hurts hook delivery. - [2026-04-05] [SILENCE_BUFFER]: Use silenceBuffer: 0.3 (300ms) in buildEdl. VAD gaps < 0.2s are auto-skipped. Preserves natural pauses. Previous FFmpeg silencedetect + 0.15 buffer was too aggressive and inaccurate. - [2026-04-05] [SILERO_VAD]: Use Silero VAD (python/detect_speech.py) for silence detection instead of FFmpeg silencedetect. ML-based, finds precise speech boundaries. Invert speech segments to get silence gaps. 10x more accurate than threshold-based detection. - [2026-04-05] [TRANSCRIPT_REVIEW]: Always review roughcut transcript output before rendering. Script prints [CUT] for removed words and [GAP] for suspicious gaps >0.5s. Verify no meaningful words were lost. - [2026-04-05] [CAPTION_SINGLE_LINE]: Caption pills must be single-line. wordsPerGroup: 2, flexWrap: nowrap. 3 words caused wrapping at 108px fontSize with 90% maxWidth. 2 words stays single-line reliably. - [2026-04-05] [MUSIC_LIBRARY]: Background music should come from your own royalty-free library (`$MUSIC_DIR`, if available). A useful convention: `yes-` prefixed = pre-approved, `maybe-` = needs user approval, `buildup-` = non-talking sections only.