# claude-video — Let any agent watch any video

Give any AI agent the ability to *watch* a video the way a human would: by reading the frames as images and following the spoken transcript. You paste a URL or local path, ask a question, and the tool does the rest.

## When to use

- User asks a question about a specific video (URL or local path)
- "What happens at 2:30?" / "Summarize this video" / "What's the hook in this clip?"
- Analyze content hooks, pacing, or narrative structure
- Diagnose bugs from a screen recording
- Cut the hype from a product update video and extract only the facts
- Turn a YouTube playlist into structured notes
- Extract a recipe, tutorial steps, or how-to from a video
- Compare what's *said* (transcript) vs. what's *shown* (frames)
- Time-stamped question answering about video content

## Installation

### Generic agent (npx skills)

```bash
npx skills add bradautomates/claude-video -g
```

The `-g` flag installs globally. `yt-dlp` and `ffmpeg` install automatically on first run — zero config.

### Claude Code (plugin marketplace)

```
/plugin marketplace add bradautomates/claude-video
```

### Prerequisites

- **Node.js** (for `npx`) — required for the installer
- That's it. `yt-dlp` and `ffmpeg` are auto-installed on first run.

### Whisper API fallback (optional)

Native captions are always tried first (free, instant). If a video has no usable captions, the tool falls back to a Whisper API. To use the fallback you need an API key for one of:

- **Groq** `whisper-large-v3` (preferred — fast and cheap/free tier)
- **OpenAI** `whisper-1` (fallback)

Without either key, videos lacking native captions will return frames only (no transcript). Most YouTube/TikTok videos have auto-generated captions.

## Usage

### Basic syntax

```
/watch <URL-or-path> [your question]
```

### Examples

```
/watch https://youtu.be/dQw4w9WgXcQ what happens at 2:30?
/watch https://www.tiktok.com/@user/video/123 summarize this
/watch ./recordings/screen-recording.mp4 what error does the user hit?
/watch ./lecture.mkv give me timestamped notes
```

## How it works under the hood

| Step | Tool | What happens |
|------|------|--------------|
| Captions | `yt-dlp` | Pull native/auto captions — free, instant, no full download |
| Fallback captions | Whisper API | Groq `whisper-large-v3` (preferred) or OpenAI `whisper-1` |
| Selective download | `yt-dlp` | Only download the segments needed (or full file if frame extraction required) |
| Frame extraction | `ffmpeg` | Keyframe / scene-change / uniform sampling → 512px-wide JPEGs |
| Frame dedup | internal | Drop near-identical frames to save tokens |
| Transcript | `yt-dlp` / Whisper | Timestamped, synced to extracted frames |
| Vision | agent | Every surviving frame is read as an image |

## Detail modes

| Mode | Frame strategy | Frame cap | When to use |
|------|----------------|-----------|-------------|
| **efficient** | keyframes only | 50 | Quick questions, short clips, token budget tight |
| **balanced** (default) | scene-change detection | 100 | General analysis, summaries, most questions |
| **token-burner** | uncapped | none | Forensic frame-by-frame review, bug diagnosis, when you need every frame |

Start with **balanced**. Drop to **efficient** if you only need captions/transcript and a rough sense of the visuals. Use **token-burner** only when the user specifically needs exhaustive visual coverage.

## Caption-first approach

The tool is **caption-first**. For questions that can be answered from the transcript alone (e.g. "summarize what's said", "what did they claim about X?"), it may pull captions and answer without downloading or extracting any frames at all. This is free and near-instant.

Frames are only extracted when the question requires *seeing* the video — visual content, on-screen text, UI bugs, "what happens at 2:30", etc.

## Use cases

- **Content/hook analysis** — "What's the hook in the first 10 seconds? Is it visual or verbal?"
- **Bug diagnosis from screen recordings** — "The user clicks X and then what breaks?" Pair with `token-burner` mode so no frame is missed.
- **Summarization** — "Give me the 5 key points with timestamps."
- **Hype-cutting** — "Strip the marketing language. What are the actual claims/features, with timestamps?"
- **Playlist to notes** — Feed it each video URL in turn; collect timestamped notes into a single doc.
- **Recipe / tutorial extraction** — "List the ingredients and steps in order with timestamps."
- **Said-vs-shown comparison** — Ask both "what do they say?" (transcript) and "what's on screen?" (frames), then diff.

## Pitfalls

1. **First run downloads yt-dlp + ffmpeg** — the initial invocation takes longer. Subsequent runs are fast.

2. **No native captions = no transcript unless an API key is set** — check whether a transcript was returned before relying on spoken content.

3. **Frame count vs. token cost** — every frame is sent to the model as an image. `token-burner` on a long video can produce hundreds of frames. Default to `balanced`.

4. **Near-identical frame deduplication is lossy by design** — static talking-head videos may collapse to a handful of frames. Use `token-burner` if dropped frames matter.

5. **Local file paths must be absolute or relative to the agent's working directory** — verify the file exists before calling.

6. **Platform-specific URL quirks** — TikTok, Instagram, and X occasionally require a logged-in session for `yt-dlp` to fetch captions.

7. **`npx skills add ... -g` requires Node** — confirm Node.js is installed before attempting the install command.

## Reference

- Project: https://github.com/bradautomates/claude-video
- Install (generic): `npx skills add bradautomates/claude-video -g`
- Install (Claude Code): `/plugin marketplace add bradautomates/claude-video`
- License: free and open source