Agent Setup Prompt
Set up a repeatable video-as-prompt run for this project before doing any full video analysis.
1. Resolve the `oracle-video` skill first. Check for `skills/oracle-video/SKILL.md`; if it is not bundled, check the common local install path at `.agents/skills/oracle-video/SKILL.md`. Read the skill and any directly referenced repeatable video process file before creating outputs. If the skill is missing, stop and report the missing path checks.
2. Create a run id like `<YYYY-MM-DD>-<short-slug>` and a run folder at `runs/<run-id>/` with this structure: `inputs/`, `receipts/`, `chunks/01/`, `analysis/`, `final/`, `keyframes/`, and `contact-sheets/`.
3. Accept either a local video path or a video URL. For a local file, copy it to `inputs/source-video.<ext>` and keep that copy immutable. For a URL, first save metadata with `yt-dlp --dump-single-json --skip-download --no-check-formats` to `receipts/source.metadata.json`, then download with `yt-dlp -f "bv*+ba/best"` into `inputs/`.
4. Write a source hash with `shasum -a 256` to `inputs/source-video.sha256`. Confirm the file exists and is non-empty.
5. Capture media facts with `ffprobe -hide_banner -show_format -show_streams -of json` and save them to `receipts/ffprobe.source.json`. Confirm the file exists and is non-empty. If `ffprobe`, `ffmpeg`, or `yt-dlp` is missing, stop setup and report the missing tool plus the failed command.
6. Prepare `chunks/01/chunk.mp4`. For a short video, copy the source video into that path. For a long video, create additional numbered chunk folders as needed and save a timestamp mapping note before any analysis. Use absolute paths for every video file that will be attached to Oracle.
7. Write `chunks/01/gemini-prompt.md`. It must request separate lanes for transcript evidence, chat evidence, visual evidence, and inference. It must require preserved timestamps, forbid invented URLs, tools, commands, files, UI labels, and speakers, and end with the exact final line `END_OF_VIDEO_ANALYSIS`.
8. Write `chunks/01/expected-sentinel.txt` containing only `END_OF_VIDEO_ANALYSIS`. Confirm the sentinel appears in `gemini-prompt.md` and that the expected sentinel file is non-empty.
9. Prepare `chunks/01/oracle-call.json` or equivalent setup notes with the required Gemini browser-mode contract: `engine: "browser"`, `model: "gemini-3-pro"`, `browserAttachments: "always"`, `files` containing an absolute path to `chunks/01/chunk.mp4`, and a stable slug. Do not use API mode for video analysis.
10. Generate keyframes and a contact sheet if `ffmpeg` is available. Save frames under `keyframes/` using a low-frequency extraction such as `fps=1/10`, then save a tiled contact sheet under `contact-sheets/`. If generation is skipped, save the exact commands to run later and report why they were not run.
11. Create evidence lane files under `analysis/`: `transcript.md`, `chat.md`, `visual-evidence.md`, and `inference.md`. Keep spoken claims, chat or comments, visible facts, and reasoned conclusions separate. Label inference clearly until code, runtime behavior, or visual checks support it.
12. Create `analysis/omissions-pass-prompt.md`. It should ask only for missing visible UI states, skipped timestamps, unsupported claims, invented details, uncertainty, and patch-style corrections. It must require the response to end with `END_OF_VIDEO_ANALYSIS`.
13. Create downstream artifact stubs under `final/`, such as `implementation-prompt.md`, `qa-checklist.md`, and `variation-matrix.md`. Leave evidence-dependent fields blank until verified analysis exists.
14. Enforce privacy boundaries before any upload or external analysis. Do not print full environment output, tokens, cookies, API keys, private endpoints, resolved secret values, or raw private config. Do not copy secret helper files into the run folder. If the video, metadata, transcript, or chat text contains credentials or private config, stop and report the issue instead of uploading.
15. Run a setup verification gate before Gemini analysis. Report `OK` or `MISSING` for: `oracle-video` skill resolved, source video exists, source hash non-empty, `ffprobe.source.json` non-empty, URL metadata present when the input is a URL, chunk file exists, Gemini prompt exists, sentinel file exists, Oracle call uses browser mode with `gemini-3-pro` and `browserAttachments: "always"`, Oracle file path is absolute, evidence lane files exist, keyframes/contact sheet exist or commands are saved, omissions prompt exists, and downstream stubs exist.
16. Stop before full video analysis unless every required setup check passes. Return the run id, absolute run folder path, commands run, verification report, failed checks, and the smallest next fix if anything is missing. Only say it is ready for Gemini analysis after the setup gate passes.
Big Idea
Video is prompt material.
Try This Now
For a short bug repro video, create a run folder and save a source hash.
Write the Gemini prompt with separate lanes: transcript, chat, visual evidence, inference.
End the prompt with END_OF_VIDEO_ANALYSIS before analysis starts.
A screen recording can show things a written prompt usually loses:
- order of operations
- cursor movement
- hesitation
- timing
- hidden assumptions
- actual UI state
- terminal output
- visible browser pages
- misclicks
- loading states
- visual hierarchy
- broken affordances
- spoken explanation
- chat context
A transcript can tell you what someone said. A video can show what happened.
Tool Reference
Use oracle-video when the task depends on what is visible or audible in a recording.
Conceptually:
oracle-packx bundles code/text context for ChatGPT.
oracle-video attaches video and sends it to Gemini through Oracle browser mode.
Correct Oracle call shape for video analysis:
mcp__oracle__consult({
prompt: "[video-analysis-slug]\n\n{contents of chunks/01/gemini-prompt.md}",
files: ["/absolute/path/to/runs/bug-repro/chunks/01/chunk.mp4"],
engine: "browser",
model: "gemini-3-pro",
browserAttachments: "always",
slug: "video-analysis-slug"
})
Required:
engine: "browser"
model: "gemini-3-pro"
browserAttachments: "always"
absolute file path
Do not use API mode for video analysis.
When To Reach For This
Use video when the important context is visible, temporal, or hard to describe.
| Use Video For | Why |
|---|---|
| bug reproduction clip | captures exact UI state, clicks, errors, timing |
| design feedback video | preserves visual hierarchy and spoken intent |
| app walkthrough | turns observed workflow into requirements and QA checks |
| loading/timing bug | shows delay, spinner, race, or order of events |
| terminal/browser workflow | preserves commands, pages, output, and sequence |
| long tutorial or workshop recording | can become chapters, exercises, references, and follow-up prompts |
Skip video when:
- a short log already proves the issue
- a failing test names the exact bug
- the needed context is one code path
- the video source cannot be preserved
- claims cannot be verified against timestamps, keyframes, or source output
How This Used To Work
What you used to expect:
Watch the video, summarize from memory, then ask the agent to fix the vague thing.
What the workflow gives you now:
Preserve the source, capture hashes and media facts, analyze with Gemini, keep
evidence lanes separate, verify claims, then turn the result into a grounded
prompt.
What you used to expect:
A transcript is the prompt.
What the workflow gives you now:
The transcript is one evidence lane. Visual state, timing, chat, cursor motion,
and inference stay separate until verified.
Core Pattern
source media
-> source receipts
-> Gemini analysis
-> evidence lanes
-> verification
-> downstream prompt
Do not skip the middle.
video without source receipts = hard to trust
analysis without timestamps = hard to verify
transcript without visuals = incomplete
claims without keyframes = risky
prompt without verification = hallucination bait
Short Path vs Long Path
| Path | Use For | Shape |
|---|---|---|
| short path | bug repros, design feedback clips, quick app walkthroughs | preserve source, one Gemini pass, keyframes, downstream artifact |
| long path | long tutorials, workshop recordings, multi-section demos | source receipts, segment/chunk, analyze chunks, omissions pass, reconcile, final packet |
Short path output examples:
- repro steps
- implementation prompt
- design-change brief
- QA checklist
- requirements note
Long path output examples:
- chapter index
- lesson notes
- exercise packet
- resource list
- follow-up prompts
- Section 6 variation matrix
Run Folder Shape
Use a run folder so the video stays tied to its analysis.
runs/<date>-<slug>/
inputs/
source-video.mp4
source-video.sha256
receipts/
ffprobe.source.json
chunks/
01/
chunk.mp4
gemini-prompt.md
expected-sentinel.txt
output.log
analysis.md
analysis/
final/
keyframes/
contact-sheets/
Create the folders:
RUN_ID="checkout-repro"
RUN_DIR="runs/$RUN_ID"
mkdir -p "$RUN_DIR"/{inputs,receipts,analysis,final,keyframes,contact-sheets}
mkdir -p "$RUN_DIR/chunks/01"
Preserve The Source
Keep the original media immutable.
For a local file:
cp /path/to/source-video.mp4 "$RUN_DIR/inputs/source-video.mp4"
shasum -a 256 "$RUN_DIR/inputs/source-video.mp4" \
> "$RUN_DIR/inputs/source-video.sha256"
For a URL source, capture metadata first when useful:
yt-dlp --dump-single-json --skip-download --no-check-formats "$URL" \
> "$RUN_DIR/receipts/source.metadata.json"
Then download:
yt-dlp -f "bv*+ba/best" -o "$RUN_DIR/inputs/source-video.%(ext)s" "$URL"
Capture media facts:
ffprobe -hide_banner -show_format -show_streams -of json \
"$RUN_DIR/inputs/source-video.mp4" \
> "$RUN_DIR/receipts/ffprobe.source.json"
Confirm:
test -s "$RUN_DIR/inputs/source-video.sha256" && echo "sha256:OK"
test -s "$RUN_DIR/receipts/ffprobe.source.json" && echo "ffprobe:OK"
Evidence Lanes
Keep these lanes separate:
| Lane | What It Means | Example |
|---|---|---|
| transcript | spoken words | “The save button disappears.” |
| chat | chat messages or comments | “This only happens after refresh.” |
| visual evidence | visible UI state | save button hidden at 00:01:34 |
| inference | reasoned conclusion | likely state reset after token refresh |
Plain rule:
A spoken claim is not a visible fact.
A visible fact is not automatically the cause.
An inference must stay labeled until code or runtime proof confirms it.
Prompt With A Sentinel
Save the Gemini prompt beside the chunk.
cat > "$RUN_DIR/chunks/01/gemini-prompt.md" <<'EOF'
Analyze this video as prompt material for a software project.
Return:
1. short summary
2. timestamped workflow
3. transcript evidence
4. chat evidence
5. visual evidence
6. inferences, clearly labeled
7. likely downstream artifacts
8. omissions or uncertainty
Rules:
- Do not invent URLs, tools, commands, files, UI labels, or speakers.
- Preserve original timestamps.
- Separate visible facts from spoken claims and inferences.
- Include the exact sentinel below as the last line.
END_OF_VIDEO_ANALYSIS
EOF
printf 'END_OF_VIDEO_ANALYSIS\n' \
> "$RUN_DIR/chunks/01/expected-sentinel.txt"
Confirm:
rg -n "END_OF_VIDEO_ANALYSIS" "$RUN_DIR/chunks/01/gemini-prompt.md"
test -s "$RUN_DIR/chunks/01/expected-sentinel.txt" && echo "sentinel:OK"
Analyze With Gemini
Use Oracle browser mode and attach the video.
mcp__oracle__consult({
prompt: "[checkout-repro-video]\n\n{contents of runs/.../chunks/01/gemini-prompt.md}",
files: ["/absolute/path/to/runs/checkout-repro/chunks/01/chunk.mp4"],
engine: "browser",
model: "gemini-3-pro",
browserAttachments: "always",
slug: "checkout-repro-video"
})
After the run, read the local output:
sed -n '1,220p' ~/.oracle/sessions/checkout-repro-video/output.log
rg -n "END_OF_VIDEO_ANALYSIS" ~/.oracle/sessions/checkout-repro-video/output.log
If the output is clipped in a tool response, treat output.log as the local source to inspect.
Verify The Analysis
Check that the answer is grounded.
| Check | Command Or Evidence |
|---|---|
| source hash exists | test -s "$RUN_DIR/inputs/source-video.sha256" |
| media facts exist | test -s "$RUN_DIR/receipts/ffprobe.source.json" |
| prompt saved | test -s "$RUN_DIR/chunks/01/gemini-prompt.md" |
| sentinel present | rg -n "END_OF_VIDEO_ANALYSIS" output.log |
| timestamps preserved | compare analysis timestamps to source duration |
| visual claims checked | keyframes or contact sheet |
| invented facts removed | no unsupported URLs/tools/commands/files |
| omissions captured | missing sections are listed as uncertainty |
Extract keyframes for visual checks:
mkdir -p "$RUN_DIR/keyframes"
ffmpeg -i "$RUN_DIR/inputs/source-video.mp4" \
-vf "fps=1/10" \
"$RUN_DIR/keyframes/frame-%04d.jpg"
Create a contact sheet:
ffmpeg -pattern_type glob -i "$RUN_DIR/keyframes/*.jpg" \
-vf "scale=320:-1,tile=4x4" \
"$RUN_DIR/contact-sheets/source-contact-sheet.jpg"
Omissions Pass
Run an omissions pass when coverage is incomplete.
Ask:
Review the prior video analysis for omissions.
Focus only on missing visible UI states, skipped timestamps, unsupported claims,
invented details, and uncertainty. Do not rewrite the whole analysis. Return a
patch-style list of corrections and end with END_OF_VIDEO_ANALYSIS.
Use the omissions pass when:
- the first analysis only summarizes the transcript
- key UI states are missing
- timestamps are vague
- the model invents tools, URLs, files, or labels
- a long recording was not chunked
- the downstream prompt depends on visual details
Turn Video Into Prompt Material
Common downstream artifacts:
| Video Source | Downstream Artifact |
|---|---|
| bug reproduction | repro steps, expected vs actual behavior, implementation prompt |
| design feedback | visual acceptance criteria, change brief, QA checklist |
| app walkthrough | requirements, acceptance criteria, open questions |
| terminal workflow | command sequence, failure point, debugging prompt |
| long recording | chapters, exercises, resources, follow-up packet |
| UI alternatives | Section 6 variation matrix |
Bug prompt shape:
# Implementation Prompt From Video
## Source
- video:
- source hash:
- ffprobe:
- analysis:
- keyframes:
## Observed Failure
- timestamp:
- visual evidence:
- transcript/chat evidence:
- expected behavior:
- actual behavior:
## Request
Implement the smallest safe fix.
## Constraints
- do not change:
- preserve:
## Verification
- command:
- what passing means:
Variation seed shape:
# Video-Derived Variation Matrix
## Goal
## Evidence From Video
- timestamp:
- observation:
- keyframe:
- why it matters:
## Invariants
- same source evidence:
- same target flow:
- same verification command:
- same review criteria:
## Variable Under Test
- variable:
- why this variable matters:
- what is out of scope:
Ask The Agent This
Analyze a bug reproduction:
Use the video analysis workflow.
Preserve the source video, save a source hash and ffprobe output, write a Gemini
prompt with END_OF_VIDEO_ANALYSIS, analyze the video through Oracle browser mode
with Gemini, then separate transcript evidence, visual evidence, chat evidence,
and inference. Turn the verified result into repro steps and an implementation
prompt for the smallest safe fix.
Review design feedback:
Turn this design feedback video into a change brief.
Keep visible UI facts separate from spoken opinions. Save keyframes for the
visual claims. Produce acceptance criteria, a QA checklist, and open questions.
Do not invent UI labels or routes that are not visible or spoken.
Create a Section 6 seed:
Use this verified video analysis to create a controlled variation matrix.
Choose one variable under test, preserve the same source evidence and
verification command for every candidate, and write three candidate settings.
What To Confirm / How To Know It Worked
| Goal | Confirmation |
|---|---|
| source preserved | original media exists under inputs/ |
| source hash saved | source-video.sha256 exists and is non-empty |
| media facts saved | ffprobe.source.json exists |
| prompt saved | chunks/01/gemini-prompt.md exists |
| sentinel required | expected-sentinel.txt exists |
| Oracle call shape correct | engine: "browser", model: "gemini-3-pro", browserAttachments: "always" |
| output inspected | ~/.oracle/sessions/<slug>/output.log read |
| sentinel present | END_OF_VIDEO_ANALYSIS found in output |
| lanes separated | transcript, chat, visual evidence, inference are separate |
| visual claims checked | keyframes/contact sheet exists |
| unsupported claims removed | no invented URLs/tools/commands/files |
| downstream artifact saved | repro, requirements, QA, brief, lesson, or matrix exists |
If It Goes Wrong
No Source Hash
Create it before analysis:
shasum -a 256 "$RUN_DIR/inputs/source-video.mp4" \
> "$RUN_DIR/inputs/source-video.sha256"
No ffprobe Output
Run:
ffprobe -hide_banner -show_format -show_streams -of json \
"$RUN_DIR/inputs/source-video.mp4" \
> "$RUN_DIR/receipts/ffprobe.source.json"
Oracle Call Used API Mode
Rerun with browser mode.
engine: "browser"
model: "gemini-3-pro"
browserAttachments: "always"
File Path Was Relative
Use an absolute video path in files.
realpath "$RUN_DIR/chunks/01/chunk.mp4"
Sentinel Missing
Read output.log. If the sentinel is truly missing, rerun or ask for an omissions pass.
rg -n "END_OF_VIDEO_ANALYSIS" ~/.oracle/sessions/<slug>/output.log
Analysis Only Summarizes Transcript
Run an omissions pass focused on visual evidence:
Find missing visual UI states, cursor actions, timing, browser/terminal state,
and unsupported claims. Preserve timestamps.
Visual Claim Has No Keyframe
Extract keyframes or create a contact sheet. Downgrade unsupported claims to inference.
URL, Tool, Command, Or File Appears Without Evidence
Remove it or label it as inference. Do not let invented details become prompt constraints.
Long Recording Was Not Chunked
Create chunks and analyze them separately. Preserve original timestamps across chunks.
References And Resources
oracle-videolocal skill:skills/oracle-video/SKILL.mdif bundled, or common local skill install under.agents/skills/oracle-video/SKILL.mdyt-dlp: https://github.com/yt-dlp/yt-dlp- FFmpeg documentation: https://ffmpeg.org/documentation.html
ffprobedocumentation: https://ffmpeg.org/ffprobe.html- Gemini app: https://gemini.google.com/app