Video analysis and evidence workflows: practical reference

Repeatable video-to-prompt workflow covering source preservation, hashing and media receipts, chunking, Oracle/Gemini analysis, timestamped evidence lanes, sentinel checks, keyframes, omissions review, privacy gates, and downstream implementation, QA, and…

Agent Setup Prompt

Set up a repeatable video-as-prompt run for this project before doing any full video analysis.

1. Resolve the `oracle-video` skill first. Check for `skills/oracle-video/SKILL.md`; if it is not bundled, check the common local install path at `.agents/skills/oracle-video/SKILL.md`. Read the skill and any directly referenced repeatable video process file before creating outputs. If the skill is missing, stop and report the missing path checks.

2. Create a run id like `<YYYY-MM-DD>-<short-slug>` and a run folder at `runs/<run-id>/` with this structure: `inputs/`, `receipts/`, `chunks/01/`, `analysis/`, `final/`, `keyframes/`, and `contact-sheets/`.

3. Accept either a local video path or a video URL. For a local file, copy it to `inputs/source-video.<ext>` and keep that copy immutable. For a URL, first save metadata with `yt-dlp --dump-single-json --skip-download --no-check-formats` to `receipts/source.metadata.json`, then download with `yt-dlp -f "bv*+ba/best"` into `inputs/`.

4. Write a source hash with `shasum -a 256` to `inputs/source-video.sha256`. Confirm the file exists and is non-empty.

5. Capture media facts with `ffprobe -hide_banner -show_format -show_streams -of json` and save them to `receipts/ffprobe.source.json`. Confirm the file exists and is non-empty. If `ffprobe`, `ffmpeg`, or `yt-dlp` is missing, stop setup and report the missing tool plus the failed command.

6. Prepare `chunks/01/chunk.mp4`. For a short video, copy the source video into that path. For a long video, create additional numbered chunk folders as needed and save a timestamp mapping note before any analysis. Use absolute paths for every video file that will be attached to Oracle.

7. Write `chunks/01/gemini-prompt.md`. It must request separate lanes for transcript evidence, chat evidence, visual evidence, and inference. It must require preserved timestamps, forbid invented URLs, tools, commands, files, UI labels, and speakers, and end with the exact final line `END_OF_VIDEO_ANALYSIS`.

8. Write `chunks/01/expected-sentinel.txt` containing only `END_OF_VIDEO_ANALYSIS`. Confirm the sentinel appears in `gemini-prompt.md` and that the expected sentinel file is non-empty.

9. Prepare `chunks/01/oracle-call.json` or equivalent setup notes with the required Gemini browser-mode contract: `engine: "browser"`, `model: "gemini-3-pro"`, `browserAttachments: "always"`, `files` containing an absolute path to `chunks/01/chunk.mp4`, and a stable slug. Do not use API mode for video analysis.

10. Generate keyframes and a contact sheet if `ffmpeg` is available. Save frames under `keyframes/` using a low-frequency extraction such as `fps=1/10`, then save a tiled contact sheet under `contact-sheets/`. If generation is skipped, save the exact commands to run later and report why they were not run.

11. Create evidence lane files under `analysis/`: `transcript.md`, `chat.md`, `visual-evidence.md`, and `inference.md`. Keep spoken claims, chat or comments, visible facts, and reasoned conclusions separate. Label inference clearly until code, runtime behavior, or visual checks support it.

12. Create `analysis/omissions-pass-prompt.md`. It should ask only for missing visible UI states, skipped timestamps, unsupported claims, invented details, uncertainty, and patch-style corrections. It must require the response to end with `END_OF_VIDEO_ANALYSIS`.

13. Create downstream artifact stubs under `final/`, such as `implementation-prompt.md`, `qa-checklist.md`, and `variation-matrix.md`. Leave evidence-dependent fields blank until verified analysis exists.

14. Enforce privacy boundaries before any upload or external analysis. Do not print full environment output, tokens, cookies, API keys, private endpoints, resolved secret values, or raw private config. Do not copy secret helper files into the run folder. If the video, metadata, transcript, or chat text contains credentials or private config, stop and report the issue instead of uploading.

15. Run a setup verification gate before Gemini analysis. Report `OK` or `MISSING` for: `oracle-video` skill resolved, source video exists, source hash non-empty, `ffprobe.source.json` non-empty, URL metadata present when the input is a URL, chunk file exists, Gemini prompt exists, sentinel file exists, Oracle call uses browser mode with `gemini-3-pro` and `browserAttachments: "always"`, Oracle file path is absolute, evidence lane files exist, keyframes/contact sheet exist or commands are saved, omissions prompt exists, and downstream stubs exist.

16. Stop before full video analysis unless every required setup check passes. Return the run id, absolute run folder path, commands run, verification report, failed checks, and the smallest next fix if anything is missing. Only say it is ready for Gemini analysis after the setup gate passes.

Big Idea

Video is prompt material.

Try This Now

For a short bug repro video, create a run folder and save a source hash. Write the Gemini prompt with separate lanes: transcript, chat, visual evidence, inference. End the prompt with END_OF_VIDEO_ANALYSIS before analysis starts.

A screen recording can show things a written prompt usually loses:

  • order of operations
  • cursor movement
  • hesitation
  • timing
  • hidden assumptions
  • actual UI state
  • terminal output
  • visible browser pages
  • misclicks
  • loading states
  • visual hierarchy
  • broken affordances
  • spoken explanation
  • chat context

A transcript can tell you what someone said. A video can show what happened.

Tool Reference

Use oracle-video when the task depends on what is visible or audible in a recording.

Conceptually:

oracle-packx bundles code/text context for ChatGPT.
oracle-video attaches video and sends it to Gemini through Oracle browser mode.

Correct Oracle call shape for video analysis:

mcp__oracle__consult({
  prompt: "[video-analysis-slug]\n\n{contents of chunks/01/gemini-prompt.md}",
  files: ["/absolute/path/to/runs/bug-repro/chunks/01/chunk.mp4"],
  engine: "browser",
  model: "gemini-3-pro",
  browserAttachments: "always",
  slug: "video-analysis-slug"
})

Required:

engine: "browser"
model: "gemini-3-pro"
browserAttachments: "always"
absolute file path

Do not use API mode for video analysis.

When To Reach For This

Use video when the important context is visible, temporal, or hard to describe.

Use Video For Why
bug reproduction clip captures exact UI state, clicks, errors, timing
design feedback video preserves visual hierarchy and spoken intent
app walkthrough turns observed workflow into requirements and QA checks
loading/timing bug shows delay, spinner, race, or order of events
terminal/browser workflow preserves commands, pages, output, and sequence
long tutorial or workshop recording can become chapters, exercises, references, and follow-up prompts

Skip video when:

  • a short log already proves the issue
  • a failing test names the exact bug
  • the needed context is one code path
  • the video source cannot be preserved
  • claims cannot be verified against timestamps, keyframes, or source output

How This Used To Work

What you used to expect:

Watch the video, summarize from memory, then ask the agent to fix the vague thing.

What the workflow gives you now:

Preserve the source, capture hashes and media facts, analyze with Gemini, keep
evidence lanes separate, verify claims, then turn the result into a grounded
prompt.

What you used to expect:

A transcript is the prompt.

What the workflow gives you now:

The transcript is one evidence lane. Visual state, timing, chat, cursor motion,
and inference stay separate until verified.

Core Pattern

source media
-> source receipts
-> Gemini analysis
-> evidence lanes
-> verification
-> downstream prompt

Do not skip the middle.

video without source receipts = hard to trust
analysis without timestamps = hard to verify
transcript without visuals = incomplete
claims without keyframes = risky
prompt without verification = hallucination bait

Short Path vs Long Path

Path Use For Shape
short path bug repros, design feedback clips, quick app walkthroughs preserve source, one Gemini pass, keyframes, downstream artifact
long path long tutorials, workshop recordings, multi-section demos source receipts, segment/chunk, analyze chunks, omissions pass, reconcile, final packet

Short path output examples:

  • repro steps
  • implementation prompt
  • design-change brief
  • QA checklist
  • requirements note

Long path output examples:

  • chapter index
  • lesson notes
  • exercise packet
  • resource list
  • follow-up prompts
  • Section 6 variation matrix

Run Folder Shape

Use a run folder so the video stays tied to its analysis.

runs/<date>-<slug>/
  inputs/
    source-video.mp4
    source-video.sha256
  receipts/
    ffprobe.source.json
  chunks/
    01/
      chunk.mp4
      gemini-prompt.md
      expected-sentinel.txt
      output.log
      analysis.md
  analysis/
  final/
  keyframes/
  contact-sheets/

Create the folders:

RUN_ID="checkout-repro"
RUN_DIR="runs/$RUN_ID"
mkdir -p "$RUN_DIR"/{inputs,receipts,analysis,final,keyframes,contact-sheets}
mkdir -p "$RUN_DIR/chunks/01"

Preserve The Source

Keep the original media immutable.

For a local file:

cp /path/to/source-video.mp4 "$RUN_DIR/inputs/source-video.mp4"
shasum -a 256 "$RUN_DIR/inputs/source-video.mp4" \
  > "$RUN_DIR/inputs/source-video.sha256"

For a URL source, capture metadata first when useful:

yt-dlp --dump-single-json --skip-download --no-check-formats "$URL" \
  > "$RUN_DIR/receipts/source.metadata.json"

Then download:

yt-dlp -f "bv*+ba/best" -o "$RUN_DIR/inputs/source-video.%(ext)s" "$URL"

Capture media facts:

ffprobe -hide_banner -show_format -show_streams -of json \
  "$RUN_DIR/inputs/source-video.mp4" \
  > "$RUN_DIR/receipts/ffprobe.source.json"

Confirm:

test -s "$RUN_DIR/inputs/source-video.sha256" && echo "sha256:OK"
test -s "$RUN_DIR/receipts/ffprobe.source.json" && echo "ffprobe:OK"

Evidence Lanes

Keep these lanes separate:

Lane What It Means Example
transcript spoken words “The save button disappears.”
chat chat messages or comments “This only happens after refresh.”
visual evidence visible UI state save button hidden at 00:01:34
inference reasoned conclusion likely state reset after token refresh

Plain rule:

A spoken claim is not a visible fact.
A visible fact is not automatically the cause.
An inference must stay labeled until code or runtime proof confirms it.

Prompt With A Sentinel

Save the Gemini prompt beside the chunk.

cat > "$RUN_DIR/chunks/01/gemini-prompt.md" <<'EOF'
Analyze this video as prompt material for a software project.

Return:

1. short summary
2. timestamped workflow
3. transcript evidence
4. chat evidence
5. visual evidence
6. inferences, clearly labeled
7. likely downstream artifacts
8. omissions or uncertainty

Rules:

- Do not invent URLs, tools, commands, files, UI labels, or speakers.
- Preserve original timestamps.
- Separate visible facts from spoken claims and inferences.
- Include the exact sentinel below as the last line.

END_OF_VIDEO_ANALYSIS
EOF

printf 'END_OF_VIDEO_ANALYSIS\n' \
  > "$RUN_DIR/chunks/01/expected-sentinel.txt"

Confirm:

rg -n "END_OF_VIDEO_ANALYSIS" "$RUN_DIR/chunks/01/gemini-prompt.md"
test -s "$RUN_DIR/chunks/01/expected-sentinel.txt" && echo "sentinel:OK"

Analyze With Gemini

Use Oracle browser mode and attach the video.

mcp__oracle__consult({
  prompt: "[checkout-repro-video]\n\n{contents of runs/.../chunks/01/gemini-prompt.md}",
  files: ["/absolute/path/to/runs/checkout-repro/chunks/01/chunk.mp4"],
  engine: "browser",
  model: "gemini-3-pro",
  browserAttachments: "always",
  slug: "checkout-repro-video"
})

After the run, read the local output:

sed -n '1,220p' ~/.oracle/sessions/checkout-repro-video/output.log
rg -n "END_OF_VIDEO_ANALYSIS" ~/.oracle/sessions/checkout-repro-video/output.log

If the output is clipped in a tool response, treat output.log as the local source to inspect.

Verify The Analysis

Check that the answer is grounded.

Check Command Or Evidence
source hash exists test -s "$RUN_DIR/inputs/source-video.sha256"
media facts exist test -s "$RUN_DIR/receipts/ffprobe.source.json"
prompt saved test -s "$RUN_DIR/chunks/01/gemini-prompt.md"
sentinel present rg -n "END_OF_VIDEO_ANALYSIS" output.log
timestamps preserved compare analysis timestamps to source duration
visual claims checked keyframes or contact sheet
invented facts removed no unsupported URLs/tools/commands/files
omissions captured missing sections are listed as uncertainty

Extract keyframes for visual checks:

mkdir -p "$RUN_DIR/keyframes"
ffmpeg -i "$RUN_DIR/inputs/source-video.mp4" \
  -vf "fps=1/10" \
  "$RUN_DIR/keyframes/frame-%04d.jpg"

Create a contact sheet:

ffmpeg -pattern_type glob -i "$RUN_DIR/keyframes/*.jpg" \
  -vf "scale=320:-1,tile=4x4" \
  "$RUN_DIR/contact-sheets/source-contact-sheet.jpg"

Omissions Pass

Run an omissions pass when coverage is incomplete.

Ask:

Review the prior video analysis for omissions.

Focus only on missing visible UI states, skipped timestamps, unsupported claims,
invented details, and uncertainty. Do not rewrite the whole analysis. Return a
patch-style list of corrections and end with END_OF_VIDEO_ANALYSIS.

Use the omissions pass when:

  • the first analysis only summarizes the transcript
  • key UI states are missing
  • timestamps are vague
  • the model invents tools, URLs, files, or labels
  • a long recording was not chunked
  • the downstream prompt depends on visual details

Turn Video Into Prompt Material

Common downstream artifacts:

Video Source Downstream Artifact
bug reproduction repro steps, expected vs actual behavior, implementation prompt
design feedback visual acceptance criteria, change brief, QA checklist
app walkthrough requirements, acceptance criteria, open questions
terminal workflow command sequence, failure point, debugging prompt
long recording chapters, exercises, resources, follow-up packet
UI alternatives Section 6 variation matrix

Bug prompt shape:

# Implementation Prompt From Video

## Source

- video:
- source hash:
- ffprobe:
- analysis:
- keyframes:

## Observed Failure

- timestamp:
- visual evidence:
- transcript/chat evidence:
- expected behavior:
- actual behavior:

## Request

Implement the smallest safe fix.

## Constraints

- do not change:
- preserve:

## Verification

- command:
- what passing means:

Variation seed shape:

# Video-Derived Variation Matrix

## Goal

## Evidence From Video

- timestamp:
- observation:
- keyframe:
- why it matters:

## Invariants

- same source evidence:
- same target flow:
- same verification command:
- same review criteria:

## Variable Under Test

- variable:
- why this variable matters:
- what is out of scope:

Ask The Agent This

Analyze a bug reproduction:

Use the video analysis workflow.

Preserve the source video, save a source hash and ffprobe output, write a Gemini
prompt with END_OF_VIDEO_ANALYSIS, analyze the video through Oracle browser mode
with Gemini, then separate transcript evidence, visual evidence, chat evidence,
and inference. Turn the verified result into repro steps and an implementation
prompt for the smallest safe fix.

Review design feedback:

Turn this design feedback video into a change brief.

Keep visible UI facts separate from spoken opinions. Save keyframes for the
visual claims. Produce acceptance criteria, a QA checklist, and open questions.
Do not invent UI labels or routes that are not visible or spoken.

Create a Section 6 seed:

Use this verified video analysis to create a controlled variation matrix.

Choose one variable under test, preserve the same source evidence and
verification command for every candidate, and write three candidate settings.

What To Confirm / How To Know It Worked

Goal Confirmation
source preserved original media exists under inputs/
source hash saved source-video.sha256 exists and is non-empty
media facts saved ffprobe.source.json exists
prompt saved chunks/01/gemini-prompt.md exists
sentinel required expected-sentinel.txt exists
Oracle call shape correct engine: "browser", model: "gemini-3-pro", browserAttachments: "always"
output inspected ~/.oracle/sessions/<slug>/output.log read
sentinel present END_OF_VIDEO_ANALYSIS found in output
lanes separated transcript, chat, visual evidence, inference are separate
visual claims checked keyframes/contact sheet exists
unsupported claims removed no invented URLs/tools/commands/files
downstream artifact saved repro, requirements, QA, brief, lesson, or matrix exists

If It Goes Wrong

No Source Hash

Create it before analysis:

shasum -a 256 "$RUN_DIR/inputs/source-video.mp4" \
  > "$RUN_DIR/inputs/source-video.sha256"

No ffprobe Output

Run:

ffprobe -hide_banner -show_format -show_streams -of json \
  "$RUN_DIR/inputs/source-video.mp4" \
  > "$RUN_DIR/receipts/ffprobe.source.json"

Oracle Call Used API Mode

Rerun with browser mode.

engine: "browser"
model: "gemini-3-pro"
browserAttachments: "always"

File Path Was Relative

Use an absolute video path in files.

realpath "$RUN_DIR/chunks/01/chunk.mp4"

Sentinel Missing

Read output.log. If the sentinel is truly missing, rerun or ask for an omissions pass.

rg -n "END_OF_VIDEO_ANALYSIS" ~/.oracle/sessions/<slug>/output.log

Analysis Only Summarizes Transcript

Run an omissions pass focused on visual evidence:

Find missing visual UI states, cursor actions, timing, browser/terminal state,
and unsupported claims. Preserve timestamps.

Visual Claim Has No Keyframe

Extract keyframes or create a contact sheet. Downgrade unsupported claims to inference.

URL, Tool, Command, Or File Appears Without Evidence

Remove it or label it as inference. Do not let invented details become prompt constraints.

Long Recording Was Not Chunked

Create chunks and analyze them separately. Preserve original timestamps across chunks.

References And Resources

Source references