Runbook — exact commands, stage by stage
Follow SPEC.md for what and why; this page is how.
Conventions. <id> = the video folder name, e.g. Video-Zen3. Run everything from inside the video folder (cd /Users/justin/Code/Project-V/<id>): shared scripts are ../scripts/…, Python is ../.venv/bin/python. Small variables: PY=../.venv/bin/python, SC=../scripts.
Legend: verified ran on the reference run · UNVERIFIED not run yet.
0. Preflight and new video folder
ls /Volumes/SSD-4T-LR/AI/Models | head; df -h . | tail -1 # SSD mounted, disk free
ollama ps; pgrep -fl "DrawThings|acestep|dt_generate" # nothing heavy running
cd /Users/justin/Code/Project-V && mkdir -p <id>/{input,jobs/<id>,output,work} && cd <id>
Folder naming: Video-<Shortname><n> (SPEC §3). To use the CLI ask the user to quit the Draw Things app.
1. Scene description → image verified → Approval 1
First, ask the user for the scene description (SPEC 1.0): the scene in their words (a reference image is welcome), what should move/glow/float, the music mood and ambience sound, and a folder name. If they give only a line, expand it into a 5–8 line scene brief and get a yes/change before going on. Save it:
cat > jobs/<id>/scene.md # the user's words + the confirmed brief (setting, light, colours, moving elements by zone, sound)
Then:
# write jobs/<id>/image_prompt.txt and image_negative.txt (templates: SPEC §9)
$SC/image_candidates.sh <id> # 4 candidates, seeds 11/22/33/44, 1920x1088, ~2-3 min
# -> work/<id>/candidates/cand1_s11.png … cand4_s44.png + work/<id>/candidates/sheet.png (numbered 2x2)
$SC/image_candidates.sh <id> 4 55 # another round (seeds 55/66/77/88) after prompt changes
Screen each candidate (SPEC 1.1), send sheet.png (SendUserFile) with one line per image, wait for the pick. Then intake:
cp work/<id>/candidates/cand<k>_s<seed>.png input/<id>.png
cp ../Video-<Previous>/jobs/<prev-id>/job.json jobs/<id>/job.json && cp ../Video-<Previous>/jobs/<prev-id>/prompts.md jobs/<id>/ # any earlier video's job as the starting template
# edit job.json: id, source "input/<id>.png", source_size [1920,1088], plate.crop [0,4,1920,1084], regions (stage 2)
The script runs draw-things-cli generate --model z_image_turbo_1.0_q8p.ckpt --no-download-missing --width 1920 --height 1088 --seed S --prompt-file … --negative-prompt-file … with DRAWTHINGS_MODELS_DIR=/Volumes/SSD-4T-LR/AI/Models, skips candidates that exist, and draws the sheet with Pillow (this ffmpeg has no drawtext).
Own image instead? copy it to input/<id>.<ext> and set source, source_size and a 16:9 plate.crop.
2. What animates → masks verified → Approval 2
Edit the polygons in jobs/<id>/job.json (source pixels), then look:
$PY $SC/mask_preview.py --job <id> --zoom x0 y0 x1 y1 # jobs/<id>/mask_overlay_preview.png (+ _zoom.png)
Open the PNGs; repeat until edges sit on real edges; save the final one as jobs/<id>/mask_overlay_v0.png. Send it to the user with the plain-words list "Moves: … Frozen: everything else." and wait for approval.
3. Audio sample → Approval 3 verified (see also §M)
Start the music server (§M), then three 30 s styles, each checked, softened, levelled:
for spec in "a_flute:21:<flute+pad prompt>" "b_piano:32:<piano prompt>" "c_koto:33:<koto prompt>"; do
IFS=: read tag seed prompt <<< "$spec"
$PY $SC/music_gen.py --tag s_$tag --seed $seed --dur 30 --bpm 46 --key "D minor" --prompt "$prompt"
ffmpeg -y -i work/music/s_$tag.flac -af "lowpass=f=6500,highshelf=f=3000:g=-3,loudnorm=I=-22:TP=-3:LRA=7,afade=t=in:d=2,afade=t=out:st=28:d=2" -c:a libmp3lame -b:a 192k work/music/s_$tag.mp3
$PY $SC/audio_qc.py work/music/s_$tag.mp3 --out work/music/qc_$tag | grep -E "harsh_events|pass" # must be pass: true
done
(Prompts: SPEC §9 and Music. Regenerate any sample that fails the check.) Send the MP3s to the user; after the pick, the ambience sample:
$PY $SC/ambience.py --dur 30 --seed 5 --out work/music/amb_30.wav
ffmpeg -y -i work/music/s_<pick>.flac -i work/music/amb_30.wav -filter_complex "[0:a]lowpass=f=6500,highshelf=f=3000:g=-3,loudnorm=I=-22:TP=-3:LRA=7[m];[1:a]loudnorm=I=-36:TP=-9:LRA=7[a];[m][a]amix=inputs=2:normalize=0:duration=first[x];[x]afade=t=in:d=2,afade=t=out:st=28:d=2" -c:a libmp3lame -b:a 192k work/music/mixed_30.mp3
(loudnorm true-peak must be −9 or higher.) Also send amb_30.wav as an MP3 at +14 dB louder so the user can judge the ambience alone. Save the approved prompt/BPM/keys to jobs/<id>/music.md.
4. Video prompt
Edit jobs/<id>/prompts.md. dt_generate.py reads the section whose header starts with the key (--pkey P5 → ## P5 — …); the first fenced block is the positive prompt, the second the negative. Template: SPEC §9.
5. Drift probe (3 seeds) [verified with the API]
for s in 101 202 303; do
$PY $SC/dt_generate.py --job <id> --round r01 --tag t_s$s --seed $s --w 1024 --h 576 --frames 121 --pkey P5
$PY $SC/qc_drift.py work/<id>/r01/t_s$s/frames 30 # want worst_drift_px <= 0.7
$PY $SC/qc_motion.py work/<id>/r01/t_s$s/frames # moving_down/up %, activity
done
ffmpeg -framerate 24 -i work/<id>/r01/t_s101/frames/f%04d.png -vf "select='not(mod(n\,10))',scale=512:-1,tile=4x3" -frames:v 1 work/<id>/r01/t_s101/sheet_full.png
For crops use crop=W:H:X:Y before scale (frame coordinates = (src_x − crop_x0) × 1024/(crop_x1 − crop_x0)).
§Q — editing the QC scripts for a new image
qc_motion.py has a region table R = {"snow": (x0,y0,x1,y1), …} in 1024×576 frame pixels; convert each qc_only polygon's bounding box with the formula above; rename/add entries for the new effects. qc_drift.py tracks features in the left 18 % and right 32 % of the frame (the room on the reference run); this is hardcoded and wrong for any other image: change the two mask lines at the top to the job's locked, textured areas (never areas with rain, bamboo, steam or fire; Lessons A14) and cross-check 2–3 separate locked objects, which should agree. If every seed fails by a wide margin, see SPEC §10 (stabilize fallback, Lessons A15).
6. Two clips [verified with the API]
$PY $SC/dt_generate.py --job <id> --round r02 --tag segA_s101 --seed 101 --w 1024 --h 576 --frames 145 --pkey P5
$PY $SC/dt_generate.py --job <id> --round r02 --tag segB_s202 --seed 202 --w 1024 --h 576 --frames 145 --pkey P5
Two best seeds from stage 5; ~3 min each; frames in work/<id>/r02/<tag>/frames/f0000.png ….
§G-CLI — the same clip with draw-things-cli UNVERIFIED
# plate at generation size (16:9 crop of the source, Lanczos to 1024x576) -> work/<id>/plate_1024.png ; positive prompt -> work/<id>/prompt.txt
DRAWTHINGS_MODELS_DIR=/Volumes/SSD-4T-LR/AI/Models draw-things-cli generate \
--model ltx_2.3_22b_distilled_1.1_q8p.ckpt --no-download-missing \
--image work/<id>/plate_1024.png --prompt-file work/<id>/prompt.txt \
--frames 145 --width 1024 --height 576 --seed 101 \
--output work/<id>/r02/segA_s101.mov --video-format prores422hq
mkdir -p work/<id>/r02/segA_s101/frames && ffmpeg -i work/<id>/r02/segA_s101.mov -start_number 0 work/<id>/r02/segA_s101/frames/f%04d.png
The CLI's --image does "aspect-preserving scale and center crop", so pass an already cropped 1024×576 image. Check frame count = 145, size, drift ≤ 0.7 px, and compare steps/sampler/shift/CFG with the API path (--steps 8 --cfg 1; other keys via --config-json). Record differences in Lessons.
§G-API — fallback (what the reference run used)
dt_generate.py posts to POST http://127.0.0.1:7860/sdapi/v1/img2img (Draw Things app, Settings → API Server on). Only a whitelist of config keys is sent; width/height multiples of 64; frames 8n+1 and ≤ 201; the request's model key picks the model.
7. Loop → Review 1 verified
$PY $SC/loop_build.py --job <id> --a work/<id>/r02/segA_s101/frames --b work/<id>/r02/segB_s202/frames --out work/<id>/loop
cat work/<id>/loop/qc_final.json
Writes loop_frames/, plate.png, mask.png, <id>_loop10s_1080p24.mp4, a 60 s preview, qc_final.json, sheet_seam.png. Gates: seam_ratio ≤ 1.3, locked_pixels_identical: true, luma_std_detrended ≤ 1.0, luma_max_jump ≤ 1.5. Claude reads sheet_seam.png and a full frame, then sends the loop (and the 60 s preview) for Review 1.
7b. 4K master (optional) → Review 1b UNVERIFIED
Do this only after Review 1 approved the 1080p loop. It upscales the 240 loop frames once; the hour is the 4K loop repeated. Details and gates: SPEC stage 7b.
mkdir -p work/<id>/loop4k
# 1) upscale every loop frame to 3840x2160 (Lanczos + light sharpen; swap for an AI upscaler if you have one)
ffmpeg -y -framerate 24 -i work/<id>/loop/loop_frames/f%04d.png \
-vf "scale=3840:2160:flags=lanczos,unsharp=5:5:0.5:5:5:0.0" \
-c:v libx265 -crf 16 -preset slow -pix_fmt yuv420p10le -tag:v hvc1 -g 240 -keyint_min 240 -sc_threshold 0 \
-movflags +faststart work/<id>/loop4k/<id>_loop10s_4k24.mp4
# 2) checks: 240 frames, 3840x2160
ffprobe -v error -select_streams v:0 -count_frames -show_entries stream=nb_read_frames,width,height -of default=nw=1 work/<id>/loop4k/<id>_loop10s_4k24.mp4
# 3) round trip: downscale the 4K loop and compare to the approved 1080p loop (want mean PSNR >= 35 dB)
ffmpeg -v error -i work/<id>/loop4k/<id>_loop10s_4k24.mp4 -i work/<id>/loop/<id>_loop10s_1080p24.mp4 \
-filter_complex "[0:v]scale=1920:1080:flags=lanczos[a];[a][1:v]psnr=stats_file=work/<id>/loop4k/psnr.log" -f null - 2>&1 | tail -2
# 4) three full-size crops for Claude to look at (animated region, locked region, mask edge): edit x,y
ffmpeg -y -v error -ss 3 -i work/<id>/loop4k/<id>_loop10s_4k24.mp4 -frames:v 1 -vf "crop=1280:720:1200:600" work/<id>/loop4k/crop_animated.png
# 5) 4K hour (same as §9, with the 4K loop); check size before upload
ffmpeg -y -stream_loop 359 -i work/<id>/loop4k/<id>_loop10s_4k24.mp4 -i work/<id>/music_60min.m4a \
-map 0:v -map 1:a -c copy -t 3600 -movflags +faststart output/<id>_1hr_4k24.mp4
ls -lh output/<id>_1hr_4k24.mp4
ffprobe -v error -select_streams v:0 -count_packets -show_entries stream=nb_read_packets -of csv=p=0 output/<id>_1hr_4k24.mp4 # 86400
cp work/<id>/loop4k/<id>_loop10s_4k24.mp4 output/<id>_loop10s_4k24_silent.mp4
The ffmpeg mechanics above (Lanczos+HEVC encode, frame count, PSNR check, -stream_loop copy) were tested on a synthetic 48-frame clip; the real quality and file size on an actual loop are not yet known.
If you use Draw Things for any upscale step, sizes must be multiples of 64: use 3840×2176 and crop crop=3840:2160:0:8. If HEVC does not stream-copy cleanly 360 times or your uploader dislikes it, use libx264 -crf 15 instead.
8. Full hour of audio → Review 2 (§M has server start/stop)
$PY $SC/music_build.py --minutes 60 --out work/<id>/music_60min \
--prompt "<approved prompt>" --keys "D minor,F major,A minor,C major,G minor" --bpms "44,40,46,42" # ~20 min; resumable
What it does: per piece generate → soften → audio_qc → retake (new seed) up to 4×; crossfade; slow leveler; ambience; fades; AAC; final audio_qc of the finished hour and the listening pack. Watch the log for harsh events N -> retry and WARNING ... least-bad take. Outputs: work/<id>/music_60min.{wav,m4a}, music_60min_pieces_qc.json, music_60min_qc/audio_qc.json, music_60min_qc/listening_pack.mp3 + listening_pack.txt.
cat work/<id>/music_60min_qc/audio_qc.json | head -30 # harsh_events must be 0, pass: true
Send listening_pack.mp3 (with the timestamp list) to the user for Review 2. If the user flags pack item k, note its timestamp t, find the piece (t ÷ 142 s), delete that piece's file(s) in work/music/<work>/, rerun (resumable), re-check, resend. To tune the check, edit LIMITS in ../scripts/audio_qc.py; single files: $PY $SC/audio_qc.py <file> --excerpts 6 (clips of the worst moments) or --pack.
9. Mux, deliver, clean up
ffmpeg -y -stream_loop 359 -i work/<id>/loop/<id>_loop10s_1080p24.mp4 -i work/<id>/music_60min.m4a \
-map 0:v -map 1:a -c copy -t 3600 -movflags +faststart output/<id>_1hr_1080p24.mp4
ffprobe -v error -show_entries format=duration,size -of default=nw=1 output/<id>_1hr_1080p24.mp4 # 3600.000000
ffprobe -v error -select_streams v:0 -count_packets -show_entries stream=nb_read_packets -of csv=p=0 output/<id>_1hr_1080p24.mp4 # 86400
cp work/<id>/music_60min.m4a output/<id>_audio_60min.m4a
cp work/<id>/loop/<id>_loop10s_1080p24.mp4 output/<id>_loop10s_1080p24_silent.mp4
ffmpeg -y -i work/<id>/music_60min_fixed.wav -c:a aac -b:a 256k output/<id>_music_60min.m4a # KEEP the stems (SPEC 1.2): music alone ...
ffmpeg -y -i work/<id>/amb_61min.wav -t 3600 -c:a aac -b:a 192k output/<id>_ambience_60min.m4a # ... and ambience alone, so the mix can be changed later
rm -rf work # scratch only; the stems, the loop and the final files in output/ stay
pkill -f start_api_server_macos; pkill -f acestep-api # stop the music server
cd .. && .venv/bin/python scripts/build_docs.py # rebuild wiki/html
Then fill <id>/RUNLOG.md, update Lessons, commit and push (git add -A && git commit && git push from Project-V/).
M. Music server
cd /Users/justin/Code/Project-V/tools/ACE-Step-1.5 && ACESTEP_CHECKPOINTS_DIR=/Volumes/SSD-4T-LR/AI/Models/ACE-Step nohup ./start_api_server_macos.sh > /tmp/acestep_server.log 2>&1 &
# ready when: curl -s http://127.0.0.1:8001/health returns 200 (music_build.py also restarts it by itself if it dies)
R. Rain layer and rain sound (Video-Zen4; approved settings, see SPEC §1.4b)
# 1) depth map of the plate (once per picture; 0 = near .. 1 = far), Depth Anything V2 Small in /Volumes/SSD-4T-LR/AI/Models/Depth-Anything-V2-Small
../tools/moss-env/bin/python <one-off script that saves work/<id>/depth.npy from input/<id>.png cropped to the plate>
# 2) splash surfaces: regions with "role": "splash" and "kind": ground | top | rail (with "line" and "width") | leaf, "n" events per loop, in jobs/<id>/job.json
# 3) procedural steam / flicker / rain on top of the finished loop frames (each step reads the previous step's frames)
../.venv/bin/python ../scripts/flicker.py --job <id> --frames work/<id>/loop --out work/<id>/loop_flicker
../.venv/bin/python ../scripts/rain_overlay.py --job <id> --frames work/<id>/loop_flicker --out work/<id>/loop_rain --strength 0.9 --splash 0.96 --size 0.5 --light 0.5
ffmpeg -framerate 24 -i work/<id>/loop_rain/f%04d.png -c:v libx264 -crf 19 -preset slow -pix_fmt yuv420p -g 240 -keyint_min 240 -sc_threshold 0 -movflags +faststart work/<id>/loop.mp4
# 4) rain sound: ~50 clips of 30 s, then the hour
../tools/moss-env/bin/python ../scripts/rain_gen.py --jobs jobs/<id>/rain_jobs.json --out work/rain_bed # [{"tag","prompt","dur":30,"seed"}, ...]
../.venv/bin/python ../scripts/rain_bed.py --clips work/rain_bed --ref <approved clip tag> --minutes 61 --out work/<id>/rain_bed_61min.wav
../.venv/bin/python ../scripts/music_build.py --minutes 60 --out work/<id>/music_60min --amb_wav work/<id>/rain_bed_61min.wav --amb_lufs -36 ...
# 5) quiet dips in the finished piano: measure the 1 s envelope, then lift the tails and let the user hear it
../.venv/bin/python ../scripts/upcomp.py in.wav out.wav --max_up 9 --ratio 0.8 --release 1.6 # test B; C = --max_up 12 --ratio 0.95 --release 1.2
rain_overlay.py options: --strength, --density (streaks), --splash (events), --size (drop size), --light (how white), --slant (0 = straight down), --refract, --only N (test one frame). Scripts: rain_overlay.py, rain_gen.py, rain_bed.py, flicker.py, steam_overlay.py, upcomp.py, stabilize.py.
S. Procedural loop from a still: snow, lamp flicker, sway, shimmer (approved snow v5, 2026-10-17)
No AI video: everything is drawn by scripts/still_loop.py from jobs/<id>/job.json and the depth map. The approved snow is the script's built-in default; do not change the LAYERS table or the snow code in still_loop.py. Tune a single video only with the snow block in its job.json.
/Users/Maxi/Code/Tools/moss-env/bin/python ../scripts/depth_map.py --job <id> # work/<id>/depth.npy (needs plate.crop in job.json)
../.venv/bin/python ../scripts/still_loop.py --job <id> --only 300 --png work/<id>/t.png # one frame to look at
../.venv/bin/python ../scripts/still_loop.py --job <id> --out work/<id>/preview_loop60.mp4 # 1080p 60 s, ~2 min
../.venv/bin/python ../scripts/still_loop.py --job <id> --size 3840 2160 --out work/<id>/loop60_4k.mp4 # 4K
Approved job.json block (The First Tram and Corner Store): "loop": {"period_s": 60.0, "fps": 24}, "snow": {"seed": 11, "density": 1.0, "size": 1.0, "strength": 1.0, "slant": 0.12} (white_share 0.22 and sharp_share 0.3 are the defaults).
Built-in snow layers (near → far; n flakes, fall speed px/s @1080p, median radius px, depth blur px, alpha, wobble px): z0.22 130 63–133 2.1 1.2 0.70 16 · z0.42 330 45–98 1.5 0.8 0.75 11 · z0.62 420 31–70 1.1 0.6 0.75 7 · z0.80 650 21–49 0.8 0.5 0.70 5 · z0.93 800 14–35 0.6 0.4 0.60 3. Sizes lognormal σ0.55 clipped 0.35–2.4× (max 4.5 px); 22 % near-white, 30 % crisp (blur 0.35 px); gust 18 px; per-flake slant 0.3–1.7× of slant.
Lamp flicker: not approved, dropped by the user. Set "flicker": {"mode": "off"} in job.json (no lamp changes at all). The experimental modes relight_all / bad_bulb are still in still_loop.py but the user rejected their results (Lessons S6-S13); do not use them without a new idea and an approved 4K on/off still.
Other regions: sway (poly, root, reach, amp_px, exclude), shimmer (poly, amp), lock (documentation/overlay only). Seam check on raw frames: render --only 0 and --only <P-1>, mean |diff| must equal a normal frame step (~0.15/255).
S0. Whole workflow: a procedural loop video from the user's own image (The First Tram and Corner Store, 2026-10-17; total about 7 h with feedback rounds)
Use this when everything that moves is weather or light. Order and what each step cost:
1. Intake (10 min). Copy the user's image into input/<id>.png (never touch the original). Brand check: if a sign or stripe copies a real chain, recolour it in the copy only (HSV hue shift masked to the sign area, see jobs/<id>/recolor_stripes.py of the Tram video: green → teal, orange → amber) and tell the user. Folder name = what the user says (Video-<Full-Title>).
2. Plate and depth (3 min). job.json: plate.crop to 16:9 of the source (3840×2112 → crop [42,0,3797,2112]), then depth_map.py → work/<id>/depth.npy.
3. Questions first, in one message (3 min). Moving parts, what stays still (tram, traffic lights), loop length (60 s so periodic things never repeat every 10 s), extras ("as long as it is natural"), brand recolour, audio, 4K yes/no. Defaults in bold; the user answers by number.
4. Overlay + preview loop (1 h, then feedback rounds). jobs/<id>/mask_overlay.py, then a 60 s 1080p preview (~5 min per render, 0.06 s per frame; 4K ~9 min). Send every version as an mp4; judge motion, never stills. Snow took 5 rounds (v1–v5, Lessons S1–S5). Freeze each approved part: after every change render one old frame (--only 500) and compare with the saved reference, it must be 0 changed pixels.
5. Audio samples (1 h). 3 piano styles × 30 s (generate 40 s, cut 30 s) + ambience sample; all must pass audio_qc.py; send MP3s; the user may ask for another batch.
6. Final loops, hours (2 h). 1080p CRF 18 and 4K CRF 16 loops; piano hour music_build.py (26 pieces, ~50 min), upcomp.py, local softening of single harsh spots, MOSS wind bed + amb_layers.py (hum + distant bell), mix, audio_qc.py, listening pack, then mux with -stream_loop 59 ... -c copy (1 h 1080p ≈ 970 MB, 4K ≈ 2.7 GB) and decode-check at 30:00 and the last 10 s.
7. Review and save. Send loop + listening pack, wait for OK; keep the stems (SPEC 1.2), delete scratch, commit only your own files (§G).
Do not offer lamp flicker on a painted still (SPEC 1.4c: 12 attempts rejected). If the user asks for it, get an on/off still approved at 4K first.
G. Saving when the working tree holds other people's edits (done on 2026-10-17; the repo had ~46 uncommitted edits by the user)
Never git add -A and never commit pre-existing edits. Recipe:
# 1) new files of this video + scripts you wrote: git add <paths>
# 2) shared files with the user's own edits (wiki/SPEC.md, runbook.md, lessons.md): stage only YOUR hunks
python3 scripts/stage_hunks.py wiki/SPEC.md wiki/runbook.md wiki/lessons.md --skip 'justin|Volumes/SSD-4T|SSD' # skips hunks that remove the user's old paths (their rewrite); --dry-run first
# 3) wiki/html/*: rebuild from a clean worktree so only your changes show up
git worktree add --detach /tmp/pv_wt HEAD && git diff --cached --binary | (cd /tmp/pv_wt && git apply) && (cd /tmp/pv_wt && ../path/.venv/bin/python scripts/build_docs.py)
# then git hash-object -w + git update-index --add --cacheinfo for each changed html file; git worktree remove --force /tmp/pv_wt
# 4) git commit (with the attribution line), git push origin main; if origin moved: cherry-pick your commit onto origin/main in a temp worktree and push
git config core.fileMode false avoids the 755/644 mode noise. Check git diff --cached | grep -ci justin = 0 before committing.
Y. Upload to YouTube (private)
See YouTube uploader. Until the API project passes Google's audit, API uploads are locked private, so upload by hand in YouTube Studio with the text from jobs/<id>/youtube.md; after approval:
/Users/Maxi/Code/Tools/yt-venv/bin/python ../scripts/yt_upload.py --file output/<id>_1hr_4k24.mp4 --meta jobs/<id>/youtube.md
Swap. Put different audio on the picture later
ffmpeg -y -stream_loop 359 -i output/<id>_loop10s_1080p24_silent.mp4 -i <new_audio>.m4a -map 0:v -map 1:a -c copy -t 3600 -movflags +faststart output/<id>_1hr_v2.mp4
For another length change 359 (loops − 1) and -t (seconds); the audio must be at least that long.