Frame selection is the whole game: notes from making LLMs watch video
Summary
Frame selection determines whether an LLM watching video truly perceives events or just sees a slideshow. The notes describe an MIT-licensed open-source pipeline called claude-real-video, outlining a multi-channel frame deduplication approach (global RGB change, action, settled) along with scene-detection and transcript fusion. It also covers serving the pipeline via MCP and practical steps to run it locally for on-device video understanding.