vibe-coding-课程-干货集训
趣讲论文视频生成
生成示例
配置清单
- 需完成以下配置，才能使用本Skill
| # | 项目 | 说明 |
| --- | --- | --- |
| 1 |
MINIMAX_API_KEY环境变量 | MiniMax TTS 语音合成 API 密钥 | | 2 |DOUBAO_API_KEY环境变量 | 豆包 SeedDream 生图 API 密钥 | | 3 | ffmpeg / ffprobe | macOS:brew install ffmpeg| | 4 | Python 3.10+ | — | | 5 | requests 库 |pip install requests| | 6 | 字幕字体 | macOS 自带Source Han Sans CN；Linux 需apt install fonts-noto-cjk| - 配置步骤
- 打开终端，依次配置以下指令。minimax key和豆包 key需要通过对应开放平台获取，豆包用的是seedream 4.5的生图api
```bash# 配置步骤brew install ffmpegpip install requestsexport MINIMAX_API_KEY="your-minimax-key"export DOUBAO_API_KEY="your-doubao-key"```
Skill文件夹结构
paper-to-video-skill/├── SKILL.md # 主文档：完整工作流定义├── scripts/ # 可执行脚本│ ├── run_pipeline.py # 一键全流程运行│ ├── run_stage.py # 按阶段运行（validate/tts/subtitle/render）│ ├── generate_images.py # Doubao SeedDream 批量生图│ ├── run_tts_by_clip.py # MiniMax TTS 音频生成│ ├── generate_subtitles.py # ASS 字幕生成（从旁白原文）│ └── assemble_and_merge.sh # ffmpeg 渲染合成└── references/ # 参考文档 ├── script-principles.md # 脚本写作原则 ├── schema.md # 数据结构定义 └── quality-checklist.md # 质量检查清单
文件具体内容
SKILL.md
⚠️注意：文档末尾「已验证的范例」部分包含本地路径，需替换为你自己的项目路径。
---name: paper-to-video-skilldescription: Turn an arXiv or technical paper into a short explainer video workflow. Use when the user wants to turn a paper into a narrated video, continue an existing paper-video project, or improve the script, visuals, subtitles, voiceover, or rendering pipeline for a paper explainer video.**---****# Paper to Video**把一篇技术论文变成一条 ****16:9、60-120秒、传播型**** 科技短视频。**## 核心原则**这不是论文摘要。这是叙事压缩。最终产物应该让一个聪明但非专业的观众在一分钟内理解论文的核心贡献。**## 项目目录结构（必须严格遵守）**<project>/├── assets/│ ├── images/ # shot01.png, shot02.png, ... (2K分辨率, 16:9)│ └── audio/│ ├── shot01.wav ... shot10.wav # 每个shot的独立音频│ ├── voice.wav # 完整旁白音频（含shot间隔+尾部留白）│ ├── timing.json # 每个shot的真实起止时间│ └── music.mp3 # 背景音乐（可选）├── prompts/│ ├── style_guide.md # 视觉风格指南│ ├── image_prompts.json # 每个 shot 的图片 prompt│ └── generate_images_doubao.py # 图片批量生成脚本├── render/│ ├── check_assets.py # 渲染前预检│ ├── render.sh # 渲染脚本│ ├── subtitles.ass # ASS 字幕文件│ ├── clips/ # 渲染中间产物│ ├── merged.mp4│ ├── with_audio.mp4│ └── final_16x9.mp4 # 最终输出└── script/ ├── paper_summary.json # 论文理解 ├── storyboard.json # 分镜脚本（核心文件） └── narration_v2.txt # 旁白文稿**## 第一步：检查项目状态**如果是继续已有项目，先检查哪些文件已存在，避免重复生成。如果是新项目，从 Step 1 开始。**## Step 1: 理解论文******输入****：论文 URL / arXiv ID / PDF****输出****：`script/paper_summary.json`{ "title": "论文标题", "authors": "作者", "url": "https://arxiv.org/abs/...", "core_problem": "论文攻击的核心问题（用一句话说清楚）", "method_summary": "方法概要（用大白话，不用术语）", "key_results": ["结果1", "结果2"], "key_context": "为什么这篇论文重要", "analogy_seeds": ["比喻候选1", "比喻候选2"]}****规则****：不要在没理解论文的冲突和主张之前就开始写脚本。**## Step 2: 确定叙事角度**回答这四个问题：1. 旧方法是什么？2. 旧方法哪里坏了？3. 新方法改了什么？4. 为什么这很重要或令人意外？****输出****：- 一句话视频前提- 工作标题- Hook（开场钩子）- Punchline（结尾判断，不是总结）**## Step 3: 写分镜脚本******输出****：`script/storyboard.json`默认 8-10 个 shot。每个 shot 的 schema：{ "shot": 1, "start": 0.0, "end": 6.0, "duration": 6.0, "voice": "完整旁白文本（TTS会读这段，字幕也从这里生成）", "image": "shot01.png", "image_prompt": "详细的图片生成 prompt（中文）", "motion": "zoom_in", "transition_in": "fade", "transition_out": "cut"}****⚠️ 关键变更****：不再有单独的 `subtitle` 字段。字幕直接从 `voice` 文本自动切分生成，确保音画同步。**### 叙事结构参考（10 shot）**| Shot | 功能 | 说明 ||------|------|------|| 1 | hook | 抛出问题或反直觉事实 || 2 | hook_payoff | 展开钩子，给出具体数据 || 3 | problem_surface | 表面问题是什么 || 4 | problem_root | 根本原因是什么 || 5 | core_conflict | 核心矛盾 || 6 | solution_step1 | 解法第一步 || 7 | solution_step2 | 解法第二步 || 8 | result | 关键结果 || 9 | impact | 影响力 || 10 | punchline | 结尾判断 |**### 脚本写作硬规则**- 短句。口语化。- 先给直觉，再给术语。- 每个 shot 只推进一个认知。- 结尾给判断，不给总结。- 每个 shot 至少有一句值得记住的话。同时生成 `script/narration_v2.txt`（纯文本旁白，每段一行，空行分隔）。**### Motion 可选值**`zoom_in` / `zoom_out` / `pan_left` / `pan_right`运动方向按 shot 循环：zoom_in → pan_right → zoom_in → pan_left → zoom_out**### Transition 可选值**`fade` / `cut`**## Step 4: 设计视觉系统******这是最关键的一步。不要跳过。******### 4a. 图片 System Prompt（所有图共用）******必须使用以下工程草图风格 system prompt****（来自验证过的手册）：你是一位技术插画师，正在为一份高端技术报告绘制概念图解。画面风格像是资深工程师在方格纸笔记本上画的技术速写。【画风】工程手稿速写风格，像是用0.5mm黑色针管笔在浅灰方格纸上画的。线条特征：精确、干净、有建筑制图的美感，直线用尺画（微微不完美），曲线自信流畅。线条粗细差异小（0.3-0.8mm），不像马克笔那么粗犷。人物风格：极简概念人，用几何形状组合——圆形头、梯形身体、线条四肢。比例偏真实（非大头），像工程图纸里的人形比例参考。人物需有清晰可辨的表情（用点和线表达），姿态语言明确夸张。填色：主体为黑色线稿。用浅灰色（#E0E0E0）做轻微的阴影和体积暗示。标注和强调用红色（#D94040）——红色只用于标注文字、圈出重点、画警示符号。背景：带极淡方格纹理的白底，方格线用 #F0F0F0 极浅灰。有偶尔的铅笔底稿痕迹（浅灰虚线）。【版式】画幅比例 16:9 横版。画面底部 20% 完全空白，不放置任何元素（包括文字标注），预留给后期字幕。所有文字标注放在画面中部或中上部。构图水平展开，元素对齐感强。文字标注为简体中文，工整手写印刷体。【系列一致性】这是系列图，需保持风格、配色、人物造型完全一致。每张图左上角有 Fig.N 编号和虚线框标题。**### 4b. 写图片 Prompt******输出****：`prompts/image_prompts.json`每个 shot 一条记录，用****中文****写场景描述：[ { "shot": 1, "filename": "shot01.png", "prompt": "画面主题：...（详细描述元素位置、大小比例、表情姿态、颜色标注）" }]**#### Prompt 写作规则**- 每个 prompt 用中文自然语言描述画面内容- 必须包含：元素位置、构图比例、颜色标注、人物表情姿态- 必须包含：左上角 Fig.N 编号、虚线框标题- 标注文字用中英双语（中文为主，关键术语附英文）- 指明哪些元素用红色强调**### 4c. 图片生成******输出****：`prompts/generate_images_doubao.py`使用 Doubao SeedDream API：- ****完整 prompt**** = System Prompt + "\n\n" + 场景描述 + " IMG_2094.CR2"- ****size****: `"2K"`（生成 2848×1600，原生 16:9）- ****watermark****: `False`（不加水印）- 跳过已存在的图片- 失败自动重试（最多 3 次）- 输出到 `assets/images/`**## Step 5: 生成音频**使用 MiniMax TTS 生成旁白音频。****输入****：`script/storyboard.json` 中每个 shot 的 `voice` 字段****输出****：`assets/audio/shot01.wav` ... `assets/audio/shot10.wav` + `assets/audio/voice.wav` + `assets/audio/timing.json`**### TTS 配置**{ "model": "speech-2.8-hd", "voice_setting": { "voice_id": "male-qn-jingying-jingpin", "speed": 0.95, "vol": 1, "pitch": 0 }, "audio_setting": { "sample_rate": 32000, "bitrate": 128000, "format": "mp3", "channel": 2 }, "language_boost": "Chinese"}**### 生成流程**1. ****每个 shot 整段调用一次 TTS****（不要按句拆分，当前模型整段效果更好）2. 下载 tar → 解压 mp3 → 转换为 wav3. 记录每个 shot 的真实时长4. 各 shot 之间插入 ****0.5s 静音间隔****5. 最后一个 shot 后追加 ****1.5s 尾部留白****6. 拼接为 `voice.wav`7. 保存 `timing.json`（每个 shot 的 start/end/duration）**### 音色选择**正式生成前，用第一个 shot 的文本试 3-4 个音色，让用户试听选择。推荐候选：| voice_id | 名称 | 风格 ||----------|------|------|| `male-qn-jingying-jingpin` | 精英青年-beta | 清晰专业，适合科普（默认） || `male-qn-jingying` | 精英青年 | 同上，非 beta 版 || `Chinese (Mandarin)_Radio_Host` | 电台男主播 | 会讲故事，节奏好 || `Chinese (Mandarin)_Sincere_Adult` | 真诚青年 | 真诚自然 |选定后锁定 voice_id，全片不换。**## Step 6: 生成字幕******⚠️ 关键规则：字幕必须从旁白原文生成，不能另写。******### 6a. 字幕切分逻辑**从每个 shot 的 `voice` 文本按自然句边界（。？！）切分，长句再按逗号/冒号二次切分，每行不超过 22 个汉字。**### 6b. 时间分配**按字数比例分配每个 shot 的音频时长给各行字幕。去掉末尾 0.3s 作为缓冲。**### 6c. ASS 字幕文件******输出****：`render/subtitles.ass`必须使用以下模板（注意字体为 `Source Han Sans CN`）：[Script Info]Title: 视频标题ScriptType: v4.00+PlayResX: 1920PlayResY: 1080ScaledBorderAndShadow: yesWrapStyle: 2[V4+ Styles]Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, EncodingStyle: Main,Source Han Sans CN,58,&H00FFFFFF,&H007DD3FC,&H00000000,&H80000000,1,0,0,0,100,100,1,0,1,3,1,2,120,120,56,1Style: Highlight,Source Han Sans CN,58,&H0080FFFF,&H007DD3FC,&H00000000,&H80000000,1,0,0,0,100,100,1,0,1,3,1,2,120,120,56,1****⚠️ 字体必须是 `Source Han Sans CN`（不是 SC），否则会乱码。******## Step 7: 渲染前检查**确认所有文件就绪：- 所有 shot 图片存在（`assets/images/shot01.png` ...）- `voice.wav` 存在- `timing.json` 存在- `subtitles.ass` 存在- storyboard.json 中每个 shot 的 duration 与 timing.json 一致**## Step 8: 渲染**运行 `render/render.sh`，流程：**### 8.1 Shot 视频生成******⚠️ 关键：zoompan 必须直接读取原始高分辨率图片，由 zoompan 的 `s=1920x1080` 输出缩放。不要预先 scale/pad 图片到 1920x1080 再 zoompan。****每个 shot 的视觉时长 = 音频时长 + 0.5s 间隔（最后一个 shot 加 1.5s 尾部留白）。Ken Burns 运动方向按 shot 循环：zoom_in → pan_right → zoom_in → pan_left → zoom_out → (循环)zoompan 滤镜参数：**# zoom_in**zoompan=z='min(zoom+0.0003,1.06)':x='iw/2-(iw/zoom/2)':y='ih/2-(ih/zoom/2)':d=${frames}:s=1920x1080:fps=25**# zoom_out**zoompan=z='if(eq(on,1),1.06,max(zoom-0.0003,1.0))':x='iw/2-(iw/zoom/2)':y='ih/2-(ih/zoom/2)':d=${frames}:s=1920x1080:fps=25**# pan_right**zoompan=z=1.04:x='if(eq(on,1),0,min(x+0.3,iw-iw/zoom))':y='ih/2-(ih/zoom/2)':d=${frames}:s=1920x1080:fps=25**# pan_left**zoompan=z=1.04:x='if(eq(on,1),iw*0.04,max(x-0.3,0))':y='ih/2-(ih/zoom/2)':d=${frames}:s=1920x1080:fps=25**### 8.2 拼接**所有 shot 视频用 concat 直接拼接（不用 xfade 转场）。**### 8.3 合成音频**merged.mp4 + voice.wav (+ 可选 music.mp3) → with_audio.mp4使用 `-shortest` 确保音视频同步。**### 8.4 烧录字幕**with_audio.mp4 + subtitles.ass → final_16x9.mp4**## Step 9: 质量检查**- 前 5 秒是否抓住注意力？- 视觉是否解释了语言无法单独表达的内容？- 全系列风格是否一致？- 字幕和音频是否同步？（内容一致 + 时间对齐）- 最后一句话是否完整播放？（有尾部留白）- 结尾是否给了判断而非复述？**## 已验证的范例**<!-- ⚠️ 替换为你自己的项目路径 -->最佳范例：`<your-path>/kimi-attnres-video/`实战验证项目：`<your-path>/2603.14473/`（论文：AI Can Learn Scientific Taste）**## 踩坑清单（实战验证）**| 坑 | 表现 | 解决方案 ||----|------|---------|| 字幕和旁白不同步 | subtitle 字段另写缩写，和 voice 说的不一样 | 字幕必须从 voice 原文切分，不能单独写 || 字体乱码 | `Source Han Sans SC` 在 macOS 上不存在 | 使用 `Source Han Sans CN` || 图片不覆盖画幅 | 预先 scale 到 1920x1080 再 zoompan | 直接用原始高分辨率图片，zoompan 内 `s=1920x1080` 输出 || 最后一句被截断 | 视频长度 < 音频长度 | 最后 shot 加 1.5s 尾部留白 || shot 间无间隔 | voice.wav 有 0.5s 间隔但视频无 | 每个 shot 视觉时长 = 音频时长 + 间隔时长 || AI 水印 | Doubao API 默认加水印 | `watermark: False` || 图片尺寸被拒 | Doubao 要求最小 3686400 像素 | 使用 `size: "2K"`（2848×1600） || TTS 语气不自然 | `audiobook_male_1` 念稿感强 | 换 `male-qn-jingying-jingpin`，先试听再生成 |**## 脚本参考**当需要写或修改脚本时，参考 `references/script-principles.md`。当需要设计图片 prompt 时，参考 `references/image-prompting.md`。```---
Scripts
- run_pipeline.py
```python#!/usr/bin/env python3"""One-click pipeline for paper-to-video generation.Runs all stages in order, with checkpoint resume support."""import jsonimport osimport subprocessimport sysfrom pathlib import PathWORKDIR = Path(os.environ.get("PAPER_VIDEO_WORKDIR", Path.cwd()))SCRIPTS_DIR = Path(__file__).parentSTAGES = [ ("images", "generate_images.py", "Generate images (if missing)"), ("tts", "run_tts_by_clip.py", "Generate TTS audio"), ("subtitles", "generate_subtitles.py", "Generate subtitles"), ("render", "assemble_and_merge.sh", "Render final video"),]STATE_FILE = WORKDIR / ".pipeline_state.json"def load_state(): if STATE_FILE.exists(): with open(STATE_FILE, 'r') as f: return json.load(f) return {"completed": [], "failed": None}def save_state(state): with open(STATE_FILE, 'w') as f: json.dump(state, f, indent=2)def run_stage(script_name: str, stage_name: str): script_path = SCRIPTS_DIR / script_name if script_name.endswith('.py'): cmd = [sys.executable, str(script_path)] else: cmd = ['bash', str(script_path)] print(f"\n{'='*50}") print(f"Stage: {stage_name}") print(f"{'='*50}\n") env = os.environ.copy() env['PAPER_VIDEO_WORKDIR'] = str(WORKDIR) result = subprocess.run(cmd, cwd=WORKDIR, env=env) return result.returncode == 0def main(): # Check prerequisites storyboard = WORKDIR / "script" / "storyboard.json" if not storyboard.exists(): print(f"Error: No script/storyboard.json found in {WORKDIR}") print("Generate storyboard first (Steps 1-3 in SKILL.md)") sys.exit(1) state = load_state() completed = set(state.get("completed", [])) print(f"Pipeline starting in: {WORKDIR}") if completed: print(f"Already completed: {list(completed)}") print(f"To restart from scratch, delete: {STATE_FILE}\n") try: for stage_id, script_name, description in STAGES: if stage_id in completed: print(f"[SKIP] {description} (already completed)") continue success = run_stage(script_name, description) if success: completed.add(stage_id) save_state({"completed": list(completed), "failed": None}) print(f"[OK] {description}") else: save_state({"completed": list(completed), "failed": stage_id}) print(f"[FAIL] {description}") print(f"\nPipeline failed at stage: {stage_id}") print(f"Fix the issue and rerun to resume from this stage.") sys.exit(1) print(f"\n{'='*50}") print("Pipeline completed successfully!") final_path = WORKDIR / "render" / "final_16x9.mp4" print(f"Final video: {final_path}") print(f"{'='*50}") if final_path.exists(): subprocess.run(["open", str(final_path)]) STATE_FILE.unlink(missing_ok=True) except KeyboardInterrupt: print("\n\nPipeline interrupted. Run again to resume.") save_state({"completed": list(completed), "failed": "interrupted"}) sys.exit(130)if __name__ == "__main__": main()```
- run_stage.py
```pythonimport argparseimport osimport subprocessimport sysfrom pathlib import PathSCRIPT_DIR = Path(__file__).resolve().parentSTAGES = { "validate": [sys.executable, str(SCRIPT_DIR / "validate_project.py")], "tts": [sys.executable, str(SCRIPT_DIR / "run_tts_by_clip.py")], "srt": [sys.executable, str(SCRIPT_DIR / "generate_srt.py")], "srt-sentence": [sys.executable, str(SCRIPT_DIR / "generate_srt_sentence_based.py")], "render": ["bash", str(SCRIPT_DIR / "assemble_and_merge.sh")],}PIPELINES = { "check": ["validate"], "audio": ["tts", "validate"], "subtitle": ["srt", "validate"], "subtitle-sentence": ["srt-sentence", "validate"], "render-all": ["render", "validate"], "post-script": ["tts", "srt", "render", "validate"], "post-script-sentence": ["tts", "srt-sentence", "render", "validate"],}def run_cmd(cmd, env): print("$", " ".join(cmd)) subprocess.run(cmd, check=True, env=env)def build_cmd(stage: str, workdir: Path): cmd = list(STAGES[stage]) if stage == "validate": cmd.append(str(workdir)) return cmddef main(): parser = argparse.ArgumentParser(description="Run paper-to-video stages from one entrypoint") parser.add_argument("stage", choices=sorted(list(STAGES.keys()) + list(PIPELINES.keys())), help="Stage or pipeline preset to run") parser.add_argument("--workdir", default=".", help="Project directory") parser.add_argument("--subtitle-mode", choices=["default", "sentence"], default="default", help="Used only with custom sequential runs") parser.add_argument("--no-validate", action="store_true", help="Skip the final validate step for custom stage chains") parser.add_argument("--stages", nargs="+", choices=sorted(STAGES.keys()), help="Custom sequence of concrete stages to run") args = parser.parse_args() workdir = Path(args.workdir).expanduser().resolve() if not workdir.exists(): raise SystemExit(f"Workdir does not exist: {workdir}") env = os.environ.copy() env["PAPER_VIDEO_WORKDIR"] = str(workdir) if args.stages: sequence = list(args.stages) if not args.no_validate and sequence[-1] != "validate": sequence.append("validate") elif args.stage in PIPELINES: sequence = PIPELINES[args.stage] else: sequence = [args.stage] if not args.no_validate and args.stage != "validate": sequence.append("validate") for stage in sequence: cmd = build_cmd(stage, workdir) run_cmd(cmd, env)if __name__ == "__main__": main()```
- generate_images.py
⚠️注意：第 17 行包含硬编码的 API Key，已替换为占位符。需设置环境变量
DOUBAO_API_KEY
```python#!/usr/bin/env python3"""Generate images for all shots using Doubao (Volcengine) SeedDream API.Reads image_prompts.json, prepends the engineering sketch system prompt."""import jsonimport osimport timefrom pathlib import Pathimport requestsWORKDIR = Path(os.environ.get("PAPER_VIDEO_WORKDIR", Path.cwd()))PROMPTS_PATH = WORKDIR / "prompts" / "image_prompts.json"OUTPUT_DIR = WORKDIR / "assets" / "images"# Doubao API configAPI_KEY = os.environ.get("DOUBAO_API_KEY", "") # ⚠️ 需配置环境变量 DOUBAO_API_KEYAPI_URL = "https://ark.cn-beijing.volces.com/api/v3/images/generations"DEFAULT_MODEL = "doubao-seedream-5-0-260128"# Engineering sketch system prompt (verified in manual)SYSTEM_PROMPT = """你是一位技术插画师，正在为一份高端技术报告绘制概念图解。画面风格像是资深工程师在方格纸笔记本上画的技术速写。【画风】工程手稿速写风格，像是用0.5mm黑色针管笔在浅灰方格纸上画的。线条特征：精确、干净、有建筑制图的美感，直线用尺画（微微不完美），曲线自信流畅。线条粗细差异小（0.3-0.8mm），不像马克笔那么粗犷。人物风格：极简概念人，用几何形状组合——圆形头、梯形身体、线条四肢。比例偏真实（非大头），像工程图纸里的人形比例参考。人物需有清晰可辨的表情（用点和线表达），姿态语言明确夸张。填色：主体为黑色线稿。用浅灰色（#E0E0E0）做轻微的阴影和体积暗示。标注和强调用红色（#D94040）——红色只用于标注文字、圈出重点、画警示符号。背景：带极淡方格纹理的白底，方格线用 #F0F0F0 极浅灰。有偶尔的铅笔底稿痕迹（浅灰虚线）。【版式】画幅比例 16:9 横版。画面底部 20% 完全空白，不放置任何元素（包括文字标注），预留给后期字幕。所有文字标注放在画面中部或中上部。构图水平展开，元素对齐感强。文字标注为简体中文，工整手写印刷体。【系列一致性】这是系列图，需保持风格、配色、人物造型完全一致。每张图左上角有 Fig.N 编号和虚线框标题。"""def generate_image(prompt: str, output_path: Path, model: str = DEFAULT_MODEL, max_retries: int = 3): """Generate image using Doubao API with retry.""" headers = { "Content-Type": "application/json", "Authorization": f"Bearer {API_KEY}" } # Full prompt = system prompt + scene description + quality suffix full_prompt = SYSTEM_PROMPT + "\n\n" + prompt + " IMG_2094.CR2" data = { "model": model, "prompt": full_prompt, "sequential_image_generation": "disabled", "response_format": "url", "size": "2K", "stream": False, "watermark": False } for attempt in range(max_retries): try: response = requests.post(API_URL, headers=headers, json=data, timeout=120) response.raise_for_status() result = response.json() image_url = result.get("data", [{}])[0].get("url") if not image_url: raise ValueError(f"No image URL in response: {result}") img_response = requests.get(image_url, timeout=60) img_response.raise_for_status() output_path.write_bytes(img_response.content) print(f" Generated: {output_path.name}") return output_path except Exception as e: print(f" Attempt {attempt + 1} failed: {e}") if attempt < max_retries - 1: time.sleep(3) else: raisedef main(): OUTPUT_DIR.mkdir(parents=True, exist_ok=True) with open(PROMPTS_PATH, 'r', encoding='utf-8') as f: prompts = json.load(f) for item in prompts: shot = item.get("shot", 0) filename = item.get("filename", f"shot{shot:02d}.png") prompt = item.get("prompt", "") if not prompt: print(f"Skip shot {shot}: no prompt") continue output_path = OUTPUT_DIR / filename if output_path.exists(): print(f"Skip shot {shot}: {filename} already exists") continue try: print(f"Generating shot {shot}...") generate_image(prompt, output_path) time.sleep(1) # Rate limit protection except Exception as e: print(f"Error generating shot {shot}: {e}") continue print(f"\nDone. Images in: {OUTPUT_DIR}")if __name__ == "__main__": main()```
- run_tts_by_clip.py
⚠️注意：需设置环境变量
MINIMAX_API_KEY。可选环境变量：TTS_VOICE_ID（默认male-qn-jingying-jingpin）、TTS_SPEED（默认0.95）、MINIMAX_BASE_URL（默认https://api.minimaxi.com）
```python#!/usr/bin/env python3"""Generate TTS audio per shot using MiniMax API.Reads storyboard.json, generates per-shot WAVs, concatenates with gaps into voice.wav.Outputs timing.json with real durations."""import jsonimport osimport tarfileimport timeimport subprocessfrom pathlib import Pathimport requestsWORKDIR = Path(os.environ.get("PAPER_VIDEO_WORKDIR", Path.cwd()))STORYBOARD_PATH = WORKDIR / "script" / "storyboard.json"AUDIO_DIR = WORKDIR / "assets" / "audio"API_KEY = os.environ.get("MINIMAX_API_KEY")BASE = os.environ.get("MINIMAX_BASE_URL", "https://api.minimaxi.com")HEADERS = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}# Timing constantsSHOT_GAP = 0.5 # silence between shots (seconds)TAIL_SILENCE = 1.5 # silence after last shot (seconds)# Default TTS config (can be overridden via env vars)DEFAULT_VOICE_ID = os.environ.get("TTS_VOICE_ID", "male-qn-jingying-jingpin")DEFAULT_SPEED = float(os.environ.get("TTS_SPEED", "0.95"))SAMPLE_RATE = 32000CHANNELS = 2def require_env(): if not API_KEY: raise RuntimeError("MINIMAX_API_KEY is required. Set it as an environment variable.") if not STORYBOARD_PATH.exists(): raise FileNotFoundError(f"Missing storyboard: {STORYBOARD_PATH}")def create_task(text: str, voice_id: str = DEFAULT_VOICE_ID, speed: float = DEFAULT_SPEED): payload = { "model": "speech-2.8-hd", "text": text, "voice_setting": { "voice_id": voice_id, "speed": speed, "vol": 1, "pitch": 0, }, "audio_setting": { "sample_rate": SAMPLE_RATE, "bitrate": 128000, "format": "mp3", "channel": CHANNELS, }, "language_boost": "Chinese", } r = requests.post(f"{BASE}/v1/t2a_async_v2", headers=HEADERS, json=payload, timeout=30) r.raise_for_status() data = r.json() if data.get("base_resp", {}).get("status_code") != 0: raise RuntimeError(f"TTS submit failed: {data}") return data["task_id"]def wait_for_task(task_id: int): for _ in range(90): time.sleep(2) r = requests.get( f"{BASE}/v1/query/t2a_async_query_v2?task_id={task_id}", headers=HEADERS, timeout=30 ) r.raise_for_status() data = r.json() if data.get("status") == "Success" and data.get("file_id"): return data["file_id"] if data.get("status") == "Failed": raise RuntimeError(f"TTS failed: {data}") raise TimeoutError(f"TTS task timeout: {task_id}")def download_mp3(file_id: int, out_mp3: Path): r = requests.get( f"{BASE}/v1/files/retrieve?file_id={file_id}", headers={"Authorization": f"Bearer {API_KEY}"}, timeout=30, ) r.raise_for_status() url = r.json()["file"]["download_url"] resp = requests.get(url, timeout=120) resp.raise_for_status() tar_path = out_mp3.with_suffix(".tar") tar_path.write_bytes(resp.content) with tarfile.open(tar_path, "r") as tar: members = [m for m in tar.getmembers() if m.name.endswith(".mp3")] if not members: raise RuntimeError("No mp3 in tar") out_mp3.write_bytes(tar.extractfile(members[0]).read()) tar_path.unlink(missing_ok=True)def run(cmd): subprocess.run(cmd, check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)def mp3_to_wav(src_mp3: Path, out_wav: Path): run(["ffmpeg", "-y", "-i", str(src_mp3), "-ar", str(SAMPLE_RATE), "-ac", str(CHANNELS), str(out_wav)])def gen_silence(duration_sec: float, out_wav: Path): run(["ffmpeg", "-y", "-f", "lavfi", "-i", f"anullsrc=r={SAMPLE_RATE}:cl=stereo", "-t", str(duration_sec), str(out_wav)])def concat_wavs(parts: list, out_wav: Path): list_path = out_wav.with_suffix(".txt") list_path.write_text("".join([f"file '{p}'\n" for p in parts]), encoding="utf-8") run(["ffmpeg", "-y", "-f", "concat", "-safe", "0", "-i", str(list_path), "-c", "copy", str(out_wav)]) list_path.unlink(missing_ok=True)def get_duration(path: Path) -> float: result = subprocess.run( ["ffprobe", "-v", "quiet", "-show_entries", "format=duration", "-of", "csv=p=0", str(path)], check=True, capture_output=True, text=True ) return float(result.stdout.strip())def main(): require_env() AUDIO_DIR.mkdir(parents=True, exist_ok=True) with open(STORYBOARD_PATH, "r", encoding="utf-8") as f: storyboard = json.load(f) all_timing = [] all_parts = [] offset = 0.0 for shot in storyboard: shot_num = shot["shot"] text = shot["voice"] mp3_path = AUDIO_DIR / f"shot{shot_num:02d}.mp3" wav_path = AUDIO_DIR / f"shot{shot_num:02d}.wav" # Skip if already generated if wav_path.exists(): dur = get_duration(wav_path) print(f"Shot {shot_num}: skip (exists, {dur:.2f}s)") else: print(f"Shot {shot_num}...", end=" ", flush=True) task_id = create_task(text) file_id = wait_for_task(task_id) download_mp3(file_id, mp3_path) mp3_to_wav(mp3_path, wav_path) mp3_path.unlink(missing_ok=True) dur = get_duration(wav_path) print(f"{dur:.2f}s") all_timing.append({ "shot": shot_num, "start": round(offset, 3), "end": round(offset + dur, 3), "duration": round(dur, 3) }) all_parts.append(str(wav_path)) offset += dur # Add gap between shots if shot_num < len(storyboard): gap_path = AUDIO_DIR / f"gap{shot_num:02d}.wav" gen_silence(SHOT_GAP, gap_path) all_parts.append(str(gap_path)) offset += SHOT_GAP # Add tail silence after last shot tail_path = AUDIO_DIR / "tail_silence.wav" gen_silence(TAIL_SILENCE, tail_path) all_parts.append(str(tail_path)) # Concat all into voice.wav voice_path = AUDIO_DIR / "voice.wav" concat_wavs(all_parts, voice_path) total_dur = get_duration(voice_path) print(f"\nTotal voice.wav: {total_dur:.2f}s") # Save timing timing_path = AUDIO_DIR / "timing.json" with open(timing_path, "w", encoding="utf-8") as f: json.dump(all_timing, f, ensure_ascii=False, indent=2) print(f"Timing saved: {timing_path}") # Cleanup temp files for f in AUDIO_DIR.glob("gap*.wav"): f.unlink() tail_path.unlink(missing_ok=True) print("Done!")if __name__ == "__main__": main()```
- generate_subtitles.py
⚠️注意：字体硬编码为
Source Han Sans CN，根据自身需要修改字体（需要你本机有这个字体）
```python#!/usr/bin/env python3"""Generate ASS subtitles from storyboard voice text + timing.json.Subtitles are the actual narration text (NOT separate summaries).Time is allocated proportionally by character count."""import jsonimport osimport refrom pathlib import PathWORKDIR = Path(os.environ.get("PAPER_VIDEO_WORKDIR", Path.cwd()))STORYBOARD_PATH = WORKDIR / "script" / "storyboard.json"TIMING_PATH = WORKDIR / "assets" / "audio" / "timing.json"ASS_PATH = WORKDIR / "render" / "subtitles.ass"def split_to_subs(text, max_chars=22): """Split narration into subtitle segments at natural break points.""" sentences = re.split(r'(?<=[。？！])', text) sentences = [s.strip() for s in sentences if s.strip()] result = [] for sent in sentences: if len(sent) <= max_chars: result.append(sent) else: parts = re.split(r'(?<=[，、：——])', sent) parts = [p.strip() for p in parts if p.strip()] merged, buf = [], "" for p in parts: if buf and len(buf + p) > max_chars: merged.append(buf) buf = p else: buf += p if buf: merged.append(buf) result.extend(merged) return resultdef char_count(text): """Count meaningful characters (skip punctuation for timing).""" return len(re.sub(r'[，。？！、：——\s]', '', text))def format_time(seconds): """Format seconds to ASS time format H:MM:SS.cc""" h = int(seconds // 3600) m = int((seconds % 3600) // 60) s = seconds % 60 return f"{h}:{m:02d}:{s:05.2f}"def main(): with open(STORYBOARD_PATH, "r", encoding="utf-8") as f: storyboard = json.load(f) with open(TIMING_PATH, "r", encoding="utf-8") as f: timing = json.load(f) # Get video title from first shot or default title = "Paper Video" # Highlight shots (can be customized) highlight_shots = set() events = [] for i, (shot, t) in enumerate(zip(storyboard, timing)): narration = shot["voice"] subs = split_to_subs(narration) total_chars = sum(char_count(s) for s in subs) shot_start = t["start"] usable_duration = t["duration"] - 0.3 if usable_duration < 1: usable_duration = t["duration"] cursor = shot_start for sub_text in subs: proportion = char_count(sub_text) / total_chars if total_chars > 0 else 1 / len(subs) sub_end = cursor + usable_duration * proportion style = "Highlight" if (i + 1) in highlight_shots else "Main" events.append({ "start": round(cursor, 2), "end": round(sub_end, 2), "text": sub_text, "style": style, }) cursor = sub_end # Write ASS file ASS_PATH.parent.mkdir(parents=True, exist_ok=True) with open(ASS_PATH, "w", encoding="utf-8") as f: f.write(f"""[Script Info]Title: {title}ScriptType: v4.00+PlayResX: 1920PlayResY: 1080ScaledBorderAndShadow: yesWrapStyle: 2[V4+ Styles]Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, EncodingStyle: Main,Source Han Sans CN,58,&H00FFFFFF,&H007DD3FC,&H00000000,&H80000000,1,0,0,0,100,100,1,0,1,3,1,2,120,120,56,1Style: Highlight,Source Han Sans CN,58,&H0080FFFF,&H007DD3FC,&H00000000,&H80000000,1,0,0,0,100,100,1,0,1,3,1,2,120,120,56,1[Events]Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text""") for ev in events: f.write(f"Dialogue: 0,{format_time(ev['start'])},{format_time(ev['end'])},{ev['style']},,0,0,0,,{ev['text']}\n") print(f"Generated {len(events)} subtitle events -> {ASS_PATH}")if __name__ == "__main__": main()```
- assemble_and_merge.sh
⚠️注意：需本地安装 ffmpeg / ffprobe。字幕字体需与系统匹配：macOS 用
Source Han Sans CN，根据自身需要修改（需要本机有对应字体包）
```bash#!/usr/bin/env bashset -euo pipefailROOT="${PAPER_VIDEO_WORKDIR:-$(pwd)}"IMAGES_DIR="$ROOT/assets/images"AUDIO_DIR="$ROOT/assets/audio"CLIPS_DIR="$ROOT/render/clips"SUBS="$ROOT/render/subtitles.ass"MERGED="$ROOT/render/merged.mp4"VOICE="$AUDIO_DIR/voice.wav"MUSIC="$AUDIO_DIR/music.mp3"OUT="$ROOT/render/final_16x9.mp4"STORYBOARD="$ROOT/script/storyboard.json"mkdir -p "$CLIPS_DIR"shot_count=$(python3 -c "import json; print(len(json.load(open('$STORYBOARD'))))")echo "Total shots: $shot_count"# Ken Burns motions cycle: zoom_in, pan_right, zoom_in, pan_left, zoom_outmotions=(zoom_in pan_right zoom_in pan_left zoom_out zoom_in pan_right zoom_in pan_left zoom_out)echo "=== Step 1: Creating shot clips ==="for i in $(seq 1 "$shot_count"); do padded=$(printf "%02d" "$i") image="$IMAGES_DIR/shot${padded}.png" duration=$(python3 -c "import json; s=json.load(open('$STORYBOARD')); print(s[$i-1]['duration'])") output="$CLIPS_DIR/shot${padded}.mp4" [ ! -f "$image" ] && { echo " SKIP shot${padded}: no image"; continue; } frames=$(python3 -c "print(int(float('$duration') * 25))") motion_idx=$(( (i - 1) % 5 )) motion=${motions[$motion_idx]} # IMPORTANT: Feed original high-res image directly to zoompan. # zoompan's s=1920x1080 handles the output scaling. # Do NOT pre-scale the image to 1920x1080. case "$motion" in zoom_in) vf="zoompan=z='min(zoom+0.0003,1.06)':x='iw/2-(iw/zoom/2)':y='ih/2-(ih/zoom/2)':d=${frames}:s=1920x1080:fps=25" ;; zoom_out) vf="zoompan=z='if(eq(on,1),1.06,max(zoom-0.0003,1.0))':x='iw/2-(iw/zoom/2)':y='ih/2-(ih/zoom/2)':d=${frames}:s=1920x1080:fps=25" ;; pan_right) vf="zoompan=z=1.04:x='if(eq(on,1),0,min(x+0.3,iw-iw/zoom))':y='ih/2-(ih/zoom/2)':d=${frames}:s=1920x1080:fps=25" ;; pan_left) vf="zoompan=z=1.04:x='if(eq(on,1),iw*0.04,max(x-0.3,0))':y='ih/2-(ih/zoom/2)':d=${frames}:s=1920x1080:fps=25" ;; esac ffmpeg -y -loop 1 -framerate 25 -t "$duration" -i "$image" \ -vf "${vf},format=yuv420p" \ -c:v libx264 -pix_fmt yuv420p -preset medium -crf 18 -r 25 \ -t "$duration" "$output" 2>/dev/null echo " shot${padded} done (${motion}, ${duration}s)"doneecho "=== Step 2: Merging ==="concat_file="$CLIPS_DIR/clips.txt"> "$concat_file"for i in $(seq 1 "$shot_count"); do padded=$(printf "%02d" "$i") clip="$CLIPS_DIR/shot${padded}.mp4" [ -f "$clip" ] && echo "file '$clip'" >> "$concat_file"doneffmpeg -y -f concat -safe 0 -i "$concat_file" -c copy "$MERGED" 2>/dev/nullecho " merged.mp4 done"echo "=== Step 3: Audio ==="if [ -f "$MUSIC" ]; then ffmpeg -y -i "$MERGED" -i "$VOICE" -i "$MUSIC" \ -filter_complex "[2:a]volume=0.18[m];[1:a][m]amix=inputs=2:duration=first:dropout_transition=2[a]" \ -map 0:v -map "[a]" -shortest "$ROOT/render/with_audio.mp4" 2>/dev/nullelse ffmpeg -y -i "$MERGED" -i "$VOICE" -map 0:v -map 1:a -shortest "$ROOT/render/with_audio.mp4" 2>/dev/nullfiecho " with_audio.mp4 done"echo "=== Step 4: Subtitles ==="ffmpeg -y -i "$ROOT/render/with_audio.mp4" -vf "ass=$SUBS" -c:a copy "$OUT" 2>/dev/nullecho " final_16x9.mp4 done"echo ""echo "Rendered: $OUT"```
references
- script-principles.md
**# Script Principles****## What good looks like**A good paper explainer script is not a spoken abstract.It feels like a smart person helping another smart person see the point quickly.**## Default structure**Use a short arc:- Hook- Setup- Conflict- Solution- Takeaway**## Hard rules**- hook fast- use short sentences- introduce jargon late- human explanation before technical label- one clip, one main movement- end with judgment, not summary sludge**## Practical writing rules****### 1. Start with the thing that changed**Bad:- "This paper proposes..."Better:- "The wild part is not that they made another model. They changed a part people barely touched for years."**### 2. Use one strong metaphor**A paper often supports many metaphors. Pick one main carrier.Examples:- assembly line- kitchen / recipe- group chat- mixing board- foundation / building block**### 3. Delay terminology until the viewer has a picture**Bad:- open with residual connection, pre-norm, dilutionBetter:- show the phenomenon first- name it after the audience already gets it**### 4. Spoken script != written prose**Favor:- short clauses- easy breath points- high oral clarityAvoid:- nested structure- parenthetical detours- stacked qualifiers**### 5. Every clip needs a memorable sentence**Not cheesy. Just repeatable.Examples:- "残差连接终于学会了挑重点。"- "真正的问题，可能就藏在这个老零件里。"**## Revision checklist**Ask for each clip:- what is the one thing this clip wants the viewer to understand?- can I say it in fewer beats?- is the metaphor still doing useful work?- is there a line worth remembering?```
- schema.md
**## 目录结构（必须遵守）**<project>/├── assets/│ ├── images/ # shot01.png, shot02.png, ...│ └── audio/│ ├── voice.wav # 完整旁白（单文件）│ └── music.mp3 # 背景音乐（可选）├── prompts/│ ├── style_guide.md│ ├── image_prompts.json│ └── generate_images_doubao.py├── render/│ ├── check_assets.py│ ├── render.sh│ ├── subtitles.ass│ ├── clips/│ ├── merged.mp4│ ├── with_audio.mp4│ └── final_16x9.mp4└── script/ ├── paper_summary.json ├── storyboard.json ├── narration_v2.txt ├── subtitles.json └── timing_marks.json**## `paper_summary.json`**{ "title": "论文标题", "authors": "作者", "url": "https://arxiv.org/abs/...", "core_problem": "论文攻击的核心问题", "method_summary": "方法概要，大白话", "key_results": ["结果1", "结果2"], "key_context": "为什么这篇论文重要", "analogy_seeds": ["比喻1", "比喻2"]}**## `storyboard.json`**一个数组，每个元素是一个 shot：{ "shot": 1, "start": 0.0, "end": 6.0, "duration": 6.0, "voice": "完整旁白文本", "subtitle": ["字幕行1", "字幕行2"], "image": "shot01.png", "image_prompt": "图片生成 prompt", "negative_prompt": "负面 prompt", "motion": "slow_zoom_in", "transition_in": "fade", "transition_out": "cut"}**### 字段说明**- `shot`: 序号，从 1 开始- `start/end/duration`: 时间轴（秒）- `voice`: 该 shot 的完整旁白- `subtitle`: 字幕提示数组（2-3 行短语，每行 ≤10 汉字）- `image`: 图片文件名（shotXX.png）- `image_prompt`: 图片生成的详细 prompt- `negative_prompt`: 所有 shot 共享的负面 prompt- `motion`: 镜头运动（slow_zoom_in / slow_zoom_out / pan_left / pan_right / focus_pulse / split_focus）- `transition_in/out`: 转场（fade / cut）**## `image_prompts.json`**[ { "shot": 1, "filename": "shot01.png", "prompt": "图片 prompt（不含 STYLE_SUFFIX）", "negative_prompt": "统一的负面 prompt" }]注意：`generate_images_doubao.py` 会在运行时自动追加 STYLE_SUFFIX。**## `subtitles.json`**[ { "start": 0.0, "end": 3.5, "text": "字幕文本" }]**## `timing_marks.json`**[ { "shot": 1, "segment": "hook", "voice_text": "旁白文本", "pause_after_ms": 260, "cut_on": "question_end", "subtitle_hint": ["字幕行1", "字幕行2"] }]**## Rerun 策略**- 改脚本 → 重新生成 TTS、字幕、受影响的 shot 渲染、合并- 只改图片 → 重新渲染受影响的 shot、合并- 只改字幕策略 → 重新生成字幕、烧字幕- 渲染前后都应运行 check_assets.py```---**## references/quality-checklist.md**> ****⚠️ Notice****：无需配置，可直接使用。```markdown**# Quality Checklist****## Opening**- does the first 3-8 seconds earn attention?- is the hook specific, not generic?- does the viewer understand why this paper might matter?**## Narrative**- is there a clear arc from old problem to new idea?- does each clip advance one idea?- is jargon introduced only after intuition?- does the ending make a judgment?**## Visuals**- do the visuals clarify the spoken idea?- is the style consistent across the series?- is emphasis visible immediately?- is subtitle-safe area preserved?**## Audio**- does the voice sound natural?- are clip boundaries smooth?- is pacing too rushed anywhere?- do any lines feel harder to hear than to read?**## Subtitles**- readable on mobile?- phrase boundaries natural?- no overcrowded blocks?- no key phrases split awkwardly?**## Final decision**A video is not done because it rendered.It is done when a cold viewer can follow it, remember it, and feel that it had a point.```