How to narrate an explainer video with an AI voice

{
  "title": "What is a data center?",
  "voice": "coral",
  "model": "gpt-audio-1.5",
  "instructions": "Warm, friendly explainer talking with a curious neighbor. Easy conversational pace, like explaining something to a friend over coffee: no dramatic pauses, keep sentences flowing, brief breaths only between sentences. Plain-spoken with a little delight in how things work. Not salesy.",
  "scenes": [
    {
      "id": "hook",
      "text": "This morning, before you finished your coffee, you probably used a data center."
    },
    {
      "id": "everyday",
      "text": "Your doctor pulled up your medical records. An airline held your seat on a flight. A friend liked the photo you posted last night. And a grocery store worked out how much milk to put on tomorrow's truck. All of that ran on computers, in buildings most of us never see."
    },
    {
      "id": "inside",
      "text": "Those buildings are data centers. Inside, a data center looks a bit like a library. Instead of books, the shelves hold computers called servers. Thousands of them, stacked in tall racks, working day and night."
    },
    {
      "id": "power",
      "text": "Data centers run on electricity. Power arrives from the grid through a substation, the same way it reaches your neighborhood, just a lot more of it. Batteries and backup generators wait on standby, so the computers stay on if the grid blinks."
    },
    {
      "id": "cooling",
      "text": "All that computing makes heat, like a laptop warming up on your lap. So the building's other big job is cooling. Fans and cooling equipment carry the heat outside. Some systems use air. Others loop the same water around and around, like the radiator in a car. And some evaporate water, a bit like sweat cooling your skin."
    },
    {
      "id": "water",
      "text": "When water evaporates, it drifts off as vapor, so it gets topped up. Some data centers use treated wastewater for that, instead of drinking water. When one state checked its utility records, most data center buildings used about as much water as a large office building. Put together, all of them used under half a percent of the state's water. How much each one uses comes down to its cooling design."
    },
    {
      "id": "network",
      "text": "Fiber-optic cables connect the building to the rest of the internet. Your request goes in, and the answer comes back out in a fraction of a second."
    },
    {
      "id": "where",
      "text": "So where do these buildings go? A data center needs three things: lots of reliable power, fast fiber connections, and enough land for the building and the equipment around it. That's why they often cluster where big power connections and fiber routes come together."
    },
    {
      "id": "people",
      "text": "Building one is a big job. At the busiest point of construction, more than a thousand workers can be on site: electricians, pipefitters, and concrete crews. Once it's running, a typical building needs around fifty people full time, keeping the power, the cooling, and the computers healthy."
    },
    {
      "id": "myth",
      "text": "So it isn't a mystery box puffing smoke. The servers don't burn anything. The backup generators do burn fuel when they run, mostly for short tests and power outages. Like any big building, what matters most is where its electricity comes from and how it's cooled."
    },
    {
      "id": "close",
      "text": "A data center is a building, a power connection, a lot of computers, a way to keep them cool, and the people who look after it. Each one is something you can ask about when a data center is proposed near you."
    }
  ]
}

// Generate one narration clip per scene with gpt-audio-1.5 and record
// durations in timeline.json. The model returns a transcript; any clip that
// strays from the script is regenerated so the narration stays verbatim.
import { readFileSync, writeFileSync, mkdirSync } from 'node:fs';
import { execFileSync, spawnSync } from 'node:child_process';

const script = JSON.parse(readFileSync('script.json', 'utf8'));
const key = process.env.OPENAI_API_KEY_CLAUDE || process.env.OPENAI_API_KEY;
const only = process.argv[2];   // a scene id to re-voice, or --timeline to skip voicing
const words = s => s.toLowerCase().replace(/[‘’]/g, "'").replace(/[^a-z0-9' ]+/g, ' ').split(/\s+/).filter(Boolean).join(' ');
mkdirSync('audio', { recursive: true });

async function voice(text) {
  const res = await fetch('https://api.openai.com/v1/chat/completions', {
    method: 'POST',
    headers: { Authorization: `Bearer ${key}`, 'Content-Type': 'application/json' },
    body: JSON.stringify({
      model: script.model,
      modalities: ['text', 'audio'],
      audio: { voice: script.voice, format: 'wav' },
      messages: [
        { role: 'system', content: `You are a voice actor. Read the user's text aloud exactly as written, word for word, adding nothing. Style: ${script.instructions}` },
        { role: 'user', content: text },
      ],
    }),
  });
  if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);
  return (await res.json()).choices[0].message.audio;
}

for (const scene of script.scenes) {
  if (only && !only.split(',').includes(scene.id)) continue;
  if (only === '--timeline') break;
  for (let attempt = 1; ; attempt++) {
    const audio = await voice(scene.text);
    if (words(audio.transcript) === words(scene.text)) {
      writeFileSync(`audio/${scene.id}.raw.wav`, Buffer.from(audio.data, 'base64'));
      // Trim lead/tail silence and shorten any mid-clip pause to ~0.35s.
      execFileSync('ffmpeg', ['-y', '-v', 'error', '-i', `audio/${scene.id}.raw.wav`, '-af',
        'silenceremove=start_periods=1:start_threshold=-45dB:stop_periods=-1:stop_duration=0.35:stop_threshold=-45dB,apad=pad_dur=0.1',
        `audio/${scene.id}.wav`]);
      console.log('voiced', scene.id, attempt > 1 ? `(attempt ${attempt})` : '');
      break;
    }
    console.log(`retry ${scene.id}: "${audio.transcript}"`);
    if (attempt === 4) throw new Error(`${scene.id} would not read verbatim`);
  }
}

// Split each scene into sentences and place each boundary at the detected
// pause nearest to where its share of the characters says it should fall.
function sentences(text) {
  return text.match(/[^.!?]+[.!?]+/g).map(s => s.trim());
}
// Lay scenes end to end with a short breath between them.
const gap = 0.6, lead = 0.8;
let t = lead;
const scenes = [];
for (const scene of script.scenes) {
  const file = `audio/${scene.id}.wav`;
  const duration = Number(execFileSync('ffprobe', ['-v', 'error', '-show_entries', 'format=duration', '-of', 'csv=p=0', file]).toString());
  const log = spawnSync('ffmpeg', ['-hide_banner', '-i', file, '-af', 'silencedetect=n=-40dB:d=0.12', '-f', 'null', '-']).stderr.toString();
  const starts = [...log.matchAll(/silence_start: ([\d.]+)/g)].map(m => +m[1]);
  const ends = [...log.matchAll(/silence_end: ([\d.]+)/g)].map(m => +m[1]);
  const mids = starts.map((s, i) => (s + (ends[i] ?? duration)) / 2).filter(m => m > 0.3 && m < duration - 0.3);
  const parts = sentences(scene.text);
  const total = parts.reduce((n, p) => n + p.length, 0);
  let chars = 0, previous = 0;
  const cues = parts.map((text, i) => {
    let at = 0;
    if (i > 0) {
      const guess = duration * chars / total;
      const near = mids.filter(m => m > previous + 1).sort((a, b) => Math.abs(a - guess) - Math.abs(b - guess))[0];
      at = near !== undefined && Math.abs(near - guess) < 1.4 ? near : guess;
    }
    chars += text.length; previous = at;
    return { text, at: +(t + at).toFixed(3) };
  });
  cues.forEach((cue, i) => cue.end = i + 1 < cues.length ? cues[i + 1].at : +(t + duration).toFixed(3));
  scenes.push({ id: scene.id, start: +t.toFixed(3), voice: +duration.toFixed(3), cues });
  t += duration + gap;
}
const total = +(t + 3.6).toFixed(3);   // hold on the end card
const timeline = { total, scenes };
writeFileSync('timeline.json', JSON.stringify(timeline, null, 2));
writeFileSync('timeline.js', `window.TIMELINE = ${JSON.stringify(timeline)};\n`);

// Captions file for the site's <video> player.
const stamp = s => new Date(s * 1000).toISOString().slice(11, 23);
writeFileSync('what-is-a-data-center.vtt', 'WEBVTT\n\n' + scenes.flatMap(s => s.cues)
  .map((c, i) => `${i + 1}\n${stamp(c.at)} --> ${stamp(c.end)}\n${c.text}\n`).join('\n'));

// One narration track with each clip at its start time.
execFileSync('ffmpeg', ['-y', '-v', 'error',
  ...scenes.flatMap(s => ['-i', `audio/${s.id}.wav`]),
  '-filter_complex', scenes.map((s, i) => `[${i}]adelay=${Math.round(s.start * 1000)}:all=1[a${i}]`).join(';')
    + ';' + scenes.map((_, i) => `[a${i}]`).join('') + `amix=inputs=${scenes.length}:normalize=0,apad=whole_dur=${total}`,
  '-ar', '48000', 'audio/narration.wav']);
console.log(scenes.map(s => `${s.id} ${s.start}s +${s.voice}s  ` + s.cues.map(c => c.at).join(' ')).join('\n'), '\ntotal', total);

// Render explainer.html frame by frame with Playwright and mux the narration.
//   node render.mjs stills 3,12.5,40     -> stills/t-3.png ...
//   node render.mjs video [--captions]   -> out/what-is-a-data-center[-captioned].mp4
import { spawn } from 'node:child_process';
import { mkdirSync } from 'node:fs';
import { createRequire } from 'node:module';
import { pathToFileURL } from 'node:url';
import { resolve } from 'node:path';

const require = createRequire(import.meta.url);
const { chromium } = require('/opt/homebrew/lib/node_modules/playwright');
const [mode = 'video', arg] = process.argv.slice(2);
const captions = process.argv.includes('--captions');
const fps = 30;

const browser = await chromium.launch();
const page = await browser.newPage({ viewport: { width: 1920, height: 1080 } });
const url = pathToFileURL(resolve('explainer.html')).href + `?render=1&captions=${captions ? 1 : 0}`;
await page.goto(url);
await page.evaluate(() => document.fonts.ready);
const frame = page.locator('#frame');

if (mode === 'stills') {
  mkdirSync('stills', { recursive: true });
  for (const t of arg.split(',').map(Number)) {
    await page.evaluate(t => window.__seek(t), t);
    await frame.screenshot({ path: `stills/t-${t}.png` });
  }
} else {
  mkdirSync('out', { recursive: true });
  const duration = await page.evaluate(() => window.__duration);
  const out = `out/what-is-a-data-center${captions ? '-captioned' : ''}.mp4`;
  const ffmpeg = spawn('ffmpeg', ['-y', '-v', 'error', '-f', 'image2pipe', '-framerate', String(fps), '-c:v', 'mjpeg', '-i', '-',
    '-i', 'audio/narration.wav', '-c:v', 'libx264', '-preset', 'medium', '-crf', '20', '-pix_fmt', 'yuv420p',
    '-c:a', 'aac', '-b:a', '160k', '-shortest', '-movflags', '+faststart', out], { stdio: ['pipe', 'inherit', 'inherit'] });
  const frames = Math.ceil(duration * fps);
  for (let i = 0; i < frames; i++) {
    await page.evaluate(t => window.__seek(t), i / fps);
    const jpg = await frame.screenshot({ type: 'jpeg', quality: 92 });
    if (!ffmpeg.stdin.write(jpg)) await new Promise(r => ffmpeg.stdin.once('drain', r));
    if (i % 300 === 0) console.log(`frame ${i}/${frames}`);
  }
  ffmpeg.stdin.end();
  await new Promise(r => ffmpeg.on('close', r));
  console.log('wrote', out);
}
await browser.close();