Dubbing a 30-Second Ad Is Not a Translation Problem

Take one line from an ad: “Learn more.” Two syllables. The video was cut around it, so it gets the slot it gets.

Translate it into Spanish and you get “Más información.” Five syllables. Same meaning, more than twice the spoken material, and nobody told the text-to-speech model that the slot has a fixed length.

That is the whole problem, and it is not a translation problem. The translation is correct. The video is the thing that will not move.

Why the naive version fails

Most of what gets called “AI dubbing” is a translation step and a text-to-speech step bolted together. Feed it an ad and you get audio that is the right words and the wrong length, so either the ad runs long, or a line crowds the next one, or the audio walks over a cut it was never written for.

The pipeline below is local-first: video in, dubbed video out, using WhisperX, Demucs, Gemini and Coqui’s XTTS. Six stages, and the interesting thing about them is that almost every stage exists to serve a duration constraint rather than a language one.

1. Extract the audio. Plain ffmpeg, a clean WAV to work from.

2. Separate the voice from everything else. Demucs splits the track into vocals and not-vocals. The goal is never to replace the whole soundtrack, it is to lift out the voice, dub it, and drop it back onto a music and effects bed that is never touched.

3. Transcribe and align. WhisperX gives word-level timestamps, not just text. This is where each line gets a hard duration budget, because a timestamp pair is exactly that.

4. Translate, and check the shape of the answer. Gemini gets a prompt written for ad copy specifically: preserve calls to action, keep brand names from a glossary, add no disclaimers the source never had. The response has to come back as a JSON array the same length as the input, and if it does not, the pipeline raises. A model returning 39 lines for a 40-line ad is a bug, not something to paper over.

5. Clone the voice per line, then make it fit the slot. XTTS clones from the isolated vocal crop of each individual segment rather than one reference for the whole ad, so a two-speaker ad keeps both voices on their own lines. Then the generated audio has to be reconciled with the slot WhisperX measured.

6. Mix, mux, optionally lip-sync. Dubbed vocals go back over the untouched music bed, muxed onto the original video. Wav2Lip is layered on top and optional.

Stage 5 is where this gets interesting, and where I got it wrong.

The bug I found while writing this post

I originally wrote that stage 5 “time-stretches the audio to exactly the target duration.” That is what the function is named. It is not what the function does.

def _stretch_to_length(y, sr, target_samples):
    x_old = np.linspace(0.0, len(y) / sr, num=len(y), endpoint=False)
    x_new = np.linspace(0.0, target_samples / sr, num=target_samples, endpoint=False)
    return np.interp(x_new, x_old, y.astype(np.float64)).astype(np.float32)

Look at the spacing of the two grids. x_old steps by (len(y)/sr) / len(y), which is 1/sr. x_new steps by (target/sr) / target, which is also 1/sr. Identical sample spacing, different total spans.

So np.interp is not compressing anything onto a new grid. It is reading the original signal at its original rate over a shorter window. I tested it against a synthetic tone to be sure, and the output is bit-identical to simply slicing the first N samples:

input : 43200 samples = 1.80s
output: 33600 samples = 1.40s
max|out - first 1.4s of input| = 0.00e+00   <- truncation
max|out - true resample|       = 2.00e+00   <- not a resample

It truncates. When the Spanish line runs longer than the slot, the end of the word is cut off, not compressed.

The other direction is worse in a quieter way. When the generated line is shorter than the slot, x_new runs past the end of x_old, and np.interp clamps to the final sample value. So the remainder of the slot is filled with the last sample held constant, a DC offset rather than silence, which is the kind of thing that clicks on playback.

I would not have found this by listening to a demo clip. I found it by writing a paragraph confidently describing behaviour I had not verified, then checking.

What it should do instead

Fitting audio to a duration is a real signal-processing problem with real solutions, and none of them are one call to np.interp.

Resampling onto a genuinely different grid changes duration but also shifts pitch, so a compressed line comes back sounding higher. That is usually not acceptable for a voice.

The correct tool is time-stretching that preserves pitch, a phase vocoder or WSOLA, which librosa.effects.time_stretch and pyrubberband both expose. Even then there is a limit: compress a line more than roughly 10 to 20 percent and it starts sounding rushed regardless of how good the algorithm is.

Which points at the better fix, further upstream. If the Spanish line does not fit, the strongest move is not to squash the audio, it is to ask the translator for a shorter line. Gemini already knows the target language. It could be given the syllable budget too, and asked for a variant that fits. That turns a signal-processing problem back into a copywriting one, which is where an advertising agency would have solved it in the first place.

What else actually breaks

Apple Silicon and WhisperX alignment. MPS acceleration is unreliable for the alignment step, so the pipeline forces CPU there regardless of available hardware. If your transcription is fine but alignment behaves strangely on a Mac, start here.

Wav2Lip on CPU gets killed. A SIGKILL with return code -9 is almost always out-of-memory, and Wav2Lip on CPU will do that on longer clips. The pipeline catches that specific case and returns an error explaining what to change, rather than a bare traceback.

Wav2Lip against modern librosa. The vendored copy calls librosa.filters.mel() positionally. Current librosa made those arguments keyword-only, so it fails outright on install. There is a patch script that rewrites the call. That is the ordinary tax of building on a research repo that was correct in 2020 and has been rotting under its dependencies since.

Lip sync is the one stage with a safety net. The pipeline always writes a _predub copy alongside the lip-synced output so the two can be compared by eye. Wav2Lip is genuinely capable of producing something nobody should ship without watching first, particularly on the fast cuts ads are full of.

Consent is enforced in the function, not the form

if not consent:
    raise ValueError(
        "Consent is required: confirm you have rights to clone this voice (see README)."
    )

There is a checkbox in the web UI too, but the checkbox is not the control. This raise is, which means no code path, UI or CLI, can clone a voice without that flag being set deliberately. For a capability like voice cloning, the safe default should not depend on every caller remembering to tick something.

What I would take from this

Duration is the spec, not a detail. The moment you dub video rather than translate text, you have taken on a timing constraint, and it should shape the design from the first line rather than get patched in once the output sounds wrong.

Validate the shape of a model’s answer, not just its content. The check that stops a short translation response from silently becoming a short ad is not a cleverer prompt, it is counting the array length.

A function name is not evidence. _stretch_to_length reads as settled, which is exactly why nobody re-read it. The bug survived because the name described the intent and everyone, me included, took the name at face value.

Write it down to find out whether you know it. This whole correction exists because documenting the pipeline forced me to state precisely what a function did, and precision is where the gap showed up. That is a reasonable argument for writing about your own systems even when nobody is asking you to.