Dubbing

A translation with ttsOptions is dubbed: each translated line is synthesised with a voice and timed to the original speech. The dub is then available as a dub track in your outputs.

{
  "language": "es",
  "engine": { "@type": "Gemini" },
  "ttsOptions": {
    "provider": { "@type": "Deepdub", "voice": "auto" }
  }
}

Voice providers

Clones each speaker’s own voice into the target language. voice: "auto" picks the voice from the speaker; nothing else is needed.

"provider": { "@type": "Deepdub", "voice": "auto" }

Field	Default
`voice`	required	`"auto"` to clone the speaker.
`model`	`"syncwords34-baked"`	Deepdub model; `backupModel` (`"dd-etts-3.4"`) is used when it is unavailable.
`promptBoost`	`false`	Work harder to reproduce the speaker’s characteristics; less robust.
`cleanMode`	`false`	More robust with noisy source audio.
`accentLocale`, `accentRatio`	none, `0.75`	Blend in an accent: `accentLocale` is a Deepdub locale, `accentRatio` from 0 (native) to 1 (full accent).

The pronunciation locale (for example es-ES for es) is chosen for you from the language; set ttsOptions.locale to choose another, such as es-XL for Latin American Spanish.

A voice for each speaker, chosen for you from SyncWords’ voices for the translation’s language. You name the provider, never the voices.

"provider": { "@type": "MultiVoice" }

Not every language has MultiVoice voices: a project that asks for one in such a language is refused with 422 when you save it.

Microsoft Azure neural voices, by name.

"provider": { "@type": "Azure", "voice": "en-US-PhoebeMultilingualNeural" }

Voice names: Azure voice gallery.

Google Cloud Text-to-Speech voices, by name.

"provider": { "@type": "Google", "voice": "es-ES-Standard-A" }

Voice names: Google voice list.

Amazon Polly voices, by name, with an engine.

"provider": { "@type": "Amazon", "voice": "Vicki", "engine": "NEURAL" }

engine is STANDARD, NEURAL (default) or GENERATIVE; not every voice has every engine. Voice names: Polly voice list.

Voice cloning for Indian languages.

"provider": { "@type": "Sarvam" }

The voice must speak the target language. Voices are not checked against the provider’s catalogue when you save a project: a voice that does not exist fails when files are processed, and one of the wrong language gives a poor dub. Choosing a voice has a first voice for each provider that is known to work, and how to try one on a short clip. Every provider’s fields: Config reference.

Speaking rate

Every provider takes the same rate options, which keep the dub in step with the original:

Field	Default
`rateStrategy`	`"dynamic"`	`dynamic` speeds a line up or slows it down to fit the time the original took; `static` keeps `rate`.
`rate`	`100`	Speaking rate in percent (20–200), for `static`.
`minDynamicRate`, `maxDynamicRate`	`80`, `160`	The limits of `dynamic`.

Clean dub or voice-over

originalAudioSplitting decides what is left of the original under the new voice:

Clean dub (default)

originalAudioSplitting: true: the original is separated into voices and background. The background (music, effects, ambience) stays and the original voices are removed, as in a studio dub.

Voice-over

originalAudioSplitting: false: the whole original stays under the new voice at a lower level, as in a news or documentary voice-over.

"ttsOptions": {
  "provider": { "@type": "Azure", "voice": "en-US-PhoebeMultilingualNeural" },
  "originalAudioSplitting": false,
  "originalAudioVolume": 0.6
}

Mix levels

Field	Default
`volume`	`1`	Level of the new voice, 0–2.
`originalAudioVolume`	`1`	Level of what is kept of the original (the background, or the whole original for a voice-over), 0–10. `0` leaves it out.
`lateUtteranceTimeout`	`"10"`	Seconds after which a line that could not be synthesised in time is skipped.

Pronunciation

ttsOptions.locale sets the pronunciation locale, such as pt-BR or es-XL. It must belong to the translation’s language. ttsOptions.dictionary maps a word to how it should be spoken:

"dictionary": { "SyncWords": "sink words", "Newsmax": "news max" }

Hearing the dub

A dub is heard through an output file’s dub track in that language:

"outputs": [
  {
    "@type": "S3FolderOutput",
    "items": [
      {
        "@type": "VideoFile",
        "ext": "mp4",
        "nameTemplate": "${name}.es.${ext}",
        "useOriginalTracks": true,
        "dubTracks": [{ "language": "es", "title": "Español" }]
      }
    ]
  }
]

This file carries the input’s own audio first and the Spanish dub second, so a player opens it on the input’s audio; leave useOriginalTracks out for a file with the dub alone. See Outputs. A project that reads an RSS or Media RSS feed can also publish the dub as a feed of its own, one per dubbed language: Feeds. A dub track is tagged with its language’s three-letter code (spa), and that tag is how a dub finds its track. Two dubs of one language — es and es-MX, or pt-BR and pt-PT — would share a tag, so a project dubs each language once: the second is refused when you save the project. Translating into both, with only one of them dubbed, is fine.

Get started

Concepts

Configure a project

Upload media

Examples

Going to production

Help

Voice providers

Speaking rate

Clean dub or voice-over

Clean dub (default)

Voice-over

Mix levels

Pronunciation

Hearing the dub

​Voice providers

​Speaking rate

​Clean dub or voice-over

Clean dub (default)

Voice-over

​Mix levels

​Pronunciation

​Hearing the dub

Voice providers

Speaking rate

Clean dub or voice-over

Mix levels

Pronunciation

Hearing the dub