# Dubbing

> Voices, voice cloning, and a clean dub or a voice-over.

A translation with `ttsOptions` is **dubbed**: each translated line is synthesised with a voice and
timed to the original speech. The dub is then available as a dub track in your outputs.

```json
{
  "language": "es",
  "engine": { "@type": "Gemini" },
  "ttsOptions": {
    "provider": { "@type": "Deepdub", "voice": "auto" }
  }
}
```

## Voice providers

**Deepdub (voice cloning)**

**Clones each speaker's own voice** into the target language. `voice: "auto"` picks the voice
from the speaker; nothing else is needed.

```json
"provider": { "@type": "Deepdub", "voice": "auto" }
```

| Field | Default | |
|---|---|---|
| `voice` | required | `"auto"` to clone the speaker. |
| `model` | `"syncwords34-baked"` | Deepdub model; `backupModel` (`"dd-etts-3.4"`) is used when it is unavailable. |
| `promptBoost` | `false` | Work harder to reproduce the speaker's characteristics; less robust. |
| `cleanMode` | `false` | More robust with noisy source audio. |
| `accentLocale`, `accentRatio` | none, `0.75` | Blend in an accent: `accentLocale` is a Deepdub locale, `accentRatio` from 0 (native) to 1 (full accent). |

The pronunciation locale (for example `es-ES` for `es`) is chosen for you from the language; set
`ttsOptions.locale` to choose another, such as `es-XL` for Latin American Spanish.

**MultiVoice**

**A voice for each speaker**, chosen for you from SyncWords' voices for the translation's
language. You name the provider, never the voices.

```json
"provider": { "@type": "MultiVoice" }
```

Not every language has MultiVoice voices: a project that asks for one in such a language is
refused with `422` when you save it.

**Azure**

Microsoft Azure neural voices, by name.

```json
"provider": { "@type": "Azure", "voice": "en-US-PhoebeMultilingualNeural" }
```

Voice names: [Azure voice gallery](https://learn.microsoft.com/azure/ai-services/speech-service/language-support?tabs=tts).

**Google**

Google Cloud Text-to-Speech voices, by name.

```json
"provider": { "@type": "Google", "voice": "es-ES-Standard-A" }
```

Voice names: [Google voice list](https://cloud.google.com/text-to-speech/docs/voices).

**Amazon Polly**

Amazon Polly voices, by name, with an engine.

```json
"provider": { "@type": "Amazon", "voice": "Vicki", "engine": "NEURAL" }
```

`engine` is `STANDARD`, `NEURAL` (default) or `GENERATIVE`; not every voice has every engine.
Voice names: [Polly voice list](https://docs.aws.amazon.com/polly/latest/dg/available-voices.html).

**Sarvam**

Voice cloning for Indian languages.

```json
"provider": { "@type": "Sarvam" }
```

The voice must speak the target language. Voices are not checked against the provider's catalogue
when you save a project: a voice that does not exist fails when files are processed, and one of
the wrong language gives a poor dub. [Choosing a voice](/vod-express/configuration/voices) has a first voice
for each provider that is known to work, and how to try one on a short clip. Every provider's
fields: [Config reference](/vod-express/configuration/reference#dubbing).

### Speaking rate

Every provider takes the same rate options, which keep the dub in step with the original:

| Field | Default | |
|---|---|---|
| `rateStrategy` | `"dynamic"` | `dynamic` speeds a line up or slows it down to fit the time the original took; `static` keeps `rate`. |
| `rate` | `100` | Speaking rate in percent (20–200), for `static`. |
| `minDynamicRate`, `maxDynamicRate` | `80`, `160` | The limits of `dynamic`. |

## Clean dub or voice-over

`originalAudioSplitting` decides what is left of the original under the new voice:

- **Clean dub (default)**: `originalAudioSplitting: true`: the original is separated into voices and background. The background (music, effects, ambience) stays and the original voices are removed, as in a studio dub.
- **Voice-over**: `originalAudioSplitting: false`: the whole original stays under the new voice at a lower level, as in a news or documentary voice-over.

```json
"ttsOptions": {
  "provider": { "@type": "Azure", "voice": "en-US-PhoebeMultilingualNeural" },
  "originalAudioSplitting": false,
  "originalAudioVolume": 0.6
}
```

## Mix levels

| Field | Default | |
|---|---|---|
| `volume` | `1` | Level of the new voice, 0–2. |
| `originalAudioVolume` | `1` | Level of what is kept of the original (the background, or the whole original for a voice-over), 0–10. `0` leaves it out. |
| `lateUtteranceTimeout` | `"10"` | Seconds after which a line that could not be synthesised in time is skipped. |

## Pronunciation

`ttsOptions.locale` sets the pronunciation locale, such as `pt-BR` or `es-XL`. It must belong to the
translation's language.

`ttsOptions.dictionary` maps a word to how it should be spoken:

```json
"dictionary": { "SyncWords": "sink words", "Newsmax": "news max" }
```

## Hearing the dub

A dub is heard through an output file's dub track in that language:

```json
"outputs": [
  {
    "@type": "S3FolderOutput",
    "items": [
      {
        "@type": "VideoFile",
        "ext": "mp4",
        "nameTemplate": "${name}.es.${ext}",
        "useOriginalTracks": true,
        "dubTracks": [{ "language": "es", "title": "Español" }]
      }
    ]
  }
]
```

This file carries the input's own audio first and the Spanish dub second, so a player opens it
on the input's audio; leave `useOriginalTracks` out for a file with the dub alone. See
[Outputs](/vod-express/configuration/outputs). A project that reads an RSS or Media RSS feed can also publish the
dub as a feed of its own, one per dubbed language: [Feeds](/vod-express/concepts/inputs#the-output-feed).

A dub track is tagged with its language's three-letter code (`spa`), and that tag is how a dub
finds its track. Two dubs of one language — `es` and `es-MX`, or `pt-BR` and `pt-PT` — would share
a tag, so a project dubs each language once: the second is refused when you save the project.
Translating into both, with only one of them dubbed, is fine.