Audio and lip-sync: directing sound and speech

Every other lesson in this course would mostly apply to any video model. This one wouldn't. Joint audio-video generation is HappyHorse-1.0's defining trick: because audio tokens travel through the same transformer layers as the video tokens (Lesson 1), sound isn't added to the picture — it's generated with it. Lips match phonemes because the mouth movement and the sound were never computed separately.

The three audio layers

Think of the generated soundtrack as three layers you can direct independently in the prompt:

1. Ambience — driven by scene words. "Rainy night market" produces rain on canvas, crowd murmur, distant traffic. You get ambience whether you ask or not; ask, and you control it.
2. Sound effects — driven by action words. Hoofbeats, door slams, sizzling. If an action matters acoustically, name it: "boots crunching on gravel" beats "walking."
3. Speech and music — driven by explicit cues, covered below.

Prompting dialogue

Put the line in quotes and attribute it to the speaker:

A weathered fisherman looks into the camera and says "The sea gives, and the sea takes," documentary style, handheld close-up, overcast harbor light, somber

The quoted text is what gets spoken, with lip-sync. Ground rules learned the hard way:

• Keep lines short. A comfortable speaking pace is ~2.5 words per second — a 5-second clip fits about one sentence. Longer lines get rushed or clipped.
• One speaker per clip is reliable; two-person exchanges work but expect more retries. For a conversation, generate one clip per line and cut them together.
• Describe the delivery: "whispers," "shouts over the wind," "says flatly." Delivery words shape both the voice and the facial performance.
• Emotion comes from the mood words. The same line reads differently under "joyful" vs "menacing" — the model acts the whole face, not just the mouth.

The six languages

Native lip-sync works in English, Mandarin, Japanese, Korean, German, and French. Write the dialogue in the target language directly:

一位茶馆老板娘微笑着说"欢迎光临,里面请",温暖的午后光线,浅景深,平静温馨

Mixing languages in one clip is unreliable — pick one per generation. If you need the same scene in three languages (ads, localization), keep the prompt identical and swap only the quoted line: composition stays close, and you get a localized set from one setup. This is exactly the workflow lip-sync platforms like happyhorseai.top wrap a UI around.

Prompting music

Ask for score the way you'd brief a composer — genre, instrumentation, energy:

…soft piano score, sparse and hopeful …driving synthwave soundtrack, building intensity

Music competes with dialogue in a short clip. For spoken scenes, either skip the score cue or ask for "quiet underscore." For montage-style clips with no dialogue, a strong music cue is often what makes the result feel finished.

When you don't want audio

Product loops, website heroes, and GIF-style assets are usually consumed muted. Some hosts expose an audio on/off toggle; where there isn't one, prompt "silent, no music" — and either way, review the clip muted before publishing, since autoplaying sound is where good clips become annoying ones.

Troubleshooting

Next lesson: the camera. Same idea as this one — the model responds to professional vocabulary, so we'll learn to speak director.