Speech marks

Overview

Speech marks provide a mapping between time and text. They inform the client when each word is spoken in the audio, enabling features like:

• Text highlighting during playback
• Precise audio seeking by text position
• Usage tracking and analytics
• Synchronization between text and audio

Two endpoints return them:

Use the streaming endpoint when you want highlighting to start before the synthesis has finished. Use the batch endpoint when you want a single response, or when you need a legacy model. POST /v1/audio/stream returns raw audio only and is unchanged.

Data structure

Speech marks use the following TypeScript interfaces:

1
type NestedChunk = {
2
  start_time: number  // Time in milliseconds when this chunk starts in the audio
3
  end_time: number    // Time in milliseconds when this chunk ends in the audio
4
  start: number       // Character index where this chunk starts in the original text
5
  end: number         // Character index where this chunk ends in the original text
6
  value: string       // The text content of this chunk
7
}
8
9
type SpeechMarks = NestedChunk & {
10
  chunks: NestedChunk[]  // Array of word-level chunks within this sentence/paragraph
11
}

Important considerations

• SSML escaping: Values are returned based on the SSML, so any escaping of &, < and > will be present in the value, start and end fields. Consider using the string tracker library to assist with mapping.

• Index gaps: The start and end values of each word may have gaps. When looking for a word at a specific index, check for start being >= yourIndex rather than checking if the index is within both start and end bounds.

• Timing gaps: Similarly, start_time and end_time of each word may have gaps. Follow the same approach as with index gaps.

• Initial silence: The start_time of the first word is not necessarily 0 like the SpeechMarks. There can be silence at the beginning of the sentence that leads to the word starting partway through.

• Trailing silence: The end_time of the last word does not necessarily correspond with the end of the SpeechMarks. There can be silence at the end that will make the SpeechMarks longer.

Example output

For the input "Hello, welcome to Speechify", the response includes:

1
const chunk: SpeechMarks = {
2
  start: 0,
3
  end: 27,
4
  start_time: 0,
5
  end_time: 1850,
6
  value: 'Hello, welcome to Speechify',
7
  chunks: [
8
    { start: 0,  end: 6,  start_time: 125,  end_time: 375,  value: 'Hello,' },
9
    { start: 7,  end: 14, start_time: 375,  end_time: 750,  value: 'welcome' },
10
    { start: 15, end: 17, start_time: 750,  end_time: 875,  value: 'to' },
11
    { start: 18, end: 27, start_time: 875,  end_time: 1850, value: 'Speechify' },
12
  ],
13
}

Note how start_time of the first word (125ms) doesn’t match the SpeechMarks start (0ms) - there’s initial silence before speech begins.

Streaming speech marks

POST /v1/audio/stream/with-timestamps takes the same request body as POST /v1/audio/stream and returns a Server-Sent Events stream instead of raw audio bytes.

$
curl -N -X POST https://api.sws.speechify.com/v1/audio/stream/with-timestamps \
>
  -H "Authorization: Bearer $SPEECHIFY_API_KEY" \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "input": "Hello world. This is a speech mark test.",
>
    "voice_id": "geffen_32",
>
    "model": "simba-3.2",
>
    "output_format": "mp3_24000_64"
>
  }'

Each event is named. speech.chunk carries a Base64-encoded run of audio, the speech marks that became final with it, or both:

event: speech.chunk
data: {"audio":"SUQzBAAAAAAA...","speech_marks":[{"type":"word","start":0,"end":5,"start_time":0,"end_time":469,"value":"Hello"}]}
event: speech.chunk
data: {"speech_marks":[{"type":"word","start":6,"end":12,"start_time":469,"end_time":1152,"value":"world."}]}
event: speech.done
data: {"billable_characters_count":40,"audio_duration_ms":4350}

The marks use the same fields as the batch endpoint, so one parser handles both.

Consuming the stream

• Concatenate the audio. Base64-decode each audio field and append it. The result is byte-for-byte what POST /v1/audio/stream would have returned.
• Apply marks against that one timeline. start_time and end_time are absolute milliseconds from the start of the synthesis. Which event a mark arrives on is a delivery detail and carries no meaning, so do not try to align a mark to the audio chunk it arrived with.
• A chunk may carry only one of the two. Marks lag their audio slightly, and the last chunk of a stream is often marks-only.
• Ignore unknown event types. New event types may be added; a client that skips names it does not recognize keeps working.
• There is no [DONE] sentinel. speech.done is the terminal event.

Errors

Before the stream starts, failures are ordinary HTTP responses with the standard error envelope. A request for a model that cannot produce speech marks fails here with 400 speech_marks_unsupported - use simba-3.0 or simba-3.2, or call POST /v1/audio/speech, which returns speech marks for every model.

Once the stream has started the status code is already committed, so a later failure arrives as a terminal event carrying the same envelope:

event: speech.error
data: {"error":{"code":"upstream_failure","message":"synthesis failed mid-stream"},"request_id":"..."}

Audio format

output_format and the Accept header select the codec exactly as they do on POST /v1/audio/stream. Because the response’s own Content-Type is text/event-stream, the media type of the audio inside the events is echoed on the Speechify-Audio-Content-Type response header.

Changing the codec or sample rate does not change the duration, so mark times stay correct for every output_format.