Skip to content

Speech To Text stt

These models do audio transcription with transformers.

STT models have task-type==lvcsr and filenames that by convention match stt-*.snsr

Some STT models support grammar-based recognition through grammar-stream. Grammar-based STT recognition uses the same grammar syntax as LVCSR, with STT-only support for <unknown/> and <dictation/>.

STT models included in this distribution.

Operation

flowchart TD
    start((start))
    fetch[/samples from ->audio-pcm/]
    audio(^sample-count)
    process[process]
    partial(^result-partial)
    intent(^nlu-intent)
    slot(^nlu-slot)
    result(^result)
    nlu{NLU<br>match?}

    slm{SLM<br>included?}
    generate[generate]
    slmstart(^slm-start)
    slmresultpartial(^slm-result-partial)
    slmresult(^slm-result)

    start --> fetch
    fetch --> audio
    audio --> process
    process --> fetch
    process -->|hypothesis| partial
    partial --> fetch
    process -->|VAD endpoint<br>or STREAM_END| nlu
    nlu -->|yes| intent
    nlu -->|no| result
    intent --> slot
    slot --> result
    slot -->|more| intent

    result --> slm
    slm -->|yes| slmstart
    slm -->|no| fetch
    slmstart -->|OK| generate
    slmstart -->|STOP| fetch
    generate -->|response| slmresultpartial
    slmresultpartial --> generate
    generate -->|done| slmresult
    slmresult --> fetch

Recognition flow.

  1. Read audio data from ->audio-pcm.
  2. Invoke ^sample-count.
  3. Invoke ^result-partial with interim recognition hypotheses every partial-result-interval ms.
  4. Continue processing until STREAM_END occurs on ->audio-pcm, one of the event handlers returns a code other than OK, or an external VAD detects a speech endpoint.
  5. If NLU is configured, invoke ^nlu-intent and ^nlu-slot for each top-level result that matches.
  6. Invoke ^result with the final recognition hypothesis.
  7. If an SLM is not available, resume processing at step 1.
  8. Invoke ^slm-start. If the handler returns STOP, resume processing at step 1.
  9. Invoke ^slm-result-partial as the model generates text.
  10. Invoke ^slm-result when text generation is complete.
  11. Resume processing at step 1.

Note

STT recognizers do not produce a final recognition hypothesis until they run out of audio samples to process, or an external VAD detects a speech endpoint.

With live audio you should use these with a VAD template such as tpl-vad-lvcsr, tpl-opt-spot-vad-lvcsr, or tpl-spot-vad-lvcsr.

Settings

^nlu-intent, ^nlu-slot, ^result, ^result-partial, ^sample-count, ^slm-result, ^slm-result-partial, ^slm-start

none

audio-stream, audio-stream-first, audio-stream-last

->audio-pcm, audio-stream-from, audio-stream-to, grammar-stream, nlu-grammar-stream

audio-stream-size, custom-vocab, partial-result-interval, samples-per-second, stt-profile

lvcsr

live-spot.c, snsr-eval.c, PhraseSpot.java, segmentSpottedAudio.java

STT grammar-based recognition 7.9.0

STT models that support grammar decoding accept grammar specifications through grammar-stream. See Grammar-based recognition for syntax, operators, NLU markup, special symbols, and weights.

STT grammars use runtime classes supplied through grammar-stream.classname or phrases-stream.classname, or defined in the grammar — including common and system classes imported as native grammar modules. Imported classes work in the same grammar on both STT and LVCSR; the legacy LVCSR-only binary class libraries are not used by STT.

Use <unknown/> in an STT grammar to allow an out-of-grammar span at a specific position. Use <dictation/> to hand off free-form speech to the STT model's statistical language component. This hand-off is one-way; matching does not return to the grammar afterward.

A command grammar

This small command grammar shows how to identify intents (actions) and entities (values) and hand off to free-form dictation for message text.

Because the <dictation/> hand-off is one-way — matching does not return to the grammar afterward — a dictation slot must be the last thing a branch matches. Each dictation slot in this grammar ({transcript <dictation/>}, {artist <dictation/>}, and {song <dictation/>}) sits at the end of its branch.

phone.grm
#SNSR 2.0
#
# phone.grm

digit = oh:0 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9;
number = $digit{10};

names = David | Deborah | Douglas | Gretchen | Heather
      | Jeffrey | Kathleen | Kurt | Lynda | Matthew | Pamela
      | Philip | Richard | Ronald | Sherri | Steven | Susan
      | Suzanne | Timothy;

# Use names as a class to allow runtime replacement
contact = ~names;

emergency = the? emergency hotline? | 9 1 1;
call = (call | dial | phone | place a call to)
       ({contact} | {number} | {emergency});
text = ( (send a? (message | text) to {contact})
       | ((message | text) {contact})
       | (send {contact} a (message | text)) )
       {transcript <dictation/>}?;
huddle = start a? huddle with {contact};
play = play music by {artist <dictation/>} | play/1 {song <dictation/>}; # (1)!
power = (power | turn) {state on | off}
      | {state (shut: | power:) down:off}
      | {state power: up:on};

intents = {call} | {huddle} | {power} | {play} | {text};

grammar = <s> $intents </s>;
  1. <dictation/> matching is greedy, so without a weight "play songs by …" would match play {song <dictation/>} and capture "songs by …" as the song. The /1 weight lowers the cost of the sole in-grammar terminal on the song branch, biasing the recognizer toward the play music by {artist <dictation/>} branch when the utterance fits it. See symbol weights.

Build a grammar-constrained model with snsr-edit:

% cd $HOME/Sensory/TrulyNaturalSDK/7.9.0

% bin/snsr-edit -t model/stt-enUS-general-medium-2.4.5-pnc.snsr \
    -o phone-control.snsr \
    -f grammar-stream phone.grm
Output written to "phone-control.snsr".

Run with snsr-eval:

% bin/snsr-eval -at phone-control.snsr -s partial-result-interval=0 # (1)!

# Say: Call eight oh oh five five five one two one two
NLU intent: call (0.0000) = call 8 0 0 5 5 5 1 2 1 2
NLU entity:   number (0.0000) = 8 0 0 5 5 5 1 2 1 2
  2250   6650 Call 8. 0 0 5 5 5 1 2 1. 2.

# Say: Send a message to Gretchen I'm on my way
NLU intent: text (0.0000) = send a message to Gretchen i'm on my way
NLU entity:   contact (0.0000) = Gretchen
NLU entity:   transcript (0.0000) = i'm on my way
 13870  16310 Send a message to Gretchen. I'm on my way.

# Say: Play music by Tom Waits
NLU intent: play (0.0000) = play music by tom waits
NLU entity:   artist (0.0000) = tom waits
 21820  23340 Play music by Tom Waits.

# Say: Shut down
NLU intent: power (0.0000) =  off
NLU entity:   state (0.0000) = off
 29210  29810 Off.
^C
  1. -a adds a VAD to find each utterance endpoint so the recognizer produces a final hypothesis from live audio (-t loads the model, so -at combines both). partial-result-interval= 0 shows only the final recognition hypothesis.

Replace the contact names list at runtime:

% bin/snsr-eval -at phone-control.snsr -s partial-result-interval=0 \
    -g phrases-stream.names "James; Jennifer; Mary; Michael; Patricia" # (1)!

# Say: Call James
NLU intent: call (0.0000) = call James
NLU entity:   contact (0.0000) = James
 18420  19180 Call. James.

# Say: Message Michael are you up for lunch today
NLU intent: text (0.0000) = message Michael are you up for lunch today
NLU entity:   contact (0.0000) = Michael
NLU entity:   transcript (0.0000) = are you up for lunch today
 28380  31260 Message Michael are you up for lunch today?
^C
  1. snsr-eval's -g option sets the phrases-stream.names stream from a string argument, replacing the names list defined in the grammar. A file can also be used. Because contact = ~names;, this overrides the class supplied to every {contact} slot.