--- source_path: "models/types/stt.md" canonical_url: "https://doc.sensory.com/tnl/7.9/models/types/stt/" --- # Speech To Text _(STT only)_ These models do audio transcription with transformers. STT models have [task-type](https://doc.sensory.com/tnl/7.9/api/setting-keys/configuration.md#task-type)` == `[lvcsr](https://doc.sensory.com/tnl/7.9/api/setting-keys/values.md#lvcsr) and filenames that by convention match `stt-*.snsr` Some STT models support [grammar-based recognition](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-based-recognition) through [grammar-stream](https://doc.sensory.com/tnl/7.9/api/setting-keys/runtime.md#grammar-stream). Grammar-based STT recognition uses the same grammar syntax as LVCSR, with STT-only support for [``](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-special) and [``](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-special). **Also see these related items:** [STT models](https://doc.sensory.com/tnl/7.9/models/index.md#stt-models) included in this distribution. ## Operation ```mermaid flowchart TD start((start)) fetch[/samples from ->audio-pcm/] audio(^sample-count) process[process] partial(^result-partial) intent(^nlu-intent) slot(^nlu-slot) result(^result) nlu{NLU
match?} slm{SLM
included?} generate[generate] slmstart(^slm-start) slmresultpartial(^slm-result-partial) slmresult(^slm-result) start --> fetch fetch --> audio audio --> process process --> fetch process -->|hypothesis| partial partial --> fetch process -->|VAD endpoint
or STREAM_END| nlu nlu -->|yes| intent nlu -->|no| result intent --> slot slot --> result slot -->|more| intent result --> slm slm -->|yes| slmstart slm -->|no| fetch slmstart -->|OK| generate slmstart -->|STOP| fetch generate -->|response| slmresultpartial slmresultpartial --> generate generate -->|done| slmresult slmresult --> fetch ``` Recognition flow. 1. Read audio data from [->audio-pcm](https://doc.sensory.com/tnl/7.9/api/setting-keys/runtime.md#-audio-pcm). 2. Invoke [^sample-count](https://doc.sensory.com/tnl/7.9/api/setting-keys/events.md#sample-count-event). 3. Invoke [^result-partial](https://doc.sensory.com/tnl/7.9/api/setting-keys/events.md#result-partial) with interim recognition hypotheses every [partial-result-interval](https://doc.sensory.com/tnl/7.9/api/setting-keys/configuration.md#partial-result-interval) ms. 5. Continue processing until [STREAM_END](https://doc.sensory.com/tnl/7.9/api/inference.md#rc_stream_end) occurs on [->audio-pcm](https://doc.sensory.com/tnl/7.9/api/setting-keys/runtime.md#-audio-pcm), one of the event handlers returns a code other than [OK](https://doc.sensory.com/tnl/7.9/api/inference.md#rc_ok), or an external [VAD](https://doc.sensory.com/tnl/7.9/api/setting-keys/values.md#vad) detects a speech endpoint. 6. If NLU is configured, invoke [^nlu-intent](https://doc.sensory.com/tnl/7.9/api/setting-keys/events.md#nlu-intent) and [^nlu-slot](https://doc.sensory.com/tnl/7.9/api/setting-keys/events.md#nlu-slot) for each top-level result that matches. 7. Invoke [^result](https://doc.sensory.com/tnl/7.9/api/setting-keys/events.md#result) with the final recognition hypothesis. 8. If an SLM is not available, resume processing at step 1. 9. Invoke [^slm-start](https://doc.sensory.com/tnl/7.9/api/setting-keys/events.md#slm-start). If the handler returns [STOP](https://doc.sensory.com/tnl/7.9/api/inference.md#rc_stop), resume processing at step 1. 10. Invoke [^slm-result-partial](https://doc.sensory.com/tnl/7.9/api/setting-keys/events.md#slm-result-partial) as the model generates text. 11. Invoke [^slm-result](https://doc.sensory.com/tnl/7.9/api/setting-keys/events.md#slm-result) when text generation is complete. 12. Resume processing at step 1. **Note:** STT recognizers do **not** produce a final recognition hypothesis until they run out of audio samples to process, or an external VAD detects a speech endpoint. With live audio you should use these with a VAD template such as [tpl-vad-lvcsr](https://doc.sensory.com/tnl/7.9/models/index.md#tpl-vad-lvcsr), [tpl-opt-spot-vad-lvcsr](https://doc.sensory.com/tnl/7.9/models/index.md#tpl-opt-spot-vad-lvcsr), or [tpl-spot-vad-lvcsr](https://doc.sensory.com/tnl/7.9/models/index.md#tpl-spot-vad-lvcsr). ## Settings **Available events:** [^nlu-intent](https://doc.sensory.com/tnl/7.9/api/setting-keys/events.md#nlu-intent), [^nlu-slot](https://doc.sensory.com/tnl/7.9/api/setting-keys/events.md#nlu-slot), [^result](https://doc.sensory.com/tnl/7.9/api/setting-keys/events.md#result), [^result-partial](https://doc.sensory.com/tnl/7.9/api/setting-keys/events.md#result-partial), [^sample-count](https://doc.sensory.com/tnl/7.9/api/setting-keys/events.md#sample-count-event), [^slm-result](https://doc.sensory.com/tnl/7.9/api/setting-keys/events.md#slm-result), [^slm-result-partial](https://doc.sensory.com/tnl/7.9/api/setting-keys/events.md#slm-result-partial), [^slm-start](https://doc.sensory.com/tnl/7.9/api/setting-keys/events.md#slm-start) **Available iterators:** _none_ **Available results:** [audio-stream](https://doc.sensory.com/tnl/7.9/api/setting-keys/results.md#audio-stream), [audio-stream-first](https://doc.sensory.com/tnl/7.9/api/setting-keys/results.md#audio-stream-first), [audio-stream-last](https://doc.sensory.com/tnl/7.9/api/setting-keys/results.md#audio-stream-last) **Available runtime settings:** [->audio-pcm](https://doc.sensory.com/tnl/7.9/api/setting-keys/runtime.md#-audio-pcm), [audio-stream-from](https://doc.sensory.com/tnl/7.9/api/setting-keys/runtime.md#audio-stream-from), [audio-stream-to](https://doc.sensory.com/tnl/7.9/api/setting-keys/runtime.md#audio-stream-to), [grammar-stream](https://doc.sensory.com/tnl/7.9/api/setting-keys/runtime.md#grammar-stream), [nlu-grammar-stream](https://doc.sensory.com/tnl/7.9/api/setting-keys/runtime.md#nlu-grammar-stream) **Available configuration settings:** [audio-stream-size](https://doc.sensory.com/tnl/7.9/api/setting-keys/configuration.md#audio-stream-size), [custom-vocab](https://doc.sensory.com/tnl/7.9/api/setting-keys/configuration.md#custom-vocab), [partial-result-interval](https://doc.sensory.com/tnl/7.9/api/setting-keys/configuration.md#partial-result-interval), [samples-per-second](https://doc.sensory.com/tnl/7.9/api/setting-keys/configuration.md#samples-per-second), [stt-profile](https://doc.sensory.com/tnl/7.9/api/setting-keys/configuration.md#stt-profile) **Available values:** [lvcsr](https://doc.sensory.com/tnl/7.9/api/setting-keys/values.md#lvcsr) **Also see these related items:** [live-spot.c](https://doc.sensory.com/tnl/7.9/api/sample/c/live-spot.md#live-spot-code), [snsr-eval.c](https://doc.sensory.com/tnl/7.9/api/sample/c/snsr-eval.md#snsr-eval-code), [PhraseSpot.java](https://doc.sensory.com/tnl/7.9/api/sample/android/enroll-trigger.md#et-code), [segmentSpottedAudio.java](https://doc.sensory.com/tnl/7.9/api/sample/java/segmentSpottedAudio.md#segmentspottedaudio-code) ## STT grammar-based recognition STT models that support grammar decoding accept grammar specifications through [grammar-stream](https://doc.sensory.com/tnl/7.9/api/setting-keys/runtime.md#grammar-stream). See [Grammar-based recognition](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-based-recognition) for syntax, operators, NLU markup, special symbols, and weights. STT grammars use runtime classes supplied through `grammar-stream.classname` or `phrases-stream.classname`, or defined [in the grammar](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-syntax-class) — including common and system classes imported as native grammar [modules](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-imports). Imported classes work in the same grammar on both STT and LVCSR; the legacy LVCSR-only binary [class libraries](https://doc.sensory.com/tnl/7.9/models/types/lvcsr.md#grammar-class-libraries-legacy) are not used by STT. Use `` in an STT grammar to allow an out-of-grammar span at a specific position. Use `` to hand off free-form speech to the STT model's statistical language component. This hand-off is one-way; matching does not return to the grammar afterward. ### A command grammar This small command grammar shows how to identify intents (actions) and entities (values) and hand off to free-form dictation for message text. Because the [``](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-special) hand-off is one-way — matching does not return to the grammar afterward — a dictation slot must be the last thing a branch matches. Each dictation slot in this grammar (`{transcript }`, `{artist }`, and `{song }`) sits at the end of its branch. **`phone.grm`** ```perl #SNSR 2.0 # # phone.grm digit = oh:0 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9; number = $digit{10}; names = David | Deborah | Douglas | Gretchen | Heather | Jeffrey | Kathleen | Kurt | Lynda | Matthew | Pamela | Philip | Richard | Ronald | Sherri | Steven | Susan | Suzanne | Timothy; # Use names as a class to allow runtime replacement contact = ~names; emergency = the? emergency hotline? | 9 1 1; call = (call | dial | phone | place a call to) ({contact} | {number} | {emergency}); text = ( (send a? (message | text) to {contact}) | ((message | text) {contact}) | (send {contact} a (message | text)) ) {transcript }?; huddle = start a? huddle with {contact}; play = play music by {artist } | play/1 {song }; # (1)! power = (power | turn) {state on | off} | {state (shut: | power:) down:off} | {state power: up:on}; intents = {call} | {huddle} | {power} | {play} | {text}; grammar = $intents ; ``` 1. `` matching is greedy, so without a weight "play songs by …" would match `play {song }` and capture "songs by …" as the song. The `/1` weight lowers the cost of the sole in-grammar terminal on the song branch, biasing the recognizer toward the `play music by {artist }` branch when the utterance fits it. See [symbol weights](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-syntax). Build a grammar-constrained model with [snsr-edit](https://doc.sensory.com/tnl/7.9/tools/snsr-edit.md#snsr-edit): ```console % cd $HOME/Sensory/TrulyNaturalSDK/7.9.0 % bin/snsr-edit -t model/stt-enUS-general-medium-2.4.5-pnc.snsr \ -o phone-control.snsr \ -f grammar-stream phone.grm Output written to "phone-control.snsr". ``` Run with [snsr-eval](https://doc.sensory.com/tnl/7.9/tools/snsr-eval.md#snsr-eval): ```console % bin/snsr-eval -at phone-control.snsr -s partial-result-interval=0 # (1)! # Say: Call eight oh oh five five five one two one two NLU intent: call (0.0000) = call 8 0 0 5 5 5 1 2 1 2 NLU entity: number (0.0000) = 8 0 0 5 5 5 1 2 1 2 2250 6650 Call 8. 0 0 5 5 5 1 2 1. 2. # Say: Send a message to Gretchen I'm on my way NLU intent: text (0.0000) = send a message to Gretchen i'm on my way NLU entity: contact (0.0000) = Gretchen NLU entity: transcript (0.0000) = i'm on my way 13870 16310 Send a message to Gretchen. I'm on my way. # Say: Play music by Tom Waits NLU intent: play (0.0000) = play music by tom waits NLU entity: artist (0.0000) = tom waits 21820 23340 Play music by Tom Waits. # Say: Shut down NLU intent: power (0.0000) = off NLU entity: state (0.0000) = off 29210 29810 Off. ^C ``` 1. `-a` adds a [VAD](https://doc.sensory.com/tnl/7.9/api/setting-keys/values.md#vad) to find each utterance endpoint so the recognizer produces a final hypothesis from live audio (`-t` loads the model, so `-at` combines both). [partial-result-interval](https://doc.sensory.com/tnl/7.9/api/setting-keys/configuration.md#partial-result-interval)` = 0` shows only the final recognition hypothesis. Replace the contact names list at runtime: ```console % bin/snsr-eval -at phone-control.snsr -s partial-result-interval=0 \ -g phrases-stream.names "James; Jennifer; Mary; Michael; Patricia" # (1)! # Say: Call James NLU intent: call (0.0000) = call James NLU entity: contact (0.0000) = James 18420 19180 Call. James. # Say: Message Michael are you up for lunch today NLU intent: text (0.0000) = message Michael are you up for lunch today NLU entity: contact (0.0000) = Michael NLU entity: transcript (0.0000) = are you up for lunch today 28380 31260 Message Michael are you up for lunch today? ^C ``` 1. [snsr-eval](https://doc.sensory.com/tnl/7.9/tools/snsr-eval.md#snsr-eval)'s `-g` option sets the [phrases-stream](https://doc.sensory.com/tnl/7.9/api/setting-keys/runtime.md#phrases-stream)`.names` stream from a string argument, replacing the `names` list defined in the grammar. A file can also be used. Because `contact = ~names;`, this overrides the class supplied to every `{contact}` slot. *[API]: Application Programming Interface *[LVCSR]: Large Vocabulary Continuous Speech Recognition model, feed-forward neural net acoustic model with FST decoder *[NLU]: Natural Language Understanding model *[SLM]: Generative Small Language Model *[STT]: Speech To Text: transformers with language model and CTC decoding *[TNL]: TrulyNatural, Sensory's large-vocabulary speech recognition technology *[VAD]: Voice Activity Detector