---
source_path: "reference/grammar.md"
canonical_url: "https://doc.sensory.com/tnl/7.9/reference/grammar/"
---
# Grammar-based recognition _(TrulyNatural only)_ _(STT only)_
Grammar-based recognition constrains a recognizer to a set of words and
structures defined by a grammar. Focusing recognition on a limited set of
phrases can improve speed and accuracy at the expense of recognizing arbitrary
input.
LVCSR models support grammar-based recognition with build-capable models. STT
models that support grammar decoding use the same grammar syntax. Some special
symbols are model-specific; those cases are noted below.
## Syntax
A [context-free grammar] is a set of rules that describes the sequences of
words that a recognizer can match.
### Definition
1. Grammars use [UTF-8][] encoding.
1. `#` marks the start of a comment, which extends to the end of the line.
1. A grammar may optionally begin with a `#SNSR` identifier line
(optionally with a version, for example `#SNSR 2.0`), mirroring the header that
distinguishes [BNF+EM](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-bnfplus) and [SRGS](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-srgs) grammars.
A grammar without the header is still recognized as native. Because a bare
`#` line is a comment, the identifier is recognized only as the exact `#SNSR`
token on the first line; any other `#...` first line remains a comment.
When a version is present it must be written as `MAJOR.MINOR` (for example
`2.0`), optionally followed by whitespace. The compiler accepts a grammar
whose declared version is less than or equal to the version it implements,
comparing major and minor in turn: a `2.0` build accepts `1.9` and `2.0`
but rejects `2.1` and `3.0`. Declaring a newer version than the compiler
supports is an error, so a grammar that relies on features from a later
dialect fails clearly instead of misparsing. A malformed version (anything
other than `MAJOR.MINOR`) is also an error. A bare `#SNSR` with no version
is treated as the current version. Grammars emitted in native SNSR format
(for example by reading back [`grammar-stream`](https://doc.sensory.com/tnl/7.9/api/setting-keys/runtime.md#grammar-stream)) are stamped
with the current `#SNSR 2.0` header.
1. A _grammar_ is a series of _rules_ representing variable definitions.
The final rule in a grammar specifies the recognition vocabulary and typically
references rules defined earlier. It should include the sentence start (``)
and end (``) markers.
1. A _rule_ is an assignment of the form `name = expr ;` where
`name` is a _symbol_ and `expr` is a sequence of _symbols_ and _operators_.
`expr` is a type of [regular expression][].
1. A _symbol_ is a sequence of characters that does not include any whitespace
or operators, optionally prefixed by sigils `$` or `~`. A symbol without a
sigil is called a _terminal_ and is part of the recognition vocabulary,
for example `temperature`. [Special symbols](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-special) are predefined
terminals that describe input characteristics such as pauses and the edges of
an utterance.
1. The `$` sigil does rule substitution _at build time_. The parser substitutes
the value of the rule named `name` for `$name`. Substitutions include an
implicit _grouping_ operator: Grammar `a = 1 | 2 | 3; b = $a ;`
is equivalent to `b = (1 | 2 | 3) ;`.
1. The `~` sigil substitutes a named recognition
class _at runtime_.
- Each class is a recognizer with its own grammar, separate from the main
grammar.
- All references to a class use instances of the same class recognizer.
- You can update each class in isolation, without having to recompile the
main grammar.
- If you have a large rule that's referenced multiple times, converting it
to a class can speed up build time significantly.
- Use classes to augment a recognition vocabulary at runtime. In a voice
dialing application, for example, you can define the entire recognition
grammar at build time but use `~contacts` instead of a predefined list of
contact names. Once loaded, the application can scan the address book and
build only the `~contacts` class.
- Specify class definitions with [grammar-stream.classname](https://doc.sensory.com/tnl/7.9/api/setting-keys/runtime.md#grammar-stream)
or [phrases-stream.classname](https://doc.sensory.com/tnl/7.9/api/setting-keys/runtime.md#phrases-stream), for example
`phrases-stream.contacts`.
- A class may also be defined _in the grammar itself_. If a
rule is referenced with the `~` sigil, the parser compiles that rule into
the class recognizer for that name. The same rule referenced with `$` is
still substituted inline; a rule referenced with `~` becomes a class.
With this grammar:
```
number = 18 | 643 | 20 | 6;
place = target | winco | susan's house;
g = directions to {place ~place} | start a timer for ~number minutes ;
```
`~number` and `~place` are recognition classes built from the `number` and
`place` rules, with no external stream required. This is a convenient
default; it does not change the runtime nature of a class. A `~name` that
does _not_ match a defined rule remains an undefined class to be supplied
at runtime, exactly as before.
- This changes an edge case of earlier behavior. Previously, if a
grammar both defined a rule `a` and referenced `~a`, the class `~a`
was left undefined until supplied at runtime. Now `~a` defaults to the
`a` rule body. For example, in `a = 1 | 2 | 3; b = say ~a ;`
the class `~a` is now `1 | 2 | 3` rather than empty. A runtime stream
still overrides it, so grammars that supply `~a` externally are
unaffected.
- When a class has both an in-grammar definition and an
external definition, the external definition wins. A class is resolved in
this order of precedence:
1. An external [grammar-stream.classname](https://doc.sensory.com/tnl/7.9/api/setting-keys/runtime.md#grammar-stream) or
[phrases-stream.classname](https://doc.sensory.com/tnl/7.9/api/setting-keys/runtime.md#phrases-stream) set at runtime.
1. An in-grammar definition (the rule referenced with `~`), including a
rule pulled in from another module by [import](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-imports).
This makes an in-grammar definition an overridable default: existing
grammars that rely on external streams are unaffected, and an application
can still override any class at runtime.
- Common and system classes
(for example dates, times, numbers, and phone numbers, conventionally
named with an `s.` prefix) are available as ordinary native grammar
_modules_.
The SDK ships them as a single grammar, `system.grm`, under
`data/grammars//`. Reference the classes you need and import
their definitions with `import "system.grm";` and the compiler flattens
them into your grammar, so a `~s.*` reference resolves to an in-grammar
definition like any other, and classes you do not reference are dropped.
See [modules and imports](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-imports) for the `import` / `from` /
`export` directives and the aggregator pattern used to build such a
library. Imported classes work in the same grammar on both STT and LVCSR.
- Older LVCSR models can also load a pre-built binary _class library_
(a separate `.snsr` repository of classes). This mechanism is
**deprecated**: it works only with LVCSR models, and Sensory no longer
publishes new ones. For new grammars, import class definitions
instead, so the same grammar runs on both STT and LVCSR. When a binary
class library is loaded, it is consulted only for a `~s.*` class that
is otherwise undefined, after the precedence above.
- A `~name` class that is referenced but never defined -- not
in the grammar, not through an external stream, and (on an older LVCSR
model) not in a loaded binary class library -- now produces a warning
when the grammar is built or run.
Because every class can be filled at runtime this is only advisory, not
an error, but it usually indicates an oversight. To keep a class
intentionally empty (so it matches nothing until supplied at runtime)
without the warning, define it as [``](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-special), for
example `city = ;`. A [system class](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-syntax-class)
(`~s.*`) that is referenced but undefined remains a hard error, since it
cannot be supplied by the application.
1. Operators include _grouping_ parentheses, brackets, and braces, _infix_
operators that indicate logical AND and OR between symbols, and _postfix_
operators that change how the preceding symbol matches input. The
[operator precedence](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-op-precedence) table lists the order and
direction in which the parser applies operators.
1. Grouping
- `( )` Parentheses enclose items that are grouped together.
- `[ ]` Square brackets enclose optional items.
`[...]` is equivalent to `(...)?`.
- `{ }` Braces implement slot-capturing lightweight NLU
markup.
- `{slotName a b c}` makes `a b c` available as the
[nlu-slot-value](https://doc.sensory.com/tnl/7.9/api/setting-keys/results.md#nlu-slot-value) of [nlu-slot-name](https://doc.sensory.com/tnl/7.9/api/setting-keys/results.md#nlu-slot-name) `slotName` when the recognizer
matches `a b c` to the input audio.
- You can nest NLU slots to an arbitrary depth.
- The outermost slots are defined as [intents](https://doc.sensory.com/tnl/7.9/api/setting-keys/results.md#nlu-intent-name) and all
the nested slots in each intent as [entities](https://doc.sensory.com/tnl/7.9/api/setting-keys/results.md#nlu-entity-name).
- Each identified intent invokes handlers registered for
[^nlu-intent](https://doc.sensory.com/tnl/7.9/api/setting-keys/events.md#nlu-intent) and [^nlu-slot](https://doc.sensory.com/tnl/7.9/api/setting-keys/events.md#nlu-slot).
- `{rule}` is shorthand for `{rule $rule}`.
- With this grammar:
```
seconds = 1 | 2 | 4 | 8 | half:0.5 a:? | a:? quarter:0.25 [of: a:];
shutterSpeed = set shutter speed to {seconds} ( second | seconds );
cmd = {shutterSpeed} ;
```
an utterance of "set shutter speed to a quarter of a second" will
produce `set shutter speed to 0.25 second` as recognition output, with
an additional [^nlu-intent](https://doc.sensory.com/tnl/7.9/api/setting-keys/events.md#nlu-intent) callback for the top-level
`shutterSpeed` slot:
```
NLU intent: shutterSpeed (0.0000) = set shutter speed to 0.25 second
NLU entity: seconds (0.0000) = 0.25
```
1. Infix operators
- These are valid between symbols and may be surrounded by whitespace.
- `^` is the conjunction operator and is implied between adjacent terminals:
Grammar `g = one two three;` will recognize only the sequence
"one two three".
- `|` is the disjunctive operator. It separates alternative items.
Grammar `g = one | two | three;` will recognize "one", or "two", or
"three".
1. Postfix operators
- These directly follow a symbol without any intervening whitespace.
- `?` A question mark following a symbol makes that symbol optional:
It requires zero or one repetitions of the symbol.
- `+` A plus sign following a symbol or a group requires one or more
repetitions of it.
- `*` An asterisk following a symbol or a group requires zero or more
repetitions.
- `{m,n}` A brace-enclosed count following a symbol or a group
is a _bounded repeat_: it requires between `m` and `n` repetitions
(inclusive). Related forms are `{m}` for exactly `m`, `{m,}` for `m` or
more, and `{,n}` for up to `n`. Whitespace inside the braces is ignored,
so `x{2,3}` and `x{ 2 , 3 }` are equivalent. Because a leading `{`
instead begins slot-capturing markup (see below), the bounded-repeat
form is recognized only in postfix position and only when its body is
numeric: `x{2}` repeats `x`, whereas `{x}` is a slot.
- `:` is the rewrite operator.
- `left:right` recognizes symbol `left` but produces terminal `right`
as a recognition result.
- `left:` recognizes symbol `left` but rewrites that to an empty
string, eliding `left` from the recognition result.
- `:right` inserts `right` into the recognition result.
If you say "one two three", grammar
`g = one :mississippi two :mississippi three ;` produces
"one mississippi two mississippi three".
- `/` A forward slash following a symbol followed by a floating point number
defines a weight to be associated with that symbol. If there's a rewrite
operator (`:`) the slash must follow the rewritten-to terminal, for
example: `one:een/0.123` Weights are in the logprob domain, convert from a
$[0, 1]$ probability to a weight with $w = -log_{10}(p)$.
The default symbol weight is `0` for a probability of `1.0`.
- `( ... )/w` A weight suffix may also follow a group `(...)`
or optional group `[...]`, with no space before the slash. This
attaches the weight to the whole group. Used on an alternative branch,
for example `g = (turn it on) | (turn it off)/2.3;`, it biases the
recognizer toward or away from that branch without changing which
utterances match.
1. `\` escape symbol. To include a literal special character in a grammar
specification, escape it with a backslash. The list of characters that support
this include: `^`, `|`, `*`, `+`, `?`, `=`, `[ ]`, `( )`, `;`, `#`, and `:`.
**Also see these related items:** [grammar-stream](https://doc.sensory.com/tnl/7.9/api/setting-keys/runtime.md#grammar-stream), [phrases-stream](https://doc.sensory.com/tnl/7.9/api/setting-keys/runtime.md#phrases-stream), [nlu-grammar-stream](https://doc.sensory.com/tnl/7.9/api/setting-keys/runtime.md#nlu-grammar-stream),
[^nlu-intent](https://doc.sensory.com/tnl/7.9/api/setting-keys/events.md#nlu-intent), [^nlu-slot](https://doc.sensory.com/tnl/7.9/api/setting-keys/events.md#nlu-slot)
### Modules and imports
A native grammar can be split across several files and pull rules from other
_modules_ with `import` directives. This keeps large grammars maintainable and
lets you share common rule sets (for example a set of number, date, or place
rules) between grammars.
1. Directives end with `;` and may appear anywhere among the rules. A grammar
that uses no directives compiles exactly as before, so this feature is fully
backward compatible.
1. A _module_ is another grammar file. Its **default export** is its final rule
— the same rule that is the recognition vocabulary when that file is
compiled on its own.
1. `import "module";` is a **verbatim import**: every name the module exports
— and its default export — is brought in **unchanged**, under those
same names. There is no filename-derived namespace: a module that exports
`s.phone-number` is referenced as `$s.phone-number`. Dots are ordinary name
characters, not a namespace separator. `import "module" as ns;` is the same
but prefixes every imported name with `ns.` (so `digit` becomes `$ns.digit`).
1. `from "module" import a, b as c;` selectively imports the exported rules `a`
and `b`, binding `b` to the local name `c`. The imported names are then used
as written (`$a`, `$c`).
1. `export name, ...;` marks local rules as visible to importing grammars, and
can also **re-export** an imported name. `export prefix*;` is a glob that
exports every visible name — local rule or imported name — that
begins with `prefix`. Exports are resolved after the whole file is parsed, so a
directive's position does not matter. A rule that is neither exported nor the
default export is **private** to its module. Exporting an undefined name is an
error; a glob that matches nothing is allowed.
1. Re-export is a **passthrough**: if a module imports a name and re-exports it,
a consumer resolves that name straight to the module that actually defines it.
This makes the **aggregator** pattern work — a module that only imports
many sub-modules and re-exports them (`export s.*;`), with no rules of its own,
is fully transparent. A module may consist of directives alone.
1. Imported rules behave exactly like local rules: a `$` reference is inlined at
build time, and a `~` reference becomes an [in-grammar class](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-syntax-class)
(convert-on-reference), keyed by the reference as written (for example
`~s.digit`). Class resolution is *set-wide*: a `~name` that does not resolve
through the referencing module's own imports is matched against any visible
rule of that name anywhere in the imported set, so one bundled module can use
a class defined by a sibling (for example a phone-number module that references
`~s.single-digit-integer`) as long as an aggregator has pulled both in. This
set-wide fallback applies to `~` classes only, not to `$` inlining. All the
class-resolution precedence rules above still apply, so an external stream can
still override an imported class.
1. Module references resolve as follows: `"/abs/path"` is absolute; `"./rel"`
and `"../rel"` are relative to the importing file's directory; and a bare
`"name"` is searched for on the loader's import path. The reference is used
verbatim as a filename — no extension is added or assumed, so name your
modules and write the references however you like (the `.grm` suffix used in
the examples below is only a convention). When you compile a grammar with the
Sensory SDK tools, the import path is the current working directory, so an
importing grammar can refer to a sibling module by name. Embedders that link
the SDK can install their own loader (for example one backed by a model
bundle) to resolve modules however they wish.
1. Import cycles (a module that transitively imports itself) are reported as
errors, as is import nesting deeper than 32 levels. A module reached by more
than one import path (a diamond) is *not* a cycle: it is fetched once and
shared. If two modules contribute the same name resolving to different rules,
that collision is reported as an error.
For example, with a shared `numbers.grm` module:
```
# numbers.grm
export digit;
digit = 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9;
```
a dialing grammar in the same directory can import it:
```
import "numbers.grm";
g = dial ~digit ~digit ~digit ~digit ;
```
Here `~digit` becomes a recognition class built from the imported `digit`
rule, with no external stream required. To keep imported names grouped under a
namespace, import `as`:
```
import "numbers.grm" as num;
g = dial ~num.digit ~num.digit ~num.digit ~num.digit ;
```
An **aggregator** collects many modules and re-exports their rules, so a
consumer imports a single file. Here `export *;` re-exports every name the
aggregator imported (use `export prefix*;` to re-export only those sharing a
prefix):
```
# slots.grm (no rules of its own)
import "phone-number.grm";
import "color.grm";
export *;
```
```
import "slots.grm";
g = call $phone-number ;
```
This aggregator pattern is how common and system classes are provided as a
grammar rather than as a binary [class library (legacy)](https://doc.sensory.com/tnl/7.9/models/types/lvcsr.md#grammar-class-libraries-legacy).
The SDK ships the system classes as a single self-contained grammar,
`system.grm`, under `data/grammars//` (for example
`data/grammars/en-US/system.grm`). Where an older LVCSR grammar relied on a
loaded binary class library for its `~s.*` classes:
```
# Legacy: ~s.phone-number resolved from a loaded binary class library.
g = call {number ~s.phone-number} ;
```
import the system grammar instead, and the compiler flattens in the definitions
of the `~s.*` classes you reference (unused classes are dropped):
```
import "system.grm";
g = call {number ~s.phone-number} ;
```
The imported version works on both STT and LVCSR, needs no separate model
loaded into the session, and produces a self-contained grammar when read back
through [grammar-stream](https://doc.sensory.com/tnl/7.9/api/setting-keys/runtime.md#grammar-stream). Models included in the SDK distribution are
configured so that a bare `import "system.grm";` resolves against the shipped
`data/grammars//` directory; for other models, add that directory to
[grammar-import-path](https://doc.sensory.com/tnl/7.9/api/setting-keys/configuration.md#grammar-import-path) or import by absolute path.
### Language declaration
A native grammar may declare its language with a `language` directive:
```
language en-US;
g = hello ;
```
1. The tag is a subset of [BCP 47][]: a two- or three-letter language code with
an optional two-letter region, written `xx`, `xxx`, `xx-YY`, or `xxx-YY` (for
example `en`, `eng`, or `en-US`). It is matched case-insensitively and stored
canonically as a lowercase language and uppercase region, so `EN` is
normalized to `en`, and `en-us` and `EN-US` are normalized to `en-US`. Longer
languages or additional subtags (scripts, variants) are not accepted.
1. Like the other directives it ends with `;` and may appear anywhere among the
rules; unlike `import`/`from`/`export` it needs no module loader. The first
declaration wins. A second declaration is fine if it names the same language,
but a conflicting one is an error. A module's declaration is carried onto a
flattened grammar, and modules that disagree on language are an error.
1. The declaration is advisory: it does not change the words a grammar accepts.
A build may use it to sanity-check the grammar against the acoustic model's
language. It is preserved across the native and SRGS formats; converting to
BNF+ drops it (BNF+ has no meaningful language declaration).
### Operator precedence
The following table lists the precedence and associativity of grammar
operators. Operators are listed in descending precedence: level `0`
is applied first and level `5` last.
Precedence | Operator | Description | Associativity
:---------:|:--------:|-------------|--------------
0 | `:` | Rewrite output
0 | `/` | Symbol weight
0 | `(…)/w`| Group weight (suffix)
1 | `( )` | Grouping
1 | `[ ]` | Optional group
1 | `{ }` | Slot-capturing semantic markup
2 | `?` | Zero-or-one symbol | left-to-right
2 | `+` | One-or-more symbols | left-to-right
2 | `*` | Zero-or-more symbols | left-to-right
2 | `{m,n}`| Bounded repeat (m..n) | left-to-right
3 | `^` | And, implied between symbols | right-to-left
4 | `|` | Alternative | right-to-left
5 | `=` | Rule assignment | right-to-left
This grammar:
```
a = one | two three four;
g = ( $a | five six) ;
```
will recognize only these phrases:
```
one
two three four
five six
```
### Special symbols
A grammar can include these special symbols:
- `` - The silence at the start of a sentence.
- `` - The silence at the end of a sentence.
- `` - Short pauses between words. The grammar compiler automatically
adds these where needed, so there is no need to do so explicitly.
Do **not** add `` to NLU grammars, use `` instead.
- `` - An explicit short pause.
- `` _(TrulyNatural only)_ - Matches when none of the alternatives are likely
(i.e. "none of the above").
+ Recognition results at the phrase level can include `` even
if this symbol was not explicitly used in the grammar. This is an
indication that the result was rejected due to [search.frame-nota](https://doc.sensory.com/tnl/7.9/api/setting-keys/configuration.md#searchframe-nota), or
that RAM or CPU constraints limited the recognizer's ability to produce a
result.
- `` - Never matches; an unmatchable alternative that no input can
traverse. Use it as a placeholder branch, for example `g = ( |
yes | no );`, where a rule may later be filled in. It is the native
equivalent of BNF+EM `` and SRGS `$VOID`.
- `` _(TrulyNatural only)_ - Similar to ``. In *some* LVCSR models the
threshold for determining whether this symbol matches better than any other
is different from that of ``.
+ _(STT only)_ In STT grammars, `` matches an out-of-grammar word span at
a specific point. Use this when the grammar should keep matching even if
part of the utterance is not in the fixed vocabulary. In word-level STT
models, `` expands to the primitive wildcard sequence
` *`. In character-level STT models, it
expands to `( | )`.
+ _(STT only)_ By default, `` has an OOV penalty so in-grammar words win
when they fit the audio. Add an explicit symbol weight, for example
`/2`, to tune this penalty in a grammar.
+ _(STT only)_ In NLU output and in the final recognition result, the matched word is
emitted unless the grammar uses a rewrite. Use `:` to match and
drop it from these results.
- `` / `` _(STT only)_ - Primitive OOV wildcard symbols
used by ``. Use them directly only when you need explicit control
over word-boundary matching or continuation weighting.
- `` _(STT only)_ - Hands off recognition to the STT model's statistical
language component for free-form speech. This is a one-way hand-off; matching
does not return to the grammar afterward.
- `.` - When used with lightweight [NLU grammars](https://doc.sensory.com/tnl/7.9/api/setting-keys/runtime.md#nlu-grammar-stream), a single
period matches any input word. If `.` is also used as the output label, the
matched input word is echoed. Use `.:*` to match any input words and remove
them from the NLU result.
### Out-of-grammar words in STT grammars
Use `` to allow an STT grammar slot to absorb words outside the fixed
grammar vocabulary. Add the empty rewrite operator (`:`) when the grammar should
match the unknown word but omit it from the captured value:
```text
digit = one | two | three | four | five | six | seven | eight | nine | zero | :;
g = {number $digit+} ;
```
A non-zero weight overrides the default OOV penalty for that arc:
```text
item = /2;
```
### Formal syntax
The following [ABNF][] (per [RFC 5234][]) describes the surface syntax of the
native grammar language: which sequences of tokens are well formed. It is a
convenience for authors building editor support or validation tooling and is
not a substitute for the descriptions above.
Two things it deliberately does not express:
- **Operator precedence and associativity.** ABNF cannot capture these, so the
`expr` and `concatenation` rules below are intentionally permissive. The
binding of operators is fixed by the [operator precedence](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-op-precedence)
table, which is authoritative.
- **Static validation.** Constraints such as unique rule names, resolvable
`$name` references, import resolution (export visibility, cycles, and the
import-depth limit), and a non-empty start rule are enforced by the compiler,
not by this grammar.
Grammars are [UTF-8][] encoded. The character classes below enumerate ASCII for
readability; any non-ASCII UTF-8 code point is also an unreserved character.
```abnf
; ---- top level ----
; An optional "#SNSR" identifier line may precede the rules. Rules and module
; import directives may be interleaved freely.
grammar = [ snsr-header ] *ws *( ( rule / directive ) *ws )
; "#SNSR" (case-insensitive), optionally followed by a version token, on its
; own line at the very start (after an optional UTF-8 BOM). A version, when
; present, must be MAJOR.MINOR and must be <= the version the compiler
; implements (a newer version is an error). Any other "#..." first line is an
; ordinary comment.
snsr-header = "#SNSR" [ 1*WSP snsr-version ] *WSP ( newline / end-of-input )
snsr-version = 1*DIGIT "." 1*DIGIT
rule = rule-name *ws "=" *ws expr *ws ";"
rule-name = name
; ---- module imports (:material-tag-outline: [7.9.0)][v7.9.0)]{ data-preview } ----
; Directives are recognized contextually: a leading "import"/"from"/"export"
; token NOT immediately followed by "=" (which would make it a rule). A
; module's default export is its final rule; only exported rules and the
; default export are visible to importing grammars.
directive = import-dir / from-dir / export-dir / language-dir
; "import "m";" imports every exported name (and the default export) VERBATIM;
; "import "m" as ns;" prefixes each imported name with "ns." ($ns.rule).
import-dir = "import" 1*ws module-ref [ 1*ws "as" 1*ws name ] *ws ";"
; "from "m" import a, b as c;" selectively imports exported rules, optionally
; renaming each with "as".
from-dir = "from" 1*ws module-ref 1*ws "import" 1*ws
import-item *( *ws "," *ws import-item ) *ws ";"
import-item = name [ 1*ws "as" 1*ws name ]
; "export a, b;" exports local rules or re-exports imported names; "export s.*;"
; is a prefix glob over every visible name (resolved after the file is parsed).
export-dir = "export" 1*ws export-item *( *ws "," *ws export-item ) *ws ";"
export-item = name [ "*" ]
; "language en-US;" declares the grammar language (a BCP 47 subset: 2- or
; 3-letter language + optional 2-letter region), case-insensitive and normalized.
; It needs no loader; first wins, a conflicting redeclaration is an error.
language-dir = "language" 1*ws lang-tag *ws ";"
lang-tag = (2ALPHA / 3ALPHA) [ "-" 2ALPHA ]
; A module reference is a quoted string (recommended) or a bare token: "/abs"
; is absolute; "./rel" and "../rel" are relative to the importing file; a bare
; "name" is searched on the loader's import path. The reference is a filename
; used verbatim -- no extension is added or assumed.
module-ref = quoted-ref / bare-ref
quoted-ref = DQUOTE *( %x20-21 / %x23-FF ) DQUOTE ; any char except '"'
bare-ref = name
; ---- expressions ----
; One or more alternatives separated by "|".
expr = concatenation *( *ws "|" *ws concatenation )
; A sequence of terms. Adjacent terms concatenate implicitly (whitespace);
; the explicit "^" conjunction is equivalent to juxtaposition.
concatenation = term *( ( 1*ws / *ws "^" *ws ) term )
; A primary optionally followed by postfix repetition operators. Postfix
; operators immediately follow the operand with no intervening whitespace.
; A "{" in postfix position with a purely numeric body is a bounded repeat;
; a "{" that begins a term is instead slot-capturing markup (see "slot").
term = primary *postfix-op
postfix-op = "?" / "+" / "*" / bounded-repeat
; Whitespace inside the braces is ignored. "{m}" is exactly m; "{m,}" is m or
; more; "{,n}" is up to n; "{m,n}" is between m and n (inclusive).
bounded-repeat = "{" *ws ( 1*DIGIT [ *ws "," *ws *DIGIT ]
/ "," *ws 1*DIGIT ) *ws "}"
primary = group / optional-group / slot / symbol
; A group (and optional group) may carry a trailing "/w" weight suffix, with no
; space before the slash; on an alternation branch this weights the branch.
group = "(" *ws expr *ws ")" [ weight ]
; "[ ... ]" is equivalent to "( ... )?".
optional-group = "[" *ws expr *ws "]" [ weight ]
; "{ slotName a b c }" captures "a b c" as slot "slotName";
; "{ rule }" is shorthand for "{ rule $rule }". Slots may nest.
slot = "{" *ws [ slot-name 1*ws ] expr *ws "}"
slot-name = name
; ---- symbols ----
; Any symbol may carry an optional rewrite and/or weight. A bare "/float"
; is an epsilon symbol carrying only a weight.
symbol = recognized [ rewrite ] [ weight ]
/ rewrite [ weight ]
/ weight
recognized = rule-ref / class-ref / terminal
; "$name" is a build-time rule substitution; "~name" is a runtime recognition
; class. Dots in "name" are ordinary characters (e.g. "$s.phone-number"). With
; module imports, "name" may be a verbatim imported name, an "as ns"-prefixed
; name, or a name bound by a selective "from ... import".
rule-ref = "$" name ; build-time rule substitution
class-ref = "~" name ; runtime recognition-class substitution
terminal = name ; a word in the recognition vocabulary
; "left:right" recognizes left, emits right; "left:" emits nothing;
; ":right" is a pure insertion (no recognized input).
rewrite = ":" [ name ]
; Weight in the log-prob domain (w = -log10(p)); default 0.
weight = "/" float
; ---- names and lexical primitives ----
; ":" and "/" are ordinary name characters; they take on their rewrite/weight
; meaning only when a symbol is decomposed. A literal ":" is written "\:".
name = 1*name-char
name-char = unreserved / escaped
; Any single character that is not reserved. Reserved characters are
; whitespace, ";", "#", and the operator/grouping characters
; ^ | * + ? = [ ] ( ) { }. Any non-ASCII UTF-8 code point is also unreserved.
unreserved = %x21-22 / %x24-27 / %x2C-3A / %x3C / %x3E / %x40-5A
/ %x5C / %x5F-7A / %x7E / %x80-10FFFF
; A backslash-escaped reserved character. The backslash is removed.
escaped = "\" escapable
escapable = "^" / "|" / "*" / "+" / "?" / "=" / "[" / "]"
/ "(" / ")" / ";" / "#" / ":" / "{" / "}"
float = [ sign ] ( 1*DIGIT [ "." *DIGIT ] / "." 1*DIGIT ) [ exponent ]
exponent = ( "e" / "E" ) [ sign ] 1*DIGIT
sign = "+" / "-"
; ---- whitespace and comments ----
; A "#" begins a comment that runs to end of line. Newlines are ordinary
; whitespace and do not terminate a rule; only ";" does.
ws = WSP / newline / comment
comment = "#" *( %x00-09 / %x0B-FF ) ( newline / end-of-input )
newline = CR / LF / CRLF
end-of-input = ""
; Core rules from RFC 5234 Appendix B: WSP, CR, LF, CRLF, DIGIT, DQUOTE.
```
The predefined [special symbols](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-special) (``, ``, ``,
``, and so on) lex as ordinary `terminal`s.
## BNF+EM v2.1 grammars
In addition to the native syntax described above, the SDK accepts grammars
written in the [BNF+EM v2.1][] format used by other speech vendors. Support for
this format eases migration of existing Cerence-style grammars.
### Format detection
A grammar is parsed as BNF+EM when its first non-comment content is a
`#BNF+EM` header, for example `#BNF+EM V2.1;`. Otherwise the [native
syntax](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-syntax) is used. A single grammar is entirely one format or the
other; you cannot mix the two.
```text
#BNF+EM V2.1;
!grammar lights;
!start ;
: ( turn ) ( on | off ) [ the ] ( | living room ) lights ;
```
### Supported subset
The SDK compiles the subset of BNF+EM most commonly used in command-and-control
grammars. Each construct maps onto an equivalent in the native syntax, so its
recognition behavior matches the corresponding native form:
BNF+EM | Native equivalent | Notes
-------|-------------------|------
`` reference | `$rule` | Rule substitution.
`!grammar NAME ;` | — | Names the grammar. Quoted names are accepted.
`!start ;` | final rule | Declares the single entry-point rule.
`` | `:` | Matches with no speech (epsilon).
`` | `` | Never matches (an unmatchable alternative).
`a | b` | `a | b` | Alternatives.
`( ... )` | `( ... )` | Grouping.
`[ x ]`, `!optional(x)` | `[ x ]` | Optional group.
`x*`, `x+`, `!repeat(x, min, max)` | `x*`, `x+`, bounded closure | Repetition.
`!tag(LABEL, expr)` | `{ LABEL expr }` | [NLU slot markup](https://doc.sensory.com/tnl/7.9/reference/grammar.md#nlu-markup).
`<...>` | `` | [Out-of-grammar words](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-unknown).
### Unsupported constructs
The following BNF+EM constructs are not yet supported. A grammar that uses one
is rejected with an error that identifies the construct and its position, rather
than compiling to something unexpected:
- `!import` and cross-grammar references (``).
- `!start` with more than one entry-point rule.
- `!slot` runtime-attachable rules.
- `!id(...)` user identifiers.
- `!pronounce`, both the inline modifier and the grammar-wide statement.
- Quoted-string terminals (`"..."`).
- Other inline `!` modifiers, and `!language`.
Support for these constructs may be added in a future release.
## SRGS 1.0 (ABNF Form) grammars
The SDK also accepts grammars written in the [W3C SRGS 1.0][] "ABNF Form", the
speech recognition grammar format standardized for [VoiceXML][] and other voice
platforms. Support for this format eases migration of existing SRGS grammars.
### Format detection
A grammar is parsed as SRGS when its first non-comment content is an `#ABNF`
header, for example `#ABNF 1.0;`. Otherwise the [native syntax](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-syntax)
(or [BNF+EM](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-bnfplus), for a `#BNF+EM` header) is used. A single grammar
is entirely one format; you cannot mix them.
```text
#ABNF 1.0;
language en-US;
root $command;
public $command = turn ( on | off ) [ the ] ( $NULL | living room ) lights;
```
### Supported subset
The full SRGS ABNF surface syntax is parsed. The subset most commonly used in
command-and-control grammars compiles to a recognizer; each construct maps onto
an equivalent in the native syntax, so its recognition behavior matches the
corresponding native form:
SRGS | Native equivalent | Notes
-----|-------------------|------
`$rule` reference | `$rule` | Rule substitution.
`root $rule;` | start rule | Declares the entry-point rule (else the last rule).
`public`/`private` scope | — | Parsed; rule visibility is recorded.
`$NULL` | `:` | Matches with no speech (epsilon).
`$VOID` | `` | Never matches (an unmatchable alternative).
`$GARBAGE` | `` | [Out-of-grammar words](https://doc.sensory.com/tnl/7.9/reference/grammar.md#grammar-unknown).
`a | b` | `a | b` | Alternatives.
`a b` (juxtaposition) | `a b` | Sequence.
`( ... )` | `( ... )` | Grouping. `( )` is the empty string.
`[ x ]` | `[ x ]` | Optional group.
`x`, `x`, `x` | bounded closure, `x*` | Repetition (`` is `x` then `x*`). An optional repeat probability `x` biases the repetition (see the note below).
`"multi word"` | multi-word terminal | A quoted string is one lexical unit.
`( ... ){ name }` | `{ name ... }` | [NLU slot](https://doc.sensory.com/tnl/7.9/reference/grammar.md#nlu-markup): a tag whose body is a bare slot name captures the tagged element as that slot (intents and entities round-trip).
`{ tag }`, `{!{ tag }!}` | `{ tag }` | [Semantic tag markup](https://doc.sensory.com/tnl/7.9/reference/grammar.md#nlu-markup). A non-slot-name (script-like) tag body is preserved as an opaque tag.
`/n.n/` alternative weight | `( ... )/w` | Likelihood on a branch; converted to a log-prob cost on the native group weight.
A repeat operator may carry an optional repetition probability,
written `x`, where `p` is the probability of taking another
repetition. It does not change which utterances match (still between `m` and
`n` copies); it biases the recognizer by charging a log-prob cost of
$-log(p)$ on each "repeat again" step and $-log(1-p)$ on each "stop" step,
the same way a branch weight biases an alternative.
The self-identifying header (`#ABNF 1.0 [encoding];`) and the header
declarations (`language`, `mode`, `base`, `tag-format`, `lexicon`, `meta`,
`http-equiv`) are accepted and preserved for round-tripping, but do not change
recognition.
### Unsupported constructs
The following SRGS constructs are parsed but cannot yet be compiled to a
recognizer. A grammar that relies on one is rejected with an error that
identifies the construct and its position, rather than compiling to something
unexpected:
- Recursive rule references (a rule that refers to itself, directly or through
a cycle).
- External rule references (`$` and `$~`).
The `*`, `+`, and `?` operators are reserved by SRGS and are not valid
repetition operators; use the postfix `` form instead. Semantic
interpretation (executing tag scripts to build a result object) and the SRGS
XML Form (`grxml`) are out of scope. Support for these may be added in a future
release.
[ABNF]: https://en.wikipedia.org/wiki/Augmented_Backus%E2%80%93Naur_form "Augmented Backus-Naur Form"
[BCP 47]: https://www.rfc-editor.org/info/bcp47 "BCP 47: Tags for Identifying Languages"
[BNF+EM v2.1]: https://docs.vdk.vivoka.com/vsdk/v4/vsdk_tutorials_vsdk-csdk_grammar_formalism.pdf "BNF+EM v2.1 grammar formalism"
[regular expression]: https://en.wikipedia.org/wiki/Regular_expression
[RFC 5234]: https://www.rfc-editor.org/rfc/rfc5234 "RFC 5234: Augmented BNF for Syntax Specifications: ABNF"
[UTF-8]: https://en.wikipedia.org/wiki/UTF-8
[VoiceXML]: https://www.w3.org/TR/voicexml20/ "Voice Extensible Markup Language (VoiceXML) 2.0"
[W3C SRGS 1.0]: https://www.w3.org/TR/speech-grammar/ "Speech Recognition Grammar Specification 1.0"
*[API]: Application Programming Interface
*[LVCSR]: Large Vocabulary Continuous Speech Recognition model, feed-forward neural net acoustic model with FST decoder
*[NLU]: Natural Language Understanding model
*[RAM]: Random Access Memory
*[SDK]: Software Development Kit
*[STT]: Speech To Text: transformers with language model and CTC decoding
*[TNL]: TrulyNatural, Sensory's large-vocabulary speech recognition technology