# DXN Technical Reference Implementation-ready grammar and binary encoding for DXN (Data eXchange Notation), `.dxn` (text) and `.dxnb` (binary). This document is normative and terse by design. Nothing beyond it should be required to implement a conforming parser, encoder, or decoder — see the [DXN tutorial](TUTORIAL.md), [examples](DXN_EXAMPLES.md), and [cheatsheet](DXN_CHEATSHEET.md) for a gentler introduction, and `dextrin`'s own module docs for one particular implementation's choices. Grammar is ISO/IEC 14977 EBNF. Reference tables are [TOON](https://toon.sh) (`name[N]{fields}:` header, one comma-separated row per record) for token-efficient machine parsing; a cell containing a comma or quote is wrapped in `"..."` with internal `"` doubled, same as CSV. --- ## 1. `.dxn` — text format ### 1.1 Lexical grammar ```ebnf document = [ header ] , value ; header = "@dxn" , ws , string ; ws = { " " | "\t" | "\n" | "\r" | "," } ; (* fully insignificant, everywhere *) comment = "#" , { any char except "\n" } ; digit = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" ; hex_digit = digit | "a" | "b" | "c" | "d" | "e" | "f" | "A" | "B" | "C" | "D" | "E" | "F" ; letter = ? any character with Unicode property XID_Start (UAX #31) ? ; ident_char = ? any character with Unicode property XID_Continue (UAX #31) ? | "-" | "?" | "!" ; identifier = ( letter | "_" ) , { ident_char } , [ "/" , ( letter | "_" ) , { ident_char } ] ; reserved = "nil" | "true" | "false" | "NaN" | "Infinity" ; string_char = ? any character except '"' or "\" ? | escape ; escape = "\" , ( '"' | "\" | "n" | "t" | "r" | "0" | "a" | "b" | "f" | "v" | ( "x{" , hex_digit , { hex_digit } , "}" ) ) ; string = '"' , { string_char } , '"' ; char_body = ? any single non-whitespace character ? | escape | "s" (* space, char-literal only *) ; char = "?" , char_body ; ``` Reserved sigils — none may begin a bare `identifier`: `#` comment, `@` open tag, `%` map/struct, `~` closed sigil family, `?` char, `:` keyword. ### 1.2 Value grammar ```ebnf value = { skip } , value_body ; skip = comment | discard ; discard = "@_" , value ; (* parses and discards one value; produces nothing *) value_body = nil | boolean | number | string | char | symbol | keyword | list | tuple | map_lit | struct_lit | set_lit | sigil | tag_form ; nil = "nil" ; boolean = "true" | "false" ; number = integer | float | decimal | rational ; integer = [ "-" ] , digit , { digit } ; float = [ "-" ] , digit , { digit } , ( ( "." , digit , { digit } , [ exponent ] ) | exponent ) | "NaN" | "Infinity" | "-Infinity" ; exponent = ( "e" | "E" ) , [ "+" | "-" ] , digit , { digit } ; decimal = [ "-" ] , digit , { digit } , [ "." , digit , { digit } ] , "M" ; rational = [ "-" ] , digit , { digit } , "/" , digit , { digit } ; symbol = identifier - reserved ; (* bare, unevaluated reference *) keyword = ":" , ( identifier | string ) ; (* value position; see map_entry for key position *) list = "[" , { value } , "]" ; tuple = "{" , { value } , "}" ; set_lit = "@{" , { value } , "}" ; map_lit = "%{" , { map_entry } , "}" ; map_entry = ( identifier , ":" , value ) (* shorthand — plain-atom keys only *) | ( value , "=>" , value ) ; (* arrow — any key type *) struct_lit = "%" , identifier , ( ( "{" , { map_entry } , "}" ) (* keyed *) | ( "[" , { value } , "]" ) ) ; (* positional *) sigil = date_sigil | time_sigil | instant_sigil | regex_sigil ; date_sigil = "~D[" , iso_date , "]" ; time_sigil = "~T[" , iso_time , "]" ; instant_sigil= "~U[" , iso_date , " " , iso_time , "Z" , "]" ; regex_sigil = "~r/" , { regex_char } , "/" , [ regex_flags ] ; regex_flags = { "i" | "m" | "s" | "u" | "x" | "f" | "r" } ; tag_form = "@" , identifier , value ; (* built-in tag name -> §1.3; else custom *) ``` Tag resolution: an `identifier` immediately following `@` is matched against the built-in tag table (§1.3) first; any name not found there is a **custom tag** — parses identically (`@tag value`), decodes to an application-defined type via a registered decoder, per §2 goal 4 of the design log. ### 1.3 Type reference ```toon types[30]{type,category,dxn_syntax,notes}: nil,scalar,nil,"" boolean,scalar,"true | false","" integer,scalar,"-?[0-9]+","arbitrary precision" float,scalar,"-?[0-9]+(.[0-9]+)?([eE][+-]?[0-9]+)? | NaN | Infinity | -Infinity","IEEE 754 double" decimal,scalar,"[0-9]+(.[0-9]+)?M","exact fixed-point" rational,scalar,"int/uint","exact ratio" string,scalar,"""...""","UTF-8" char,scalar,"?c","Unicode codepoint" symbol,scalar,"bare identifier","unevaluated reference, e.g. type name" keyword,scalar,":name | :""..."" | name: (key pos.)","Elixir atom" list,collection,"[ ... ]","" tuple,collection,"{ ... }","" map,collection,"%{ ... }","no ordering guarantee" ordered-map,collection,"@ordered %{ ... }","order is part of value identity" set,collection,"@{ ... }","" sorted-set,collection,"@sorted-set @{ ... }","" struct-keyed,collection,"%Name{ ... }","requires schema to fully interpret" struct-positional,collection,"%Name[ ... ]","field order per schema; both struct forms parse equal" array,collection,"@array[ ... ]","fixed-size/indexed" date,temporal,"~D[YYYY-MM-DD]","ISO 8601" time,temporal,"~T[HH:MM:SS(.ffffff)?]","ISO 8601" timestamp,temporal,"~U[YYYY-MM-DD HH:MM:SSZ]","ISO 8601, UTC only" datetime,temporal,"@datetime ""...""","ISO 8601, offset required, non-UTC" duration,temporal,"@duration ""P...""","ISO 8601 duration" uuid,extended,"@uuid ""...""","RFC 4122" uri,extended,"@uri ""...""","RFC 3986" bytes,extended,"@bytes ""...""","base64" regex,extended,"~r/pattern/flags","PCRE-style via Erlang :re" custom-tag,extended,"@tag value","open extension point, any unrecognized tag name" header,meta,"@dxn ""1.0""","optional, first form only" ``` ### 1.4 Semantic notes not expressible in grammar alone - **Map key equality/ordering.** `%{ }` makes no ordering guarantee even though entries are physically sequential in source text; `@ordered %{ }` makes ordering part of the value's identity. This is a semantic distinction, not a syntactic one. - **Struct interpretation requires a schema.** Both struct forms parse without one; a reader lacking the schema for a given type name returns an opaque tagged value rather than failing. - **`@_` (discard)** is resolved during parsing — it never appears in a parsed value tree, and has no `.dxnb` encoding (§2.1). - **`NaN` has no sign variant.** `-NaN` is not valid; IEEE 754's NaN sign bit has no meaningful, observable use here. - **Duration is not reducible to a single elapsed-time scalar** — `years` and `months` are calendar-relative (variable length), so they, along with weeks/days/hours/minutes/seconds, are tracked as independent fields (see §2.3 for the exact field set, shared with `.dxnb`). --- ## 2. `.dxnb` — binary format Built on CBOR (RFC 8949). Every `.dxnb` value is a self-describing CBOR item; DXN types not covered by a standard CBOR major type use a CBOR tag (standard where one is registered, DXN-private otherwise — §2.3). ### 2.1 Envelope ```ebnf dxnb_file = magic , version , cbor_item ; magic = %x44 %x58 ; (* "DX" *) version = %x01 ; (* format major version, currently 1 *) ``` Mandatory (unlike the optional text header) — fixed 3-byte cost regardless of document size, and binary tooling depends on magic-number sniffing far more than text tooling does. No end-of-document marker needed; a CBOR item is self-delimiting. Structs are **always positional** in `.dxnb` (§1.3's keyed form exists only for human readability, which doesn't apply to raw bytes) — one wire shape per struct value, `[type-name-or-ref, field-1, field-2, ...]`. `@_` (discard) and `#` comments have no binary encoding — both are resolved away before a value exists to encode. ### 2.2 Type mapping ```toon cbor_map[29]{type,cbor_major_or_tag,payload,integrity_note}: nil,"simple(22)","","" boolean,"simple(20/21)","","false/true" integer,"major0/1; tag2/3","two's complement bignum bytes past 64-bit","arbitrary precision preserved exactly" float,"major7","IEEE 754 double","NaN/Infinity encode natively, no tag needed" decimal,"tag4","[exponent,mantissa]","" rational,"tag30","[numerator,denominator]","" string,"major3 (untagged)","UTF-8 bytes","default/untagged case" char,"tag T_CHAR","major0 codepoint","" symbol,"tag T_SYMBOL","major3 text","" keyword,"tag T_KEYWORD","major3 text","" list,"major4 (untagged)","items","default/untagged case" tuple,"tag T_TUPLE","major4 items","" map,"major5 (untagged)","pairs","default/untagged case" ordered-map,"tag T_ORDERED","major5 pairs","order preserved by tag presence, not byte sequence" set,"tag T_SET","major4 items","" sorted-set,"tag T_SORTED_SET","major4 items, sorted order","" struct,"tag T_STRUCT","[type-name-or-ref, field...]","positional only, §2.1" array,"tag T_ARRAY","major4 items","" date,"tag T_DATE","integer days since 1970-01-01","Unix epoch, day granularity" time,"tag T_TIME","integer microseconds since midnight","" timestamp,"tag1, integer form only","integer microseconds since epoch","MUST use integer, not float tag-1 form — float loses microsecond precision at scale" datetime,"tag T_DATETIME_OFFSET","[epoch-microseconds int, offset-minutes]","same integer-only requirement as timestamp" duration,"tag T_DURATION","bitmask byte + present fields, §2.3","" uuid,"tag37","16 raw bytes","not the 36-char text form" uri,"tag32","major3 text","" bytes,"major2","raw bytes","no base64 in binary" regex,"tag T_REGEX","[pattern text, flags byte]","flags bit layout in §2.3" custom-tag,"tag T_CUSTOM","[tag-name-or-ref, value]","binary form of the open extension point" header,"envelope, §2.1","magic+version bytes","not a CBOR item; metadata, not data" ``` ### 2.3 Private tag assignments and bit layouts ```toon private_tags[15]{name,number,wraps}: T_CHAR,200,"integer codepoint" T_SYMBOL,201,"text string" T_KEYWORD,202,"text string" T_TUPLE,203,"array" T_ARRAY,204,"array" T_ORDERED,205,"map" T_SET,206,"array" T_SORTED_SET,207,"array" T_STRUCT,208,"array: [type-ref, field...]" T_DATE,209,"integer" T_TIME,210,"integer" T_DATETIME_OFFSET,211,"array: [epoch-us, offset-min]" T_DURATION,212,"bitmask byte + fields" T_REGEX,213,"array: [pattern, flags-byte]" T_CUSTOM,214,"array: [tag-ref, value]" ``` Allocated from CBOR's one-extra-byte tag range (24–255), placed at 200–214 specifically to stay clear of the actively-managed low range (0–40 has multiple assignments and gaps IANA could fill later). Not IANA-registered — collision-free only within `dextrin`-produced documents; register formally before any non-`dextrin` interop is expected. **Duration bitmask** (`T_DURATION` payload) — one byte, low bit first; each set bit is followed, in bit order, by that field's value as a signed integer: ```toon duration_bits[7]{bit,field,unit}: 0,years,count 1,months,count 2,weeks,count 3,days,count 4,hours,count 5,minutes,count 6,microseconds,"count (fractional seconds folded in, matching time/timestamp precision)" ``` Bit 7 is reserved (must be 0). **Regex flags byte** (`T_REGEX` payload, second element) — one byte, low bit first, each bit a boolean modifier matching Erlang `:re`/Elixir `Regex` semantics: ```toon regex_bits[7]{bit,flag,meaning}: 0,i,caseless 1,m,multiline 2,s,dotall 3,u,unicode 4,x,extended 5,f,firstline 6,r,ungreedy ``` Bit 7 is reserved (must be 0). ### 2.4 Value sharing (repeated symbols, keywords, struct type names) Use CBOR's registered string-reference extension: tag 256 marks the enclosing scope as using shared references, tag 25 references the *n*th previously-seen string within it. Apply to repeated `T_SYMBOL`/ `T_KEYWORD` text and struct type names; leave one-off values inline. Not required for a conforming decoder to produce, but a conforming decoder MUST accept it. ### 2.5 Value sharing (arbitrary repeated values) Same motivation as §2.4, generalized from text to *any* `.dxnb` item — a repeated `map`/`list`/`struct`/etc. subtree, not just repeated symbol/keyword text. Uses CBOR tags 28 (mark the following item as shareable) and 29 (reference the *n*th previously tag-28-marked item, 0-indexed across the whole document). An encoder MAY wrap a repeated subtree's first occurrence in tag 28 and encode every later occurrence as a bare tag-29 index instead of repeating its bytes. This is a size optimization only, never an identity/aliasing feature: - A decoder MUST resolve every tag-29 reference to an independent copy of the tag-28-marked value it points to. The decoded result MUST be indistinguishable from a document that wrote every occurrence out in full — no decoder-visible aliasing, no shared mutable identity. - A tag-29 index that refers to a not-yet-seen tag-28 item, or that would create a cycle (directly or through nested references), is malformed; a decoder MUST reject it rather than attempt to represent a graph. `.dxnb` values are trees, always — §2.5 shortens the encoding of a tree with repeated subtrees, it does not turn the format into one that can express sharing as observable structure. Not required for a conforming encoder to produce; a conforming decoder MUST accept it. Tags 28/29 do not collide with the private range in §2.3 (200–214) or with §2.4's tags (256, 25) — confirm both against the current IANA CBOR tags registry before final implementation, as with any tag number cited normatively here. --- ## 3. Worked example ``` @dxn "1.0" %{ id: @uuid "550e8400-e29b-41d4-a716-446655440000" name: "Ada Lovelace" active: true score: 19.99M tags: @{:admin :staff} meta: @ordered %{created: ~U[1990-01-01 00:00:00Z]} address: %Point[51.05, 13.74] result: {:ok, 200} handle: ~r/^[a-z0-9_]{3,20}$/i } ``` Every construct above is fully specified by §1.2 + §1.3; every field round-trips through §2 without loss (§2.2's `integrity_note` column records the two places — bignum and timestamp precision — where that took a specific encoding choice to guarantee). --- ## 4. Schema documents (`.dxns`) A `.dxns` file is valid `.dxn` — §1's grammar, unchanged, no schema -specific syntax. What makes a document a *schema* document is purely the shape of the value it decodes to: a `map` whose values are either `%schema{...}` (a struct schema) or a `type_expr` (a **named type**, §4.5). `%schema{}`/`%field{}` (§4.4) are recognized by name directly, the same way a built-in tag name is (§1.2) — not looked up against a schema of their own, which would be circular. ### 4.1 `type_expr` forms ```toon type_expr[13]{form,dxn_shape,meaning}: any,":any","matches any value at all" primitive,":integer / :string / ...","one keyword per §1.3 type name" reference,"bare symbol, e.g. Address","names a struct schema or a named type in this document (or Ns/Name, §4.6)" list-of,"{:list-of elem}","list whose every element matches elem" set-of,"{:set-of elem}","set whose every element matches elem" tuple-of,"{:tuple-of a b c}","fixed-arity positional tuple, one type_expr per slot" map-of,"{:map-of key-type val-type}","every key matches key-type, every value matches val-type" enum,"{:enum lit lit ...}","value must equal one of the given literals (any type)" one-of,"{:one-of a b}","union — value must match at least one variant" all-of,"{:all-of a b}","intersection — value must match every variant" nilable,"{:nilable t}","sugar for {:one-of :nil t}" refine,"{:refine t %{constraints}}","base type plus constraints, §4.2" struct,"%schema{fields: @ordered %{...}}","struct/record shape, §4.4" ``` A bare `type_expr` (no wrapping tuple/struct) is always a `primitive` or a `reference` — the tuple form is only needed once a constructor takes arguments. ### 4.2 `refine` constraints ```toon refine_constraints[10]{key,applies_to,meaning}: min,"integer/float/decimal/rational","value >= given number" max,"integer/float/decimal/rational","value <= given number" exclusive-min,"integer/float/decimal/rational","value > given number" exclusive-max,"integer/float/decimal/rational","value < given number" multiple-of,"integer/float/decimal/rational","value is an exact multiple of given number" min-length,string,"codepoint count >= given integer" max-length,string,"codepoint count <= given integer" pattern,string,"value matches given regex (§1.3's regex type)" min-count,"list/set/tuple","element count >= given integer" max-count,"list/set/tuple","element count <= given integer" ``` `{:refine t %{k: v, ...}}`'s constraint map may combine any number of these keys; a value must satisfy `t` itself *and* every listed constraint. ### 4.3 Required vs. optional fields No separate flag: a field key ending in `?` is optional; any other key is required. `?` is already a legal `ident_char` (§1.1) — no grammar addition. The `?` itself is not part of the field's name. ### 4.4 Struct schemas (`%schema{}`/`%field{}`) ```ebnf schema_entry = identifier , ":" , schema_form ; schema_form = "%schema{" , [ "closed:" , boolean ] , [ "forbidden:" , "[" , { identifier } , "]" ] , [ "refine-fn:" , identifier ] , "fields:" , "@ordered" , "%{" , { field_entry } , "}" , "}" ; field_entry = ( identifier , [ "?" ] , ":" , type_expr ) | ( identifier , [ "?" ] , ":" , field_form ) ; field_form = "%field{" , "type:" , type_expr , [ "default:" , value ] , [ "description:" , string ] , "}" ; ``` ```toon schema_keys[3]{key,default,meaning}: closed,false,"if true, no field outside fields: may be present at all" forbidden,[],"explicit deny-list of field names, checked even on an open schema" refine-fn,none,"names a cross-field/whole-value predicate, resolved outside this document (implementation-defined mechanism)" ``` `fields:` MUST be `@ordered %{...}`, not a plain `map` — field order is the canonical position ↔ name mapping a struct's positional wire form (§2.1) needs; a plain `map`'s "no ordering guarantee" (§1.4) would leave that mapping undefined. A keyed `%Name{...}` text literal may still list its own fields in any order — names alone resolve those — this requirement is on the *schema's* `fields:` only. `closed` and `forbidden` are independent: `closed` alone rejects any field not listed in `fields:`, reported generically ("unknown field"); `forbidden` rejects specific named fields even when the schema is otherwise open, reported by name specifically — the distinction matters because a deprecated/renamed field deserves a more specific error than "never heard of this field." `%field{type:, default:, description:}` is reached only when a field needs metadata beyond a bare `type_expr` — a default value (used when the field is absent; independent of `?`, which controls whether *absence itself* is an error) or a human-readable description. It never carries its own optionality flag — that's always the key's `?` suffix. ### 4.5 Named types Any top-level entry whose value is not a `%schema{}` defines a **named type**: a reusable name for a `type_expr`, e.g. `PositiveInt: {:refine :integer %{min: 1}}`. It's referenced exactly like a struct name — a bare symbol (§4.1's `reference` form) — since both are resolved the same way: by name, against whatever this document (or an already-known document, §4.6) defines. A named type's own body MAY reference another named type. Whether it may reference one declared *alongside* it in the same document (a same-document forward or sibling reference) is implementation-defined: a `.dxns` document's top-level `map` carries no ordering guarantee (§1.4), so there is no normative order to resolve such a reference against. A conforming reader MAY refuse same-document sibling references outright, MAY resolve them via some deterministic pass (e.g. two-phase compilation), or MAY support arbitrary forward reference — implementations are free to differ here, and a schema author SHOULD NOT rely on behavior beyond "a named type can reference one already known before this document was processed." ### 4.6 Cross-file references `identifier`'s own grammar already supports one `/` (§1.1), `Namespace/Name` — the mechanism this document uses for referencing a struct schema or named type defined in a separate `.dxns` file/unit. Resolving `Namespace/Name` to an actual document (a file path, a registry lookup, a network fetch, ...) is entirely reader-defined; this specification fixes only the reference *syntax*, not a resolution convention. ### 4.7 Enforcement A struct value decoded without a schema for its name is an opaque tagged value (§1.4) — never an error. Once a schema is known for a given name, a conforming reader MUST enforce, for every value of that struct: - every required field (§4.3) is present; - every present field's value matches its declared `type_expr` (§4.1/§4.2); - if `closed: true`, no field outside `fields:` is present at all; - no field named in `forbidden:` is present, regardless of `closed`. A violation of any of these MUST be reported as an error, distinctly from a value that simply fails to parse at all — but *how* it's reported (exception, error tuple, a validation-result value, ...) is implementation-defined. Whether `refine-fn:` predicates run at all, and how a predicate name is resolved to actual executable logic, is also implementation-defined — `.dxns` documents themselves are normatively cross-implementation because a schema *shape* is plain data; a `refine-fn:` name is deliberately an escape hatch to logic that isn't.