Dictionary format

A dictionary tells bite how to decode the messages of one or more interfaces. It is a directory of JSON files; nothing about a message set is compiled into the decoder. bite check <dir> validates a dictionary and runs its test vectors. Writing one from an interface control document is covered by icd-to-dictionary.

The 23 bytes of a weather-station Observation message, 0200174b583700fb2e18bcda9b4236000004030201beef, with the bit range of each of its nine fields marked. A table gives the dictionary entry for each field (start bit, stop bit, type, options) and the value it decodes to: message_type Observation, message_length 23, station_id KX7, temperature -12.34, pressure 101325, wind_direction 271.5, humidity 45.5, sample_serial 16909060, checksum beef.

Four dictionaries ship in dictionaries/:

DictionaryWhat it is for
ntpreal traffic you already have: the Network Time Protocol header of RFC 5905 on UDP port 123, to capture on your own network. Messages identified by three bits of the first byte, exact fixed-point scaling, an identifier named and not defined
weather-stationevery field type: UDP messages identified by a payload field, with signed, scaled, float, boolean, text, little-endian and non-byte-aligned fields. The one to start a dictionary from
vehicle-canthe frame identifier as discriminator: CAN frames with little-endian signals in lsb0 numbering
glasshousetranscribed from a document: two interfaces from a synthetic interface document, with a source for every field, a declared length for every message and a vector for each. The worked example

Layout

<dictionary>/
  dictionary.json                 optional
  <INTERFACE>/
    metadata.json                 required; makes the directory an interface
    messages/
      <Message Name>.json         one file per message; the file stem is the name
  vectors/
    <any-name>.json               optional test vectors

An interface is a set of messages that share a transport and one way of telling the messages apart: one ICD, one CAN bus, one serial link.

dictionary.json

KeyTypeMeaning
namestringdictionary name
versionstringdictionary version
descriptionstringfree text
sourceobjectthe document the dictionary was transcribed from, see Keys for authoring
lintobjectunits added to the vocabulary and findings allowed with a reason, see Keys for authoring

metadata.json

KeyTypeDefaultMeaning
message_typeobjectrequiredhow a message is identified, see below
portintegernoneUDP or TCP port. A packet whose source or destination port equals it belongs to this interface; the source port is tried first. Two interfaces may not claim one port.
bit_order"msb0" or "lsb0""msb0"how bit positions are numbered, see Bit numbering
byte_order"big" or "little""big"byte order of byte-aligned fields under msb0
message_length{start, stop}nonewhere a message states its own length
description (or mission)stringnonefree text
icd{version, date}nonethe document the interface was transcribed from
protocol{network, transport}noneinformational
sourceobjectnonewhere the interface is defined in the document
notestringnonea remark for the next maintainer
icd_numberingobjectnonehow the document numbers words and bits, see Keys for authoring

message_type

KeyTypeDefaultMeaning
source"payload" or "frame_id""payload"payload: the identifier is bits [start, stop) of the payload. frame_id: it is the identifier that arrives with the frame, such as a CAN identifier.
start, stopinteger0bit range, for payload; 1 to 64 bits
maskstringnonefor frame_id: ANDed with the identifier before lookup
valuesobject{}identifier to message name

An identifier with no entry in values decodes to a record whose message type is the identifier in decimal and which has no fields. A name in values with no file in messages/ is identified but not decoded; bite check warns about it.

messages/<Message Name>.json

KeyTypeMeaning
descriptionstringfree text
fieldsobjectfield name to field definition
declared_bitsintegerthe length the document gives the message, in bits; bite coverage measures gaps against it
sourceobjectwhere the message is defined in the document
notestringa remark for the next maintainer

Field

KeyTypeDefaultMeaning
start, stopintegerrequiredbit range [start, stop)
typestring"uint"uint, int, float, bool, bytes, ascii
signedbooleanfalsetrue is shorthand for "type": "int"
byte_order"big" or "little"interface'soverrides the interface byte order
scale, offsetnumber1, 0value is raw * scale + offset, as a float. For an integer field the result is rounded to the decimal places scale and offset are written with, so raw 35 with scale 0.01 is 0.35 and not 0.35000000000000003. A scale or offset that needs more than 15 decimal places (a power of two such as 2^-32, or 1/3) and a result of more than 15 significant digits are not rounded
valuesobjectnoneraw integer to label
unitstringnoneinformational
descriptionstringnoneinformational
sourceobjectthe message'swhere the field is defined in the document
notestringnonea remark for the next maintainer: an erratum, an encoding the format cannot express
TypeWidthDecodes to
uint1 to 64 bits, any alignmentunsigned integer
int1 to 64 bits, any alignmenttwo's-complement integer
float32 or 64 bitsIEEE 754 float
bool1 to 64 bitsfalse when all bits are zero
byteswhole bytes, byte-aligned, any lengthhex string
asciiwhole bytes, byte-aligned, any lengthtext with trailing NUL bytes and spaces removed

A field with no type that is byte-aligned and wider than 64 bits is bytes.

Keys in values (fields and message_type) are integers written as strings, in decimal or 0x hexadecimal; field value keys may be negative. Labels are looked up on the raw integer, before scale and offset.

A field that reaches past the end of the payload decodes to null. The other fields of the message still decode.

Bit numbering

msb0 is the numbering protocol documents use: bit 0 is the most significant bit of byte 0, and a field is read as a big-endian bit string. A field of bits [12, 20) is the low four bits of byte 1 followed by the high four bits of byte 2. With "byte_order": "little" a byte-aligned field has its bytes reversed; a little-endian field must be byte-aligned.

lsb0 is the numbering of little-endian ("Intel") signals on CAN: bit 0 is the least significant bit of byte 0, and the payload is read as one little-endian integer. A field of bits [12, 20) is the high four bits of byte 1 (its low half) and the low four bits of byte 2 (its high half).

Test vectors

Each file in vectors/ is a payload and what it must decode to:

{
    "interface": "WX",
    "payload_hex": "01000c4b58370000015180600000",
    "expect": {
        "message_type": "Heartbeat",
        "fields": {
            "station_id": "KX7",
            "uptime": "86400",
            "mode": "Calibrating"
        }
    }
}
KeyMeaning
interfaceinterface name
payload_hexpayload bytes; whitespace ignored
frame_idframe identifier, for frame_id interfaces
expect.message_typeexpected message name
expect.fieldsfield name to expected value as text: the label where the field has one, the value otherwise, null for a field beyond the payload. Fields not listed are not checked.
source, notewhere the expected values come from

An expected value is compared as text, except for a field that decodes to a float (type float, or any field with scale or offset): there the two are compared as numbers to nine significant digits, so a vector can say 21.0 where the decoder writes 21, and 0.666666667 for a field whose scale is not a decimal, where the decoder computes 0.6666666666666666.

Keys for authoring

The decoder ignores these keys; a dictionary without them decodes the same. They are read by bite coverage, bite lint, bite diff, bite range and the CSV commands, and the conventions say when each is required.

source

Where a definition was transcribed from. Allowed in dictionary.json, metadata.json, a message, a field and a vector. A member that is absent is taken from the level above: a field from its message, a message from its interface, an interface from the dictionary.

KeyMeaning
documentthe document's identifier
revisionits revision
sectionsection number
tabletable number
pagepage number

Each member is text; a number is accepted and written as text by bite fmt. A source is complete when it resolves a document, a revision and at least one of section, table and page.

icd_numbering

In metadata.json: how the document addresses bits, so that bite range, bite import-csv and bite export-csv can convert between its "word 3, bits 12-5" and [start, stop).

KeyTypeDefaultMeaning
word_bitsintegerrequiredword size in bits, 1 to 64
bit0"msb" or "lsb"requiredwhich end of a word the document numbers first
first_word0 or 1requiredthe number of the first word
first_bit0 or 10the number of the first bit of a word
word_order"big" or "little""big"big: a word is sent most significant byte first, and ranges are under "bit_order": "msb0". little: least significant byte first, and ranges are under "bit_order": "lsb0".

lint

In dictionary.json.

KeyTypeMeaning
unitslist of stringsunits accepted in addition to the unit vocabulary
allowlist of objectsfindings that are accepted, each with a reason

An entry of allow:

KeyMeaning
rulethe rule's identifier, such as C004
atthe location the entry covers, with everything under it: ZONE, ZONE/Alarm or ZONE/Alarm/severity. Everywhere when absent.
reasonwhy the finding is accepted; required

bite lint and bite coverage leave an allowed finding out of their failures and report an entry that matches nothing. bite check --strict honours an entry for the three warnings that have a rule: overlapping fields (C002), a message named and not defined (C004), a message defined and not named (C005).

Limits

A dictionary may come from someone else, and in the browser decoder from any file a visitor is handed. The loader bounds what one can cost in memory and time. Each limit is a constant in bite/src/limits.rs.

LimitValueConstantOver the limit
size of one dictionary file4 MiB (4,194,304 bytes)MAX_FILE_BYTESerror; the file is not read
files in one dictionary: dictionary.json, every metadata.json, message and vector together10,000MAX_FILESthe dictionary does not load
fields in one message4,096MAX_FIELDS_PER_MESSAGEerror; the message is left out
entries in one values object (labels of a field, names of message_type)65,536MAX_VALUESerror; the field is left out, or the interface identifies no message
payload length considered65,535 bytes (bit 524,280)MAX_PAYLOAD_BYTESerror; a field or message_type range whose stop is past it is left out

65,535 bytes is the largest UDP datagram or TCP segment, so no field of a captured packet lies beyond it. Because no loaded field reaches further, decoding reads at most that many bytes of a payload however long it is.

JSON nested more than 128 levels deep is refused by the parser as not valid JSON. A key or label quoted in a finding is cut to 80 characters.

What bite check reports

SeverityFindingEffect on loading
errorfile is not valid JSON or lacks a required keyfile left out
errorfield range empty or inverted; integer wider than 64 bits; float not 32 or 64 bits; bytes/ascii not byte-aligned; little-endian field not byte-aligned under msb0field left out
errormessage_type range not 1 to 64 bits; a values key or mask that is not an integer (decimal or 0x hexadecimal, at most one sign)value left out
errortwo interfaces claim one portthe first, by name, keeps it
errora file, message, values object or bit range over a limitas the limits table says
warningfields overlapboth kept
warningmessage file that no message_type value nameskept, never selected
warningmessage_type value with no message fileidentified, not decoded
warninglabel that is not text, or a label on a float, bytes or ascii fieldlabel ignored

bite check exits non-zero on any error or failed vector; --strict also fails on warnings that lint.allow does not cover.

The loader reads each file with the last of two equal keys winning, so a field name written twice in one message loses a field without a warning. bite fmt refuses such a file. bite coverage reports the bits no field covers, and bite lint the conventions; see bite/README.md for every command.

The source of this page is a Markdown file in the repository.