Architecture
first-bite turns frames on a wire into decoded records on a bus. It reads frames from a network interface or a capture file, decodes the payload of each against a dictionary, and publishes one record per packet (JSON by default, CBOR selectable) to ZeroMQ, from where sinks forward records to object storage, REST endpoints, a Zenoh fabric or a file.
Crates
| Crate | Kind | Does |
|---|---|---|
bite | library and bite CLI | loads and validates a dictionary; decodes a payload into typed fields; exports JSON Schema. Knows nothing about networks. |
bite-wasm | WebAssembly module | the bite library for browsers: a dictionary from files in memory, decoding, JSON Schema and vectors, called from JavaScript |
mind | binary mind | reads frames from a source, finds the transport payload, asks bite to decode it, publishes the record |
flow | library | ZeroMQ publishers, and subscribers that forward records to sinks or through transforms |
cotify | library | builds Cursor-on-Target XML from a decoded record and a field mapping |
datamodels | library | the record type that crosses crate boundaries |
logger | library | log line format |
The dictionary is data
Nothing about a message set is compiled in. A dictionary is a directory of JSON files that names interfaces, the messages of each, and the bit range, type and value labels of every field; dictionary-format.md is the reference. mind loads the directory named by --dictionary or BITE_DICT_PATH once at start into an immutable, indexed structure.
A dictionary for a message set that cannot be published lives outside this repository and is used the same way. bite check <dir> validates it and runs the test vectors it carries, so a private dictionary is tested by the public decoder without its contents entering this tree.
From frame to record
- Source.
mind/src/source.rsdefinesFrameSource, with three implementations:LiveCapture(libpcap on an interface, promiscuous, with an optional BPF filter),FileCapture(a pcap or pcapng file, read once) andUdpFrames(one Ethernet frame per UDP datagram, for running without capture privileges). A frame is a timestamp, a link type and borrowed bytes. - Pipeline.
mind/src/pipeline.rsslices the frame withetherparse, keeps UDP and TCP packets, and looks up the interface by port (Dictionary::interface_for_ports, source port first). The interface reads the message type from the payload and decodes every field of that message. - Record. Every UDP or TCP packet yields one
PacketInfoWithParsedMessage(datamodels/src/packets.rs): timestamp, addresses, ports, the payload and the whole frame in base64, deployment metadata fromDATA_SOURCE_*environment variables, and, when the port belongs to an interface, the interface name (subsystem), the message name (message_type) and the decoded fields (message). A packet on a port no interface claims is still published, undecoded. - Publish.
flowsends the record on each configured ZeroMQ PUB socket under a topic derived from the record, by defaultinterface/message.
Field values in message keep their type: integers and scaled values are numbers, booleans are booleans, the dictionary's label stands where the raw value has one, and a field beyond the end of a short payload is null. bite schema <dir> prints a JSON Schema for it. record.md is the record's contract: every field, the earlier all-text version 1 (FIRST_BITE_RECORD_VERSION=1), and the two encodings, JSON and CBOR, a publisher or sink can write.
Sinks and transforms
A subscriber is a ZeroMQ SUB socket with a topic filter and a sink. Subscribers are listed in the YAML file named by SUBSCRIBER_CONFIG_PATH; with no file there are none, and records are available only on the PUB socket. Each sink is a cargo feature of flow, forwarded by mind.
| Sink | subscriber_type | Feature | Status |
|---|---|---|---|
| Azure Blob Storage | objectstorage / azureobject | azure (default) | built |
| S3-compatible storage | objectstorage / awsobject | aws (default) | built |
| HTTP POST | rest | always | built |
| local file | local | always | built |
UDP datagrams, unicast or multicast; Cursor-on-Target with a cot: mapping (cot-to-tak.md) | udp | always | built |
TCP stream, optionally inside TLS with a client certificate; Cursor-on-Target with a cot: mapping | tcp | always | built |
| Zenoh | zenoh | zenoh | built |
| Zenoh on a goat data fabric (optional, off by default; goat/joining.md) | goat | goat | built |
The ZeroMQ sockets are plaintext and stay on the host; the HTTP sinks and the Zenoh sink are the paths that cross a network. A security block in the configuration states the level every path is held to (plaintext on loopback only, TLS, or mutual TLS, with a minimum TLS version, a key-exchange policy and an optional requirement for the FIPS module), mind refuses to start when a path cannot meet it, and mind security show prints what each path will use. All TLS goes through one crypto provider installed at start. security/transport.md is the guide.
A subpub is a subscriber, a transform and a publisher: it reads records, turns each into something else, and publishes the result on another ZeroMQ socket. The transform a build always has is Cursor-on-Target (subtype: cot), driven by a mapping in the configuration that says which fields give the identifier, the type and the position. Subpubs run when mind is started with --activate-subpubs. A udp or tcp subscriber takes the same mapping and sends the event to a TAK server or to TAK clients, with no subpub between.
A build with the cargo feature goat-transformers has a second transform, subtype: xf. It hands the record's payload to the goat-transformers engine, which reads each format from a data file, a pack: tools/bite2pack writes the pack of a dictionary, and the target (Cursor-on-Target XML, GeoJSON) has one of its own. The feature is off by default and cot is unchanged by it; flow/README.md holds the measured comparison of the two.
Cursor-on-Target here is a per-message field mapping. A node that needs tracks normalised into Cursor-on-Target, ASTERIX CAT062 or SAPIENT with one identity scheme across sensors runs Open Billie for that; related-projects.md says how first-bite fits with it and with octo-pod.
Beyond IP
bite decodes a payload and, optionally, a frame identifier; it does not care where they came from. The vehicle-can dictionary identifies messages by CAN identifier and reads little-endian signals in lsb0 bit numbering, and its vectors pass. What exists for non-IP links today is the decoder and the bite decode --frame-id command.
| Piece | Status |
|---|---|
decode CAN-style frames: frame_id discriminator, lsb0 numbering | built |
a FrameSource for SocketCAN, and a record shape for frames with no IP addresses | designed, not built |
| importing a DBC file as a dictionary | not built |
| MIL-STD-1553 and ARINC 429 sources | not built |
In the browser
The library half of bite has no dependency on a filesystem or a network: Dictionary::from_files builds a dictionary from file paths and text held in memory, and gives the same dictionary and the same issues as Dictionary::load gives for a directory with those files. bite-wasm compiles it for wasm32-unknown-unknown and gives it a JavaScript interface. make wasm writes the module to site/pkg/, and site/decoder/ is a static page that uses it: it loads the example dictionaries or a folder the visitor chooses, decodes a payload, and draws the payload bit by bit with the bits of each field marked. Nothing is sent anywhere; the page needs only a static file server.
| Piece | Status |
|---|---|
build a dictionary from files in memory (Dictionary::from_files) | built |
decoder as a WebAssembly module with a JavaScript interface (bite-wasm) | built |
decoder page: bit-level view of a payload, visitor's own dictionary (site/decoder/) | built |
| publishing the page with the documentation site | not built |
Performance
bite decodes a weather-station message in 0.47 µs, 2.1 million messages per second on one core, and the whole pipeline handles 450,000 packets per second (performance.md). The pipeline makes one pass per packet: slice headers, one hash lookup for the interface, one for the message, one bit extraction per field, one serialisation of the record.
Presenting first-bite
| What | Where | Rebuild |
|---|---|---|
| Illustrations, light and dark | docs/img/ | make figures |
| Technical brief (PDF, A4, eleven pages) | docs/brief/first-bite-brief.pdf, from docs/brief/brief.html | make brief |
| Documentation site | site/, built into site/_build and published by .github/workflows/pages.yml | make site |
The figures, the site and the brief share one palette and one set of numbers: docs/img/README.md records the visual system, and every figure quoted is in performance.md.
The source of this page is a Markdown file in the repository.