Architecture

first-bite turns frames on a wire into decoded records on a bus. It reads frames from a network interface or a capture file, decodes the payload of each against a dictionary, and publishes one record per packet (JSON by default, CBOR selectable) to ZeroMQ, from where sinks forward records to object storage, REST endpoints, a Zenoh fabric or a file.

Frames come from a network interface, a capture file or a UDP socket into the mind binary. Its source hands each frame to its pipeline, which passes the payload to the bite library; bite decodes it against the dictionary and returns typed fields. The pipeline hands a record to the flow library, whose ZeroMQ publisher feeds sinks and transforms; the Cursor-on-Target transform uses the cotify library. The datamodels crate holds the record type and logger the log format.
Two native systems exchange binary messages on a link. A tap or a switch mirror port sends a copy of every frame one way to first-bite, which captures it, decodes it against a dictionary of JSON files and publishes one record of typed values per message, as JSON or CBOR, on a ZeroMQ PUB socket. Object storage, an HTTP endpoint, a Zenoh data fabric, a Cursor-on-Target transform and a subscriber of the owner's own read the records. Nothing is sent back to the link, so the native systems are not delayed, blocked or altered.

Crates

CrateKindDoes
bitelibrary and bite CLIloads and validates a dictionary; decodes a payload into typed fields; exports JSON Schema. Knows nothing about networks.
bite-wasmWebAssembly modulethe bite library for browsers: a dictionary from files in memory, decoding, JSON Schema and vectors, called from JavaScript
mindbinary mindreads frames from a source, finds the transport payload, asks bite to decode it, publishes the record
flowlibraryZeroMQ publishers, and subscribers that forward records to sinks or through transforms
cotifylibrarybuilds Cursor-on-Target XML from a decoded record and a field mapping
datamodelslibrarythe record type that crosses crate boundaries
loggerlibrarylog line format

The dictionary is data

Nothing about a message set is compiled in. A dictionary is a directory of JSON files that names interfaces, the messages of each, and the bit range, type and value labels of every field; dictionary-format.md is the reference. mind loads the directory named by --dictionary or BITE_DICT_PATH once at start into an immutable, indexed structure.

A dictionary for a message set that cannot be published lives outside this repository and is used the same way. bite check <dir> validates it and runs the test vectors it carries, so a private dictionary is tested by the public decoder without its contents entering this tree.

From frame to record

  1. Source. mind/src/source.rs defines FrameSource, with three implementations: LiveCapture (libpcap on an interface, promiscuous, with an optional BPF filter), FileCapture (a pcap or pcapng file, read once) and UdpFrames (one Ethernet frame per UDP datagram, for running without capture privileges). A frame is a timestamp, a link type and borrowed bytes.
  2. Pipeline. mind/src/pipeline.rs slices the frame with etherparse, keeps UDP and TCP packets, and looks up the interface by port (Dictionary::interface_for_ports, source port first). The interface reads the message type from the payload and decodes every field of that message.
  3. Record. Every UDP or TCP packet yields one PacketInfoWithParsedMessage (datamodels/src/packets.rs): timestamp, addresses, ports, the payload and the whole frame in base64, deployment metadata from DATA_SOURCE_* environment variables, and, when the port belongs to an interface, the interface name (subsystem), the message name (message_type) and the decoded fields (message). A packet on a port no interface claims is still published, undecoded.
  4. Publish. flow sends the record on each configured ZeroMQ PUB socket under a topic derived from the record, by default interface/message.

Field values in message keep their type: integers and scaled values are numbers, booleans are booleans, the dictionary's label stands where the raw value has one, and a field beyond the end of a short payload is null. bite schema <dir> prints a JSON Schema for it. record.md is the record's contract: every field, the earlier all-text version 1 (FIRST_BITE_RECORD_VERSION=1), and the two encodings, JSON and CBOR, a publisher or sink can write.

Sinks and transforms

A subscriber is a ZeroMQ SUB socket with a topic filter and a sink. Subscribers are listed in the YAML file named by SUBSCRIBER_CONFIG_PATH; with no file there are none, and records are available only on the PUB socket. Each sink is a cargo feature of flow, forwarded by mind.

Sinksubscriber_typeFeatureStatus
Azure Blob Storageobjectstorage / azureobjectazure (default)built
S3-compatible storageobjectstorage / awsobjectaws (default)built
HTTP POSTrestalwaysbuilt
local filelocalalwaysbuilt
UDP datagrams, unicast or multicast; Cursor-on-Target with a cot: mapping (cot-to-tak.md)udpalwaysbuilt
TCP stream, optionally inside TLS with a client certificate; Cursor-on-Target with a cot: mappingtcpalwaysbuilt
Zenohzenohzenohbuilt
Zenoh on a goat data fabric (optional, off by default; goat/joining.md)goatgoatbuilt

The ZeroMQ sockets are plaintext and stay on the host; the HTTP sinks and the Zenoh sink are the paths that cross a network. A security block in the configuration states the level every path is held to (plaintext on loopback only, TLS, or mutual TLS, with a minimum TLS version, a key-exchange policy and an optional requirement for the FIPS module), mind refuses to start when a path cannot meet it, and mind security show prints what each path will use. All TLS goes through one crypto provider installed at start. security/transport.md is the guide.

A subpub is a subscriber, a transform and a publisher: it reads records, turns each into something else, and publishes the result on another ZeroMQ socket. The transform a build always has is Cursor-on-Target (subtype: cot), driven by a mapping in the configuration that says which fields give the identifier, the type and the position. Subpubs run when mind is started with --activate-subpubs. A udp or tcp subscriber takes the same mapping and sends the event to a TAK server or to TAK clients, with no subpub between.

A build with the cargo feature goat-transformers has a second transform, subtype: xf. It hands the record's payload to the goat-transformers engine, which reads each format from a data file, a pack: tools/bite2pack writes the pack of a dictionary, and the target (Cursor-on-Target XML, GeoJSON) has one of its own. The feature is off by default and cot is unchanged by it; flow/README.md holds the measured comparison of the two.

Cursor-on-Target here is a per-message field mapping. A node that needs tracks normalised into Cursor-on-Target, ASTERIX CAT062 or SAPIENT with one identity scheme across sensors runs Open Billie for that; related-projects.md says how first-bite fits with it and with octo-pod.

Beyond IP

bite decodes a payload and, optionally, a frame identifier; it does not care where they came from. The vehicle-can dictionary identifies messages by CAN identifier and reads little-endian signals in lsb0 bit numbering, and its vectors pass. What exists for non-IP links today is the decoder and the bite decode --frame-id command.

PieceStatus
decode CAN-style frames: frame_id discriminator, lsb0 numberingbuilt
a FrameSource for SocketCAN, and a record shape for frames with no IP addressesdesigned, not built
importing a DBC file as a dictionarynot built
MIL-STD-1553 and ARINC 429 sourcesnot built

In the browser

The library half of bite has no dependency on a filesystem or a network: Dictionary::from_files builds a dictionary from file paths and text held in memory, and gives the same dictionary and the same issues as Dictionary::load gives for a directory with those files. bite-wasm compiles it for wasm32-unknown-unknown and gives it a JavaScript interface. make wasm writes the module to site/pkg/, and site/decoder/ is a static page that uses it: it loads the example dictionaries or a folder the visitor chooses, decodes a payload, and draws the payload bit by bit with the bits of each field marked. Nothing is sent anywhere; the page needs only a static file server.

PieceStatus
build a dictionary from files in memory (Dictionary::from_files)built
decoder as a WebAssembly module with a JavaScript interface (bite-wasm)built
decoder page: bit-level view of a payload, visitor's own dictionary (site/decoder/)built
publishing the page with the documentation sitenot built

Performance

bite decodes a weather-station message in 0.47 µs, 2.1 million messages per second on one core, and the whole pipeline handles 450,000 packets per second (performance.md). The pipeline makes one pass per packet: slice headers, one hash lookup for the interface, one for the message, one bit extraction per field, one serialisation of the record.

Presenting first-bite

WhatWhereRebuild
Illustrations, light and darkdocs/img/make figures
Technical brief (PDF, A4, eleven pages)docs/brief/first-bite-brief.pdf, from docs/brief/brief.htmlmake brief
Documentation sitesite/, built into site/_build and published by .github/workflows/pages.ymlmake site

The figures, the site and the brief share one palette and one set of numbers: docs/img/README.md records the visual system, and every figure quoted is in performance.md.

The source of this page is a Markdown file in the repository.