Skip to content

Error reference — goat and goatbuilder

Generated from ops/errors/catalog.yaml by ops/ci/check-error-catalog.py --write; do not edit here. Look a code up with goat explain <code> or goatbuilder explain <code>; paste any error text to goat explain - / goatbuilder explain - to have it interpreted (ADR 1199, error-interpretation.md).

goat

Code Tool What it means Why What to do Read Exit
E_PROFILE_MISSING goat no profile for this context — goat was run before goat profile init, or the context named has no profile.toml, certificate or key. the bundle was never installed on this machine
the bundle was installed into a different context than the one in use
goat profile init <bundle.cbor>
goat --context <name> … (when the bundle was installed under another context)
docs/design/goat-cli-error-codes.md 3
E_PROFILE_DENIED goat the profile exists but this user cannot read it — the profile directory is there and permission was denied reading it, typically a root-owned 0700 ~/.goat probed without sudo, or bare sudo that reset $HOME to /root. the profile was created as root
bare sudo goat … looked in /root/.goat
sudo chown -R $USER ~/.goat
sudo -E goat … (keeps your $HOME)
docs/design/goat-cli-error-codes.md 3
E_BUNDLE_INVALID goat the bundle did not verify — at goat profile init the bundle's structure, signature or expiry failed. The help line says which of the three. a truncated or wrong download (malformed CBOR)
a signing CA this machine does not trust
an expired bundle
goat ca ls (what this machine trusts)
goat ca add <root.pem> (when the CA is the problem)
ask the operator for a re-issued bundle when it has expired
docs/design/goat-cli-error-codes.md 3
E_OFF_MESH goat not connected to the mesh — the local mesh client reports no connected tunnel, so nothing on the goatnet is reachable. the mesh client is stopped or logged out
this device was never enrolled
the device is on a network that blocks the mesh
netbird status -d
goat heal && goat doctor
docs/design/goat-cli-error-codes.md 5
E_CLOCK_SKEW goat the device clock is wrong — the device clock differs from the management server's authenticated HTTPS Date by more than 60 seconds, so certificates cannot be judged valid. the clock drifted while the device was off or offline synchronise the clock (chrony/ntp), then: goat doctor docs/design/time-hygiene-trusted-time.md 6
E_ROUTER_UNREACHABLE goat the Zenoh router did not answer — the configured router's TCP path or Zenoh session timed out or could not be reached. the mesh tunnel is up but the route to the router is not
the router service is down on its host
the endpoint in the profile is stale
goat doctor --all
compare from another enrolled peer before changing anything local
docs/design/goat-cli-error-codes.md 5
E_CERT_EXPIRED goat the mTLS certificate is outside its validity — the local certificate is expired, not yet valid, or was rejected by the router. the certificate's validity interval has passed
the device clock is wrong (see E_CLOCK_SKEW)
goat profile show
goat profile rotate <new-bundle.cbor> (after the operator re-issues it)
docs/design/internal-tls-ca.md 6
E_DOCTOR_INCONCLUSIVE goat doctor saw a failure it cannot narrow — a local TLS or configuration failure, or a session rejection, without enough evidence for one cause. router-side admission refused this identity
a local TLS setup problem the probe cannot distinguish from the router's
follow the hint on the failing doctor step
if the local certificate is valid, send the step output to the goatnet operator
docs/design/goat-cli-error-codes.md 1
E_PORTAL_UNREACHABLE goat the portal API did not answer — the portal was not reachable for network reasons (DNS, TCP or TLS), not an authorization failure. the portal host is not reachable over the mesh from this device
mesh DNS is not resolving the portal name
goat doctor --all
check from another peer: if it answers there, the problem is local
docs/design/goat-cli-error-codes.md 4
E_PORTAL_NOT_FOUND goat the portal answered that what was named is not registered — the portal is up and answered HTTP 404 for the request path shown: the org, topic, schema, machine or person the command named does not exist there. An answer, not a network fault. a typo in the org prefix, topic key or handle
the thing is registered on another goatnet than the active context's
it was removed, or never created
goat vendor ls (the orgs that exist)
goat topic ls <org>/ (the topics under one org)
goat context show (which goatnet this context talks to)
docs/design/goat-cli-error-codes.md 7
E_PORTAL_RATE_LIMITED goat the portal is rate-limiting this client — the portal is up and answered HTTP 429: this client sent too many requests. The hint carries the portal's Retry-After when it sent one. a script calling goat in a tight loop
an older goat whose topic ls with no prefix made one request per org
wait the Retry-After the hint names, then retry
space the calls out, or list one org at a time: goat topic ls <org>/
goat update (current goat lists the whole goatnet in one request)
docs/design/goat-cli-error-codes.md 8
E_PORTAL_SERVER_ERROR goat the portal failed on its own side — the portal is up and answered HTTP 5xx for the request path shown. Nothing on this device's network or credential is at fault. the portal or a service behind it (its database, the registry) is failing
the portal is restarting during a deploy (often a 502 or 503)
retry after the Retry-After the hint names, or in a minute
if it persists, send the code, status and path to a platform admin; the portal's log on the goatnet's management host names the cause
docs/design/goat-cli-error-codes.md 9
E_PORTAL_REFUSED goat the portal refused the request — the portal is up and answered a non-2xx status no command curates (a 400, 405, 410, 413 or 422) for the request path shown. this goat build and the portal are at different API versions
a value the command sent is one the portal does not accept
goat doctor (shows whether goat is behind the served version)
goat update
docs/design/goat-cli-error-codes.md 4
E_PORTAL_BAD_RESPONSE goat the portal answered in a shape this goat build does not understand — the portal returned 2xx but the body did not deserialize. The portal is healthy; the CLI and the portal are at different versions. goat is behind the portal
the portal deployed ahead of the CLI
goat update (or re-install from get.<site>)
goat --version, and report the endpoint if already current
docs/design/goat-cli-error-codes.md 4
E_TOKEN_INVALID goat the portal rejected the token — the portal accepted the connection and refused the bearer token (HTTP 401): the credential itself is expired or misissued. the token in ~/.goat/contexts//portal.token has expired
the token belongs to another goatnet
goat profile rotate <new-bundle.cbor> (after re-running onboarding) docs/design/goat-cli-error-codes.md 4
E_PORTAL_FORBIDDEN goat the portal refused this identity — HTTP 403 to the probe. The token was accepted; the identity lacks the access. the active context is a different identity than you meant
the grant for this surface was never made
goat context show
ask the goatnet operator for the access grant; rotating the token will not grant it
docs/design/goat-cli-error-codes.md 4
E_NOT_ADMIN goat an operator-only action without operator authority — a goat admin action was refused, client-side (no operator credentials in the profile) or by the portal/NetBird API (HTTP 403, nothing changed). the profile is using the vp_ participant token, not the op_ operator key
the role is not authorized for this action
GOAT_PORTAL_TOKEN=op_… goat admin …
goat admin operator-key request --reason … (when you hold no op_ key)
docs/design/platform-admin-lifecycle.md 4
E_TOPIC_CONFLICT goat the topic already exists — goat topic create was asked for a name that is already registered. someone created it first
a re-run of a create that already succeeded
goat topic show <topic>
pick a new name, or version the topic (docs/design/pubsub-best-practices.md)
docs/design/goat-cli-error-codes.md 1
E_MACHINE_UNRESOLVED goat the machine named is not exactly one enrolled machine — a goat machine verb was given a handle or slot anchor that matches no enrolled machine this profile can see, or a handle that matches machines under more than one custodian. Nothing was changed. a typo in the handle
the machine is under another custodian's custody, which a non-operator profile cannot see
the machine was revoked
two custodians each enrolled a machine with the same handle
goat machine ls (shows each machine's handle, custodian and slot anchor)
pass the 32-hex slot anchor instead of the handle when the handle is ambiguous
docs/design/goat-cli-error-codes.md 1
E_MESH_PEER_UNRESOLVED goat the enrolled machine has no single mesh peer — the machine is enrolled, but no NetBird peer carries its name, or several do. A machine's peer is named machine-<handle>-<host> by its setup key, or <handle> when renamed by hand. Nothing was changed. the machine never joined the mesh with its setup key
the peer was renamed off its handle in the NetBird dashboard
the machine joined twice, leaving a stale peer beside the live one
goat admin peer ls (find the machine's peer and its name)
goat machine mesh-key <machine> (when it never joined)
rename the peer back to, or remove the stale peer with goat admin peer cut <peer-id>
docs/design/goat-cli-error-codes.md 1
E_NETBIRD_FAILED goat the NetBird management API call failed — a call to the goatnet's NetBird management API did not succeed for a reason other than authorization, so nothing was changed. The help line carries the status or the transport error. the NetBird admin token was rejected (HTTP 401)
the management server is unreachable from this machine
the management server answered an unexpected status
the peer record lacked a setting the update has to send back unchanged
check the token: export <CONTEXT>_NB_API_TOKEN (or GOAT_NB_API_TOKEN) with a current admin token
goat admin peer ls (proves the URL and token work)
retry once the management server answers
docs/design/goat-cli-error-codes.md 4
E_ACL_DECLARATION_MISSING goat no ACL declaration was supplied — goat acl test or goat warrant needs the goatnet's netbird_acl_taxonomy.yml and netbird_acl_matrix.yml and found them nowhere. goat carries no deployment's declaration, so it stops rather than check other rules. the command ran outside a checkout and the context holds no declaration
the goatnet does not serve its declaration at get./acl/, so goat context refresh fetched none
goat acl test --declaration <dir> (a directory holding both files)
export GOAT_ACL_DECLARATION_DIR=<dir>
set acl_declaration_dir = "<dir>" in ~/.goat/contexts/<name>/profile.toml
goat context refresh (fills ~/.goat/contexts/<name>/acl/ when the goatnet serves it)
run from a checkout that carries ops/ansible/netbird_acl_taxonomy.yml and netbird_acl_matrix.yml
docs/design/goat-cli-error-codes.md 1
E_RECIPIENT_KEY_UNPUBLISHED goat this anchor's recipient key is not in the recipient-cert store — goat access key asked the portal's recipient-cert store for the anchor's public leaf cert and the portal answered 404, so the key service cannot wrap sealed data to that anchor. The text after the dash is the portal's own reason. the anchor's leaf was minted before the recipient-cert store existed, so its public cert was never stored
the anchor is not the one meant (a typo, or an anchor from another goatnet)
goat access key --anchor <anchor> (check the anchor; with no --anchor it checks your own profile's)
goat admin rotate <anchor> (an operator re-issues the leaf in place, ADR 1044, which stores its public cert; install the new bundle's cert and key on the peer)
goat access key --anchor <anchor> (run again: it reports PUBLISHED)
docs/design/goat-cli-error-codes.md 1

goatbuilder

Code Tool What it means Why What to do Read Exit
GB-001 goatbuilder unknown component — the component named is not declared in ops/pins/fleet-pins.yaml (or, on a fixture, in goatbuilder.yaml). a typo in the component name
the component is declared under its ledger name, not its image name
grep -n '^ [a-z0-9-]*:' ops/pins/fleet-pins.yaml \| head -60
goatbuilder validate
docs/design/goatbuilder.md 2
GB-002 goatbuilder unknown goatnet — the --env value is not a goatnet key the pins declare. A key is spelled exactly as the pins spell it, underscores included. the goatnet was typed with a hyphen where its key has an underscore goatbuilder deploy --env <goatnet> …
goatbuilder doctor --all (lists every environment)
docs/design/goatbuilder.md 2
GB-003 goatbuilder the group is not one the component deploys to — --group named a group that is not a member of the goatnet, or one where the component is declared absent. the group belongs to another goatnet
the pins mark the component absent: true for that group
goatbuilder deploy --env <env> --component <c> --dry-run (prints the member groups) docs/design/goatbuilder.md 2
GB-004 goatbuilder the component declares no image variable — the pins entry has no var: so there is nothing to template into a play. an incomplete pins entry addvar:to the component in ops/pins/fleet-pins.yaml, then: goatbuilder validate docs/design/goatbuilder.md 2
GB-005 goatbuilder no deploy playbook resolves for the component — neither -deploy.yml nor -deploy.yml exists in ops/ansible, or the declared deploy_playbook is not there. the component is deployed by another play and does not declare it
the playbook was renamed
declare deploy_playbook: <play>.yml on the component in ops/pins/fleet-pins.yaml
goatbuilder doctor --all (the playbook row names what was looked for)
docs/design/goatbuilder.md 2
GB-006 goatbuilder the door to the goatnet did not open — the goatnet's management host was not reachable over the mesh, so the SSM door was tried, and it did not open. no AWS credentials in the keychain for aws-vault (the profile in deploy.aws_profile)
the SSM agent on the management host is not connected (TargetNotConnected)
the door registry row is stale
ops/scripts/door-open.sh <goatnet> --status
aws-vault exec <profile> --no-session -- aws sts get-caller-identity
docs/design/env-doors.md 1
GB-007 goatbuilder a secret the play needs is not in your environment — the play templates one or more variables from the environment and they are not set, so the run was refused before any host was touched. the refs file for this component was never written
the command was run without op run
put the lines for the named variables from <owner>/goat-credentials refs/goat.refs into .envrc.local.d/<file>.refs
OP_ACCOUNT=<account> op run --env-file=.envrc.local.d/<file>.refs -- <the same command>
docs/design/secret-store-requirements.md 1
GB-008 goatbuilder a mutation needs --yes when there is no terminal — deploy, release, rollback, fastlane, standup, teardown, join-node, federate, apply and grow change state; without a terminal to confirm on, they refuse unless --yes is passed (contract C10). grow asks only when stdin and stderr are both a terminal, so with stderr sent to a file it needs --yes too. run from a script, CI or a pipe add --yes
add --dry-run to see the plan without changing anything
docs/design/goatbuilder.md 2
GB-009 goatbuilder not a current main checkout — releases and deploys run from the invoking checkout, and only a main checkout kept current with origin carries the pin that was merged. A branch or detached checkout can deploy a stale pin and verify green. running from a feature-branch worktree
running from a detached HEAD
main is behind origin/main
git checkout main && git pull --ff-only origin main
goatbuilder doctor --all (the checkout row)
docs/design/release-candidate-validation.md 1
GB-010 goatbuilder a Python dependency is missing — the pins and intents render through PyYAML and jsonschema, and python3 cannot import one of them. a fresh machine
a venv that is not on PATH
pip install pyyaml jsonschema (or pipx/venv), then re-run; re-running is safe docs/design/goatbuilder.md 2
GB-011 goatbuilder only a repository admin may nominate a release candidate — the GitHub login in use does not hold the admin role on the repository; the rc-validate workflow deletes a tag pushed by anyone else (ADR 1173). gh is logged in as a different account
the account is a maintainer, not an admin
gh auth status
ask a repository admin to push the tag, or to grant the role
docs/design/release-candidate-validation.md 1
GB-077 goatbuilder this checkout's origin is not where release candidates are cut — the repository that origin names does not run rc-validate.yml, so a tag pushed there is never validated or built where the ladder reads images; the run refuses before the tag (ADR 1173). the checkout is a clone of a mirror or a fork, not the primary repository git remote get-url origin
run from a checkout whose origin is the primary repository
docs/design/release-candidate-validation.md 1
GB-078 goatbuilder no release signer is known, so the update refuses — goatbuilder update checks SHA256SUMS against the repository whose release workflow signed it. It found none: the working directory is in no checkout whose ops/goatbuilder/goatbuilder.yaml names repo.release_source or repo.source, and this binary was built outside a release workflow, so it names no publisher itself. Nothing was installed. goatbuilder update ran outside a deployment checkout with a locally built binary goatbuilder -C <checkout> update (a checkout whose goatbuilder.yaml names repo.release_source)
install a goatbuilder from a release, which carries its own publisher
docs/design/goatbuilder-distribution.md 1
GB-265 goatbuilder the run is waiting on the other driver — The peering is declared with two drivers (drivers: two): an edge driver that reaches only the edge host and a controller that reaches the core. This phase needs the edge, so the controller stopped before acting and handed it to the edge driver. The stop prints the exact command to run there, the file it writes, and the two commands to run here after it; with -o json the report carries them as the member handoff. It stops as a by-hand stop does (exit 0). the peering's phase needs the edge host, and this machine is the controller run the command the stop names on the edge driver, bring its result file here, then run the import and the command after it that the stop names docs/design/federation-two-drivers.md 0
GB-266 goatbuilder the controller's checkout is not clean, so no hand-off is given — A two-driver peering's hand-off pins the edge driver's checkout to the controller's commit, so the controller gives one only when its own checkout is a git checkout with nothing uncommitted. The run stopped at the phase that needs the edge, before acting, and names each file with changes. phase 10 placed rendered files, or admission wrote peerings.yml, and they are not committed yet
a file was edited by hand and not committed
the deployment's directory is not a git checkout
commit each file the refusal names, then run the same command again docs/design/federation-two-drivers.md 1
GB-269 goatbuilder a secret for the node still holds a secret store's reference — A value join-node hands to a node, such as the first-login password phase 2 seeds the SSH key with, still begins with a secret store's reference (op://, bao://, bw:// or the declared ref_prefix) after the secrets injector ran. Sent as it is, the node would take the reference as the secret. Nothing was sent to the node. The message names the variable and the scheme, never the reference. the store's tool is not signed in, so with-secrets could not resolve the reference
the variable was set to a reference under a store that resolves none (env, env-file)
sign in to the declared store's tool (for 1password: op signin), then re-run: goatbuilder join-node <environment> --site <site> --node <node> --class <class> --address <address> --from 2
or export the value itself in the variable the message names
docs/design/goatbuilder.md 1
GB-267 goatbuilder this part of a two-driver peering is not built yet — A peering declared with two drivers (drivers: two) is handed off between the edge driver and the controller only at phases 1, 2 and 3, where the hand-off is built. These cases are refused before anything runs: the edge driver's own run (goatbuilder federate <partner> --side edge) of a phase other than 1 or 3; the controller's goatbuilder federate import of a phase other than 1 or 3; a two-driver run that would reach a phase after 3 that needs the edge (11, 12, 13 or 14), whose hand-off is designed and not built; and a two-driver run from phase 3 with FED_MEASURE=here, which measures one machine and so has no halves to hand off. That includes every goatbuilder federate deploy, which runs phase 10 then 11: it is refused before its plan is approved or rechecked, so phase 10 never admits the edge on our core. goatbuilder federate --side edge was run on the edge driver for a phase other than 1 or 3
goatbuilder federate import --sha256 was run for a phase other than 1 or 3
a two-driver run that would reach phase 11, 12, 13 or 14, such as --from 4 with no --to, or goatbuilder federate deploy (a run from 4 to 10 is not refused)
a two-driver run from phase 3 with FED_MEASURE=here
run the peering with one driver for now: remove drivers: two from its peerings: entry, from a machine that reaches both the edge host and the core host
for phase 3: unset FED_MEASURE (or set it to ssh), so phase 3 measures in halves over SSH
docs/design/federation-two-drivers.md 1
GB-271 goatbuilder the peering's WireGuard interface does not hold the agreed peer and address — Phase 11 writes the agreement's link values (the counterparty's public key, its allowed address and our link address) to the link's environment file, restarts goat-peering-wg@.service and reads the interface back with wg show <interface> allowed-ips and ip -4 addr show dev <interface>. The read-back did not find them, so the link the agreement describes is not up on this edge. The end of the message lists what the interface holds: each peer's public key and allowed addresses, then its addresses. No peer and no address is the interface as it is before phase 11. the link unit is still running with the environment it had before phase 11: the agreement's peer and address were written to the link's environment file, but goat-peering-wg@.service was not restarted to read them (a phase 11 cut off between writing the file and the restart, or run by a role from before the restart-on-read check)
the unit was restarted and goat-peering-wg-link up stopped partway: its journal says where
something other than this role changed the interface after the unit set it up (another peering's unit, or wg set by hand)
on the edge: sudo systemctl restart goat-peering-wg@<interface>.service, then sudo wg show <interface> and ip -4 addr show dev <interface>: the agreed peer and link address are there
if they are not: sudo journalctl -u goat-peering-wg@<interface>.service -n 50
goatbuilder federate <peering> --from 11
docs/operations/sovereign-federation-runbook.md 1
GB-272 goatbuilder the partner's acceptance result does not verify — In a bilateral acceptance session each side signs its phase 14 result and the other verifies it under the key bound to that side at phase 5, the same key that signs its federation package. The signature document beside the partner's result did not verify: it is malformed, it names another key, the result read is not the one it signs, or the signature does not verify under the bound key. Nothing from it is combined (docs/design/federation-acceptance.md §6.1). the result or its signature document was altered, or the result was read before the partner finished writing it
the partner signed with a key other than the one bound to them at phase 5 (their federation key was rotated, or the wrong key file was used)
the signature document is not goat-acceptance-signature/v1, or a member is missing or malformed
check with the partner, on the call, the key fingerprint confirmed at phase 5 and the one their signature document names
ask the partner to run phase 14 again in the same session, so a new result and signature are published
docs/design/federation-acceptance.md 2
GB-273 goatbuilder the partner's acceptance result is not from this session — The partner's signed acceptance result is for another agreement, session or attempt, is signed by or for another party, or does not echo the challenge issued to us, so it cannot be from this session's samples. It is refused before its signature is checked, and nothing from it is combined (docs/design/federation-acceptance.md §6.1). an earlier session's or attempt's result was read again (a replay, or a result left from before)
the two sides hold different agreements: the digests read aloud at phase 6 differ
our own result was read back as if it were the partner's
compare the agreement digest and the session id with the partner on the call
run phase 14 again on both sides in one session
docs/design/federation-acceptance.md 2
GB-274 goatbuilder the two sides derived different acceptance sessions — Both sides of a peering work out the same acceptance session from the agreement each holds and an attempt number, with no file between them (docs/design/federation-acceptance.md §6.2). The partner's session is not the same as ours: the refusal names the members that differ. Nothing is combined with a partner on another session. the two sides hold different agreements: the digests read aloud at phase 6 differ, or one side composed again since
the two sides are on different attempts
a session file was edited, so its digest no longer covers its members
compare the agreement digest and the attempt number with the partner on the call
run phase 14 again on both sides with the same attempt
docs/design/federation-acceptance.md 2
GB-275 goatbuilder phase 12 would deploy an edge that has none of the files it mounts — Phase 8 mints the peering leaves, and places our CA beside them, in /pki on the machine goatbuilder federate runs on. The edge router mounts each of those files by path on the edge host. The edge group's variables point the edge at files in /pki, the inventory names the edge as another machine, and goat_zenoh_edge_install_certs is off, so nothing copies them there: the edge would start with nothing to mount and never reach the core. Phase 12 is refused before it starts. goatbuilder federate runs on an operator's workspace, not on the edge host, and the edge group's install of the peering certificates is off
the inventory names this machine by an address of its own, not as ansible_connection=local
set goat_zenoh_edge_install_certs: true for the edge group (the role then installs the leaves, the partner's CA and our CA's certificate from this machine, never a CA key); run goatbuilder federate allocate again when the deployment is rendered; then goatbuilder federate <peering> --from 12
if this machine is the edge host, mark it ansible_connection=local in the inventory
docs/design/federation-certificate-lifecycle.md 1
GB-285 goatbuilder the directory binds our previous signing key, and the seed that hands it over is missing — Phase 5's directory step publishes our pointer signed by the key that signed our package. The directory binds our sovereign_id to the first key that wrote it, and takes a new key only through the entry, when the request is signed by the key it replaces. The directory binds another key than ours: our signing key was rotated, and FED_PREVIOUS_SIGNING_KEY is not set, is not readable or holds the seed of a third key, or the pointer names another of our peerings' packages. Nothing was published; the directory still serves what it held. The step is advisory, so phase 5 goes on. our signing key was rotated (FED_SIGNING_KEY names the new seed) and FED_PREVIOUS_SIGNING_KEY was not set
FED_PREVIOUS_SIGNING_KEY names a seed other than the one the directory binds (the message names both keys)
with one package per counterparty, the pointer names another peering's package, and the key is handed over only through the pointer
FED_PREVIOUS_SIGNING_KEY=<file of the seed that signed our last package> goatbuilder federate <partner> --from 5
with several peerings, run phase 5 of the peering whose package the pointer names first; the others then publish under the new key
if the previous seed is lost, the directory's operator clears the entry (saltlick clear), and the next write claims it again
docs/design/federation-key-rotation.md 0
GB-261 goatbuilder the release's signature does not verify, so the update refuses — goatbuilder update verifies the keyless signature over the release's SHA256SUMS (SHA256SUMS.cosign-bundle) itself, offline, against the Sigstore trusted root: the certificate chains to a Fulcio CA, Rekor's signed entry timestamp verifies, the entry names this file, and the certificate is the release signer's goatbuilder-release.yml at the served version's goatbuilder/v tag. One of these failed, or the surface serves no signature. The message names the check. Nothing was installed. the surface serves a SHA256SUMS or a binary that the release workflow did not sign
the signature is another repository's, another workflow's, or another version's
Sigstore rotated its Fulcio CA or Rekor key after this goatbuilder was built
the surface serves no SHA256SUMS.cosign-bundle
check that the surface serves the release unchanged: the goat-get role's goat_get_goatbuilder_release_tag, and the release page's SHA256SUMS
after a Sigstore key rotation: GOATBUILDER_UPDATE_TRUSTED_ROOT=<a trusted_root.json holding the new key> goatbuilder update
goatbuilder -C <checkout> update, so that repo.release_source names the repository that publishes goatbuilder
docs/design/goatbuilder-distribution.md 1
GB-262 goatbuilder the verb has no such phase — goatbuilder explain <verb> <phase> answers from the phase catalog (ops/phases/catalog.yaml), by the phase's number or its id. The verb has no phase by that number or id. The message names the phases it has. a phase number outside the verb's range, or a mistyped id goatbuilder explain <verb> (lists the verb's phases, by number and id) docs/design/goatbuilder-legibility.md 2
GB-012 goatbuilder the candidate's image build did not succeed — after the tag was pushed, the image workflow did not appear, or ran and failed, so there is no image to validate. the workflow is not triggered by this tag shape
a build failure on the tagged commit
gh run list --workflow <component>-image.yml --limit 5
gh run view <id> --log-failed
docs/design/release-candidate-validation.md 1
GB-013 goatbuilder the image is not in the registry — the tag's image was not found under repo.image_namespace, so no digest can be pinned. crane is missing or unauthenticated — rc.sh resolves the digest with crane and discards its error, so a missing crane reads as the image not existing
the build published under another name
the package is private and the token cannot read it
brew install crane (then re-run; goatbuilder doctor --all: the tool:crane row says when it is present)
crane digest <repo.image_namespace>/<image>:<tag>
gh api /orgs/<owner>/packages/container/<image>/versions \| head
docs/design/release-candidate-validation.md 1
GB-014 goatbuilder a live validation row failed — the candidate deployed and a live row on the goatnet failed. On dev the candidate is dead; on preprod after a dev pass it stays open for investigation. a row exited 2 — it was never run: its toolchain is missing on this laptop (Playwright for the intake rows, a goat context for the invite rows); the candidate is untested, not dead
a real regression in the candidate
the row's runner settings are stale for this goatnet
if the row exited 2: install what it names (goatbuilder doctor --browser-tests fails on a missing browser toolchain, tool:live-rows; tool:goat-contexts names a missing goat context), then resume with --from rows — do not delete the tag
ops/validation/flow-map.yaml names the rows; read the failed row's evidence under ops/validation/evidence/
fix on main and cut the next rc (dev), or investigate before graduating (preprod)
docs/design/release-candidate-validation.md 1
GB-015 goatbuilder the checkout could not fast-forward after the pin landed — the pin PR merged, and the local main has diverged from origin/main, so the deploy could not proceed from a current tree. The PR is in; re-running resumes. local commits on main git -C <repo> status; move local commits to a branch; git pull --ff-only origin main; re-run the same command docs/design/release-candidate-validation.md 1
GB-016 goatbuilder no runner settings for the goatnet — ops/validation/goatnet-runners.yaml has no entry for the goatnet the live rows should run on. a new goatnet without its runner row add the goatnet's runner block to ops/validation/goatnet-runners.yaml docs/design/release-candidate-validation.md 1
GB-017 goatbuilder not converged on every host — the play ran, and at least one host's ledger does not name the declared digest with action=deploy. Done is read from the ledger, never assumed. the play failed part-way on that host
the host is single-homed on another router role and was skipped by design (ADR 1084)
the host was unreachable
goatbuilder deploy --env <env> --component <c> --verify-only
read the log named in the deploy row; grep -n 'fatal:' <log>
docs/design/universal-deploy-rollback.md 3
GB-018 goatbuilder a required flag is missing or wrong — release needs --ack and --version or --tag; --from takes tag|dev|rows|preprod|prod; --from prod needs --tag. the command was typed from memory goatbuilder release --help docs/design/goatbuilder.md 2
GB-019 goatbuilder the goatnet's CA key is not on this machine — re-signing the intents needs //ca.key.age and it is not there. this laptop never held that goatnet's CA key run from the machine that holds it, or obtain it through the credentials runbook in <owner>/goat-credentials docs/design/consistency-foundation.md 1
GB-020 goatbuilder graduate.py did not produce the expected record — graduation ran, and wrote no record, or the record names a different candidate_ref than the tag resolves to. an open graduation set blocking the move (ADR 1125)
a candidate record edited by hand
python3 ops/release/graduate.py status
read the record under ops/release/records/
docs/design/release-graduation.md 1
GB-021 goatbuilder fastlane refused — fastlane is the deliberate exception to the ladder and refuses an unpinned digest, an open graduation set, or an artifact whose rollback is not a pin revert. the ref has no digest under repo.image_namespace
a graduation set is open for the component
goatbuilder fastlane <component> --ref <ref> --dry-run
python3 ops/release/graduate.py status
docs/design/release-graduation.md 1
GB-022 goatbuilder intent-sign failed — re-signing a goatnet's intents after a pin move failed; the signed index is unchanged. the CA key passphrase was wrong or age could not decrypt it
the enrollment Go module is not at ops/enrollment
cd ops/enrollment && go run ./cmd/intent-sign … (the fix line names the goatnet) docs/design/consistency-foundation.md 1
GB-023 goatbuilder rollback refused or guided by artifact type — the component's artifact type is data, cert, forward-only or reversible:false (refused, exit 3), or config/acl (guided, exit 4): its rollback is not a pin flip. the component carries a migration or a certificate goatbuilder rollback <component> --status
follow the guided steps the row prints (config/acl)
docs/design/universal-deploy-rollback.md 3
GB-024 goatbuilder the fixture is not a shape the mock converger knows — on --mock, converger.yaml names a behaviour the scripted converger does not know, a record is unreadable, or nothing is declared for the environment. a hand-edited fixture compare with ops/goatbuilder/fixtures/basic/ docs/design/goatbuilder-conformance.md 2
GB-025 goatbuilder the config does not validate — ops/goatbuilder/goatbuilder.yaml failed the versioned schema or a cross-field rule. The config is generated; fix the source and regenerate. a hand edit to the generated file
a pins or ladder change the generator was not run for
python3 ops/goatbuilder/gen-config.py && goatbuilder validate docs/design/goatbuilder-conformance.md 1
GB-026 goatbuilder status reads real hosts only through a converger — without --mock, status has no converger to read hosts through yet. the push converger's status path is not built goatbuilder deploy --env <env> --component <c> --verify-only
goatbuilder status --mock ops/goatbuilder/fixtures/basic --env dev
docs/design/goatbuilder-conformance.md 2
GB-027 goatbuilder not inside a repository — the repository root could not be resolved from the working directory and -C was not given. run outside a checkout goatbuilder -C /path/to/goat … docs/design/goatbuilder.md 2
GB-028 goatbuilder a command the verb runs failed — git, gh, crane, python or another tool exited non-zero; its own words follow and are interpreted below when the catalog knows them. see the tool's message re-run the quoted command by hand to see its full output docs/design/goatbuilder.md 1
GB-029 goatbuilder the play failed on a group — ansible-playbook exited non-zero for the group; the log path is in the row. a fatal task on a host (see the ansible entries below)
a host unreachable over the chosen transport
grep -n 'fatal:' <log> \| head
goatbuilder doctor --env <env>
docs/design/goatbuilder.md 1
GB-030 goatbuilder no host of the group is in the loaded inventories — the group is declared in the pins but the inventories loaded for this goatnet carry no host for it, so nothing was deployed (declared, not built). a site declared ahead of its hardware
the wrong inventory pair for this goatnet
grep -n '^\[<group>\]' ops/ansible/inventory/*.ini docs/design/env-doors.md 1
GB-031 goatbuilder an op:// reference in your shell is incomplete — op run validates every environment variable whose value starts with op://, not only the env-file, and refuses the whole run when one has fewer than vault/item/field. a variable exported from your shell profile for another project holds a two-part op:// reference env \| grep 'op://' (find the variable)
env -u <THAT_VAR> op run --env-file=… -- <command>
or complete the reference to op://<vault>/<item>/<field>
docs/design/secret-store-requirements.md 1
GB-032 goatbuilder the candidate record is open but the goatnet runs a different ref — graduate.py's C6 check: a record for this component is still open while the goatnet now runs another ref — the record the preprod step superseded was never abandoned. a release run on a checkout older than #2039, which abandons the superseded record in the preprod change
a candidate re-cut while the previous record stayed open
re-run the same command from a current main: rc.sh andgoatbuilder releaseabandon the superseded record in the preprod change (git pull --ff-only origin main first on an older checkout)
python3 ops/release/graduate.py status (what is open)
docs/design/release-candidate-validation.md 1
GB-033 goatbuilder promote refused a graduation step — goatbuilder promote (the release ladder, ADR 1120/1125) refused: the record, the set, the ladder or the ref does not meet the rule the step names. Nothing graduates by itself; a refusal changes nothing. the component is not ready (soak, live rows, convergence evidence) — the lines under the refusal name each blocker
the component is a member of an open graduation set and must graduate with it
a ref that is not digest-pinned (…@sha256:…), or a hotfix without --ack
no open record for the component, or the record named does not exist or already graduated
goatbuilder promote status (what is open, soaking, ready)
goatbuilder promote readiness <component> (each blocker by name)
goatbuilder promote graduate-set <set> --ack '<name>' (when it is a set member)
docs/design/release-graduation.md 2
GB-034 goatbuilder the generated pins or intents differ from fleet-pins.yaml — goatbuilder pins --check regenerated every member's pins.yml and every goatnet's intent from ops/pins/fleet-pins.yaml and they differ from what is committed, or an intent is unsigned or tampered, or a governed pin is defined outside pins.yml. fleet-pins.yaml changed without regenerating
a hand edit to a generated pins.yml or intent
goatbuilder pins (regenerate), then re-sign the intents it names, then commit docs/design/consistency-foundation.md 1
GB-035 goatbuilder the goatnet's intent names no usable CA — an intent has no ca_id, or one from which the CA's directory under cannot be derived, so it cannot be signed. a new goatnet whose intent was written without ca_id add ca_id to the goatnet in ops/pins/fleet-pins.yaml, then: goatbuilder pins docs/design/consistency-foundation.md 1
GB-036 goatbuilder the component declares no image workflow — release finds a candidate's image build by the component's image_workflow in fleet-pins.yaml, and there is none. a component added without image_workflow add image_workflow: <name>-image.yml to the component in ops/pins/fleet-pins.yaml docs/design/release-candidate-validation.md 2
GB-037 goatbuilder the ladder is missing a live rung — ops/release/ladder.yaml has no live rung with a goatnet where the step needs one, or does not read. a rung marked not live while a goatnet is rebuilt
a malformed ladder.yaml
goatbuilder promote status (the rungs as read)
python3 -c 'import yaml; yaml.safe_load(open("ops/release/ladder.yaml"))'
docs/design/release-graduation.md 1
GB-038 goatbuilder --from preprod-rows cannot resume this candidate — resuming at the preprod rows needs preprod to run the candidate on main and exactly one open record for it; one of the two is missing. the preprod pin was never landed for this candidate
the record was closed, or a second one opened
goatbuilder release <component> --tag <tag> --from preprod --ack '<name>' (pins preprod and opens the record) docs/design/release-candidate-validation.md 1
GB-039 goatbuilder a rollback on a fixture needs its environment — on a fixture (--mock) a rollback declares the earlier ref in one environment and converges it there, so it needs --env; --list and --status read host ledgers, which a fixture does not have. --mock given without --env
--list or --status given with --mock
goatbuilder --mock <fixture> rollback <component> 1 --env <env> --ack "<your name>" --dry-run
goatbuilder --mock <fixture> status --env <env> (the state a fixture has)
docs/design/goatbuilder-conformance.md 2
GB-040 goatbuilder that directory is not a tutor workspace — the tutor creates its workspace in an empty or new directory and deletes only a directory that carries its .goatbuilder-tutor marker; it refused to use or delete one that does not. --workspace (or $GOATBUILDER_TUTOR) points at a directory with other files in it goatbuilder tutor --workspace <an empty or new directory>
unset GOATBUILDER_TUTOR (the default is ~/.goatbuilder/tutor)
docs/design/goatbuilder.md 2
GB-041 goatbuilder there is no such lesson — the tutor was asked for a lesson number it does not have. a number past the last lesson goatbuilder tutor --list docs/design/goatbuilder.md 2
GB-042 goatbuilder the live rows cannot run from this machine — the candidate is UNTESTED, not dead — a release step reached the live acceptance rows, and this machine lacks what they need (the console's Playwright toolchain for a browser row, or a tool or context the runner reported with exit 2). The rows did not run; nothing about the candidate failed. a browser row is mapped and ops/goat-id-console/web/e2e/node_modules is missing, or node/npm is not installed
the acceptance runner exited 2 (a tool or a goat context missing on this machine or on the host it runs goat on)
(cd ops/goat-id-console/web/e2e && npm ci && npx playwright install chromium), then re-run the same command
read the runner's output above for the missing tool or context, fix it, re-run the same command — never delete the tag
docs/design/release-candidate-validation.md 1
GB-043 goatbuilder the door to the rung did not open for its live rows — the rows for a rung this laptop is not meshed to run goat on that rung's management host through its door (goat_via door:), and the door did not open or printed no ssh target. aws-vault cannot reach the declared AWS profile (signed out, a malformed credential, or an expiring session)
session-manager-plugin is not installed
ops/scripts/door-open.sh <env> --status, then ops/scripts/door-open.sh <env>
aws-vault exec <profile> --no-session -- aws sts get-caller-identity
docs/design/env-doors.md 1
GB-044 goatbuilder a release refuses before the tag because a secret is not set — every rung this run will deploy to templates the component's secrets from your environment, and at least one is not set. Nothing was tagged, pinned or deployed. the run was not wrapped in op run with a .refs file naming those variables
goat-credentials refs/goat.refs has no line for a variable (ops/ci/check-secrets-coverage.py names it)
put the reference lines for the NOT SET names in .envrc.local.d/<file>.refs, then OP_ACCOUNT=<account> op run --env-file=.envrc.local.d/<file>.refs -- <the same command> docs/operations/goatbuilder-first-run.md 1
GB-045 goatbuilder the published declarations are out of order or altered — a declaration is applied only from an intact, ordered log (contract C2). A row whose signature does not match its content, or a serial not greater than one already accepted, is refused before anything is published, so a replayed or edited declaration never reaches a machine. A new serial at an earlier digest is accepted; that is a revert. a row of declared/publications.jsonl was edited by hand after it was published
an older copy of the log was restored over a newer one, or a row was replayed
restore declared/publications.jsonl from the source of record
to go back to an earlier ref, declare it anew: goatbuilder --mock <fixture> rollback <component> 1 --env <env> --ack "<your name>"
docs/design/goatbuilder-conformance.md 1
GB-046 goatbuilder the local declaration is not the source of record's — a deploy applies only what the source of record declares (contract C3). A local copy behind it is fast-forwarded first, or refused under --frozen; a local copy ahead of it or diverged from it is always refused, because a declaration that is not in the source of record is never applied. the checkout or the fixture's declared/ was not fast-forwarded and --frozen was passed
a declaration was edited locally instead of in the source of record
fast-forward, then re-run: cp source/pins.yaml declared/pins.yaml (on a repository: git pull --ff-only)
make the change in the source of record and re-run
docs/design/goatbuilder-conformance.md 1
GB-047 goatbuilder a declaration must name who made it — every declaration, and every record it converges, carries the person who made it (contract C4). A deploy, rollback or standup run without --ack is refused before anything is published. --ack was not passed, or was empty re-run with --ack "<your name>" docs/design/goatbuilder.md 1
GB-048 goatbuilder the goatnet is not proven built — the acceptance assay ran and at least one applicable row failed, or could not answer, or no row passed. Only PASS counts toward done; an UNKNOWN row is a row nobody proved (ADR 1115). a FAIL row is the goatnet failing that check; its caught_would_have text says what it means
an UNKNOWN row could not be probed from here, most often a goat CLI row run where no goat context with a fabric identity exists (E_PROFILE_MISSING)
a SKIPPED negative row whose paired positive did not pass
goatbuilder assay <site> --only <row-id> (re-run one row after a fix)
goatbuilder assay <site> --var goat_via=<user@host> (run the goat CLI rows where a goat context is)
goatbuilder assay <site> --plan (which rows apply to this site)
docs/design/goatnet-acceptance-suite.md 3
GB-049 goatbuilder the acceptance assay cannot run here — assay derives the site's values from its group_vars and runs ops/validation/run-acceptance.py with python3; one of those could not start, so nothing was probed. the site name has no ops/ansible/group_vars/_hosts/vars.yml, or --values names no file
python3 or PyYAML is not installed
the runner stopped with an error rather than a verdict
ls ops/ansible/group_vars/ (the site names this repository declares)
pip3 install pyyaml
python3 ops/validation/run-acceptance.py --self-test
docs/design/goatnet-acceptance-suite.md 1
GB-050 goatbuilder nothing is declared on that goatnet — diff or adopt found no fleet-pins component whose member groups include the goatnet. the goatnet key is right but no component is pinned there yet
every component is absent on it
goatbuilder doctor --env <goatnet> --all (lists what the goatnet runs) docs/design/goatbuilder.md 2
GB-051 goatbuilder the plan request is incomplete or not planned yet — plan writes deploy and rollback plans; a deploy plan needs --env, and a rollback plan reads a host ledger, so it has no fixture form. plan deploy without --env
a verb other than deploy or rollback
plan rollback with --mock
goatbuilder plan deploy <component> --env <goatnet> --out <file>
goatbuilder plan rollback <component> --n 1 --out <file>
docs/design/goatbuilder-conformance.md 2
GB-052 goatbuilder that file is not a goatbuilder plan — apply runs only a file plan wrote, which carries schema goatbuilder.plan/v1. the wrong file was passed to apply
the plan was edited by hand and its schema line lost
goatbuilder plan <verb> <component> … --out <file> (write it again) docs/design/goatbuilder-conformance.md 2
GB-053 goatbuilder the plan no longer matches the tree — apply recomputed the plan's decisions and at least one moved since it was written; it names each, and runs nothing. main's declaration or the pins file changed since the plan
a newer deploy or rollback moved the rollback target
the plan is old
goatbuilder plan <same arguments> --out <file>, review it, then goatbuilder apply <file> docs/design/goatbuilder-conformance.md 1
GB-054 goatbuilder adopt could not create its worktree — adopt --yes writes its declarations into a fresh worktree of origin/main beside the checkout; git refused to create it. origin/main is not fetched
a branch or directory of that name already exists
git fetch origin main
git worktree list (remove a stale adopt worktree)
docs/design/goatbuilder.md 2
GB-055 goatbuilder a rollback with no --site would flip every goatnet — the component is deployed on more than one goatnet and no --site was given; with no site the roll narrows to nothing, so it would flip every goatnet at once, prod included. Nothing was touched; the error lists the goatnets to choose from. rollback (or plan rollback) was run on a component that spans several goatnets without --site
the goatnet to roll back was left off the command
goatbuilder rollback <component> <N> --site <goatnet> (one of the goatnets the error names)
ops/scripts/rollback.sh <component> <N> --site <goatnet>
docs/design/universal-deploy-rollback.md 1
GB-060 goatbuilder a standup input is required and not set — A role or play the site's standup will reach asserts this variable, or reads it through \| mandatory, and nothing the site supplies sets it: not group_vars, the inventory, host_vars, the play's vars, a role default, nor an extra var the plan passes. The scan lists every such input at once, before any host is touched. the site's config surface was made from a reference goatnet and this key was not carried over
the role gained a required input after the site was configured
set the variable in ops/ansible/group_vars/<group>/ (or the inventory)
goatbuilder doctor --site <site> (re-scan; every gap is listed)
docs/design/goatbuilder-speed.md 1
GB-061 goatbuilder a secret the standup templates is not in this environment — A value the site supplies is lookup('env', NAME), a play or role the standup reaches uses it, and the declared secrets posture cannot supply NAME (ops/secrets/posture.py, ADR 1204). the run is not under the site's secrets posture
the posture's references file has no line for this name
run under your secrets posture: ops/scripts/with-secrets goatbuilder standup <env> --site <site>
add the reference for NAME to your .refs file
docs/design/pluggable-dependencies.md 1
GB-062 goatbuilder a standup input may be required — The input is required under a when: condition the scan cannot evaluate from the site's own values (it depends on a registered result, a fact, or a template). This is a warning, never a refusal: the scan does not claim more than it knows. the condition reads a value only known when the play runs read the role at the file named and decide whether the condition will hold for this site
set the input anyway if it will
docs/design/goatbuilder-speed.md 0
GB-063 goatbuilder a standup step's host limit selects no host — The plan runs this play with --limit , and the pattern matches no host of the target group in the inventory, so the play runs on nothing and reports success. the limit assumes the mgmt--01 host naming and this site names its hosts differently
the step installs a service this joining site does not run yet, which its end state names as a gap with the gate row that closes it
goatbuilder endstate <site> names each gap with its gate row (I1, I4 for an admin surface); a gap is work still to build, not a reason to skip the step
if it should, fix the limit in goat-standup.sh and goatbuilder standup together
docs/design/goatbuilder-speed.md 0
GB-064 goatbuilder the checkout is behind the trunk in what a deploy reads — A real run reads ops/ansible (roles, plays, group_vars, inventory, the rendered pins) and ops/pins from the checkout it runs in. The trunk carries commits to those paths that this checkout lacks, so a file rendered from it would remove on the host what those commits added, and the play would report success (contract C3, F-410). Refused before any host is touched, in both avenues (ops/scripts/fresh-source.sh). the checkout was not brought onto the trunk after a merge
a feature branch was cut before a change to a role, group_vars or the pins landed
on main: git pull --ff-only <trunk remote> main
on a branch: git fetch <trunk remote> main && git rebase <trunk remote>/main
docs/design/goatbuilder-conformance.md 1
GB-065 goatbuilder the trunk could not be fetched, so the checkout cannot be shown current — A real run proves its checkout is current with the trunk before any host is touched (contract C3). The trunk remote did not answer a fetch, so that proof is missing and the run is refused. An islanded site, cut off by design, runs with --offline: it is judged against the cached trunk and the ledger row it writes says source: offline. no route to the git remote (the laptop is offline, the forge is down)
the remote's credentials are not loaded (an ssh key not in the agent)
restore the connection to the trunk remote and re-run
on an islanded site: re-run with --offline
docs/design/goatbuilder-conformance.md 1
GB-066 goatbuilder the site's declared end state cannot be derived or read — endstate derives what a site should run from its own declarations (group_vars, the fleet pin table, the inventory) and the service catalog, then reads the site's hosts once over Ansible. One of those could not start, so nothing was graded. the site name has no ops/ansible/group_vars/_hosts/ directory
python3 or PyYAML is not installed on this machine
the catalog file ops/goatbuilder/endstate/catalog.yaml does not parse
ls ops/ansible/group_vars/ (the site names this repository declares)
python3 ops/goatbuilder/endstate/endstate.py model <site> (the derived end state, without reading any host)
python3 ops/goatbuilder/endstate/endstate.py --self-test
docs/design/goatnet-end-state.md 1
GB-067 goatbuilder a site's service presence is undeclared or contradicted — Every service the catalog lists must be declared present or absent for every governed site, by exactly one answer: a flag in the site's group_vars, the fleet pin table (an absent deviation), or the founder rule. The baseline ops/goatbuilder/endstate/baseline.txt lists the gaps that exist today and may only shrink: a new gap, or a baseline row that is no longer true, fails. a new service in the catalog with no flag declared for a site
the fleet pin table renders a component the site's flag says it does not run (or the reverse)
a gap was closed and its baseline row was left behind
set the service's flag (its catalogvar:) in ops/ansible/group_vars/<site>_hosts/vars.yml
addabsent: truefor the member in ops/pins/fleet-pins.yaml, then regenerate and re-sign the pins
delete the stale row from ops/goatbuilder/endstate/baseline.txt
docs/design/goatnet-end-state.md 1
GB-068 goatbuilder the goatnet is not in its declared end state — endstate graded every service the site declares: present or absent, deployed (its ledger names the declared ref), active (every container running), healthy (its healthcheck or isitup probe), functional (the acceptance rows that exercise it), and its zero-trust edges (its served name resolves and its certificate verifies, its OIDC client is registered in Hydra, its SPIRE entry exists). At least one grade failed or could not be read, or something runs that the end state does not declare (an orphan). a declared service is not running, or its ledger is pending, rolled back or at another ref
an edge is broken, such as an OIDC client never registered or a name that does not resolve
an orphan, such as a container left behind by a retired service, or a Hydra client or SPIRE entry no service declares
a grade could not be read, most often because a host was unreachable or the assay did not run
goatbuilder endstate <site> (each failing grade with its reason)
goatbuilder deploy --component <c> --env <goatnet> (restore a declared service)
goatbuilder endstate <site> --json (the grades as data)
docs/design/goatnet-end-state.md 3
GB-069 goatbuilder the rollout preflight cannot run — preflight reads the component's declared rollout facts from ops/pins/fleet-pins.yaml (owns_state, rollback, restore_from, coupled_with) and the open graduation sets. The component is not in the pins, or the pins could not be read, so there is nothing to grade. the component name is misspelled or not a fleet-pins component key
ops/pins/fleet-pins.yaml is missing, unreadable, or did not parse
on a fixture (--mock), declared/artifacts.yaml did not parse
grep -n '^ [a-z].*:' ops/pins/fleet-pins.yaml (the component keys this repository declares)
goatbuilder preflight <component> --env <goatnet> (name the component as its fleet-pins key)
docs/design/goatbuilder-rollout-preflight.md 1
GB-070 goatbuilder survey could read no goatnet at all — survey reads every goatnet it was asked for in parallel and reports what it could not read as unknown. When not one goatnet answered, there is nothing to compare, and the run stops instead of printing an empty survey that would look consistent. this laptop is on no goatnet's mesh profile and no door opened (no AWS credentials for the SSM door)
every goatnet named with --env is misspelled or not in ops/pins/fleet-pins.yaml
ansible is not installed, or ops/scripts/with-aws cannot run
goatbuilder doctor --env <goatnet> --all names the transport each goatnet would use and what is missing
name goatnets by their fleet-pins key: --env <goatnet> (repeatable)
docs/design/goatbuilder.md 1
GB-071 goatbuilder a site's leaf certificate is not on this controller — A play that installs a site's leaf (the zenoh router, the LAN mgmt identity, the dashboard pair, the names collector) found the cert or key nowhere it looks: the site manifest's out_dir (/-leaves), then a copy staged in this checkout or the main checkout. It refuses before touching a host. the leaf was never minted for this site, or was minted on another operator's machine
the manifest's out_dir points somewhere this machine does not have
GOAT_LEAF_DIR is set to a directory that does not hold it
mint it into the out_dir: ops/scripts/mint-site-certs.sh <site> --only <leaf> (idempotent; an existing leaf is skipped)
copy an existing pair into <deploy.state_dir>/<site>-leaves/ (key mode 0600), never into the repository
docs/operations/goatnet-standup.md 1
GB-072 goatbuilder goatbuilder cannot name the download surface to update from — update (and the behind-version notice) reads the release a goatnet serves at get../goatbuilder/. It looks at GOATBUILDER_UPDATE_URL, then --site (named from the site's group_vars), then the surface the last successful update used. None of them named a surface that serves goatbuilder for this machine. the first update on this machine was run without --site, and GOATBUILDER_UPDATE_URL is not set
the site named with --site has no ops/ansible/group_vars/_hosts/ in the checkout goatbuilder was run in
the site does not serve goatbuilder (goat_get_goatbuilder_enabled is not true in its group_vars)
no goatbuilder release is built for this operating system and architecture
goatbuilder update --site <site> (a goatnet that serves goatbuilder)
run it from a goat checkout, or pass -C <checkout>
GOATBUILDER_UPDATE_URL=https://get.<site>.<zone>/goatbuilder goatbuilder update
docs/design/goatbuilder-distribution.md 1
GB-073 goatbuilder the goatbuilder download surface could not be read — update reads version.json, the platform binary and its .sha256 from get../goatbuilder/. One of them did not arrive: the connection failed, the server answered with an error, the answer was not what the surface publishes, or the binary stopped arriving for 30 seconds. Nothing was installed; the binary in place is the one that ran. this machine is not on the goatnet's mesh (get. is mesh-only)
the surface is mid-deploy, or does not publish this platform's binary
the link stalled; a slow link is fine, a silent one for 30 seconds is not
curl -fsS https://get.<site>.<zone>/goatbuilder/version.json (is the surface reachable from here?)
retry the update; it is idempotent
GOATBUILDER_UPDATE_STALL_SECS=120 goatbuilder update (a slower link)
docs/design/goatbuilder-distribution.md 1
GB-074 goatbuilder the downloaded goatbuilder is not vouched for by its release — Before touching anything, update checks that the .sha256 the surface serves agrees with the release's SHA256SUMS, that the SHA256SUMS signature verifies against the release workflow's identity (when cosign is on PATH), and that the downloaded binary hashes to that digest. One of these failed, so it refused to install; the installed binary is unchanged. the surface is mid-deploy and the binary, its checksum and SHA256SUMS are from different releases
the download was altered or truncated on the way
the signature over SHA256SUMS is not from the release workflow of the repository goatbuilder.yaml names
retry in a few minutes; the surface's files are written together
goatbuilder update --check (what the surface serves now)
cosign verify-blob --bundle SHA256SUMS.cosign-bundle --certificate-identity-regexp '^https://github.com/<owner>/<repo>/' --certificate-oidc-issuer https://token.actions.githubusercontent.com SHA256SUMS (the check by hand)
docs/design/goatbuilder-distribution.md 1
GB-075 goatbuilder goatbuilder could not replace its own binary — The download was verified, but writing it next to the running binary or renaming it into place failed. The rename is atomic, so the binary in place is either the old one, whole, or the new one, whole. the install directory is not writable by this user (a binary under /usr/local/bin installed with sudo)
the disk is full
run the update as the user that installed goatbuilder (sudo goatbuilder update --site <site> for a system-wide install)
install by hand from get.<site>.<zone>/goatbuilder/, checking the .sha256 first
docs/design/goatbuilder-distribution.md 1
GB-076 goatbuilder the rollout is not cleanly reversible or not ready — preflight graded the three rollout facts (State, Rollback, Scope) and the readiness half, and at least one row is not a clean, declared, validated pass. An undeclared fact fails closed (stateful, rollback none) and reads not-clean on purpose, so the gap is visible rather than guessed green; an UNKNOWN is not a pass either. The facts are still printed — a stateful, restore-required rollout is a valid state to deploy knowingly, with --ack. a fact is undeclared, so it failed closed (declare owns_state / rollback / restore_from / coupled_with in ops/pins/fleet-pins.yaml)
a declared rollback contradicts the artifact type (pin-revert over a data/cert/forward-only component, or one flagged reversible false)
a readiness check failed — the ref is not pullable, pins --check found drift, the secrets gate is unsatisfied, the checkout is stale, or the end state is not declared
a validation could not answer (UNKNOWN) — no registry probe, or no reachable ledger for the rollback --dry-run
goatbuilder preflight <component> --env <goatnet> (each not-ok row with its reason)
declare the component's rollout facts in ops/pins/fleet-pins.yaml (owns_state, rollback, restore_from, coupled_with)
goatbuilder deploy --component <c> --env <goatnet> --ack "<you>" (deploy knowingly; the accepted risk is recorded)
docs/design/goatbuilder-rollout-preflight.md 3
GB-100 goatbuilder the federation verbs cannot find this party's context directory — goatbuilder federate init (and verify or apply without --as) reads this party's declaration, federation.toml, and init also reads the mTLS certificate and CA roots beside it. goatbuilder has no profile of its own, so it takes the directory from --context-dir, else $GOAT_CONTEXT_DIR, else the goat profile ($GOAT_PROFILE_DIR, else ~/.goat), and none of those was set. none of --context-dir, GOAT_CONTEXT_DIR, GOAT_PROFILE_DIR or HOME is set, as in a bare container or a service unit goatbuilder federate init --context-dir <dir> (the directory holding federation.toml, mtls.cert.pem and ca-roots.pem)
export GOAT_PROFILE_DIR=<goat profile directory> (the verbs then read its active context, as goat does)
goatbuilder federate verify <agreement> --as <sovereign_id> (verify and apply need no context when --as is given)
docs/design/goatbuilder.md 2
GB-110 goatbuilder the inventory does not name a federation host phase 3 can reach — federate phase 3 measures the edge and the core from this machine, reaching each host the way the peering plays do: the host in the edge group and the host in the core group of inventory/peering-edges.ini, with its address, user, port, key and SSH options. The inventory could not be read, the group has no host or several, or the host uses a connection phase 3 does not use. ansible-inventory is not installed here, or the inventory does not parse
the edge group (_edge) or the core group (_core) is empty or missing in inventory/peering-edges.ini
the group has several hosts and none was named
the host uses an ansible_connection other than ssh or local
cd ops/ansible && ansible-inventory -i inventory/peering-edges.ini --playbook-dir . --graph (what the plays see)
add the host to the group, or name it: goatbuilder federate init --edge-host <host> --core-host <host>
docs/design/federation-router-discovery.md 1
GB-111 goatbuilder a federation host did not answer over SSH — Before measuring anything, phase 3 runs true on the edge and on the core over SSH, with the inventory's connection settings and the plays' privilege escalation. That command failed, so nothing was measured. The message carries the ssh command it ran. this machine is not on a network that reaches the host (mesh profile, door or jump host)
the host refused the key (see X-SSH-001)
the host key changed (see X-SSH-003)
run the ssh command the message prints, without its last argument, and read ssh's own error
goatbuilder doctor --env <env> (the transport row)
docs/design/federation-router-discovery.md 1
GB-112 goatbuilder sudo on a federation host wants a password — The host answered over SSH, but sudo -n true failed: phase 3 reads the federation interface and the container runtime as root, the way the plays become root, and it never types a password. the host's operator user is not NOPASSWD in sudoers (the baseline play sets it)
ansible_become_user names a user this account cannot become without a password
run ops/ansible/mgmt-host-baseline.yml against the host once
set ansible_become: false for the host if its user can already read wg and the container runtime
docs/design/federation-router-discovery.md 1
GB-113 goatbuilder no core router to measure on the core host — Phase 3 measures the core router's image and version on the core host. It takes the router the operator names with --router-container, else the name the deployment's variables give (zenoh_stack, or site_id with the goat-zenoh role's default, as -router), else the one running core router discovery finds. That router was not running, could not be read, or several candidates were found. the declared router is not running on the core host, or the variables give a different name than the one deployed
nothing declares the router and no running container carries org.goat.component=zenoh-core or a standard name
several core routers run on the host
goatbuilder federate init --router-container <name> (name it by hand)
FED_ROUTER_CONTAINER=<name> goatbuilder federate <partner> (the one command passes it to phase 3 as --router-container)
set zenoh_stack or site_id for the core host in the deployment's group_vars or host_vars
ssh <user>@<core> sudo docker ps --format '{{.Names}} {{.Label "org.goat.component"}}' (what is running)
docs/design/federation-router-discovery.md 1
GB-114 goatbuilder an edge half of phase 3 cannot be joined — With two drivers, the edge driver measures phase 3's edge half with goatbuilder federate init --over-ssh <partner> --side edge --half-out <file> (or, inside the one command, goatbuilder federate <partner> --side edge --from 3 --to 3) and prints its sha256 and a confirm code. The controller joins it with --side core --edge-half <file> --edge-half-sha256 <code> (or imports it with goatbuilder federate import <partner> 3 <file> --sha256 <code>, which runs the same file checks, and --from 3 joins it) before it builds and signs the package. The half carries the edge's WireGuard public key, which the package is signed with, so it is joined only when it is the file the edge driver wrote. The file given could not be read; its sha256 is not the one confirmed; it is not an edge half; it was measured for another peering, on another host than this checkout's inventory names as the edge, or for another federation interface than the package asks for; or it was measured more than 24 hours ago, or later than now. the file was changed or not copied whole on its way from the edge driver, or another file was given
the confirm code was mistyped, or read from another run of the edge half
the edge half was measured for another partner or on another edge host
the declaration or the init flags differ between the two machines, so the edge was measured for another interface (or none)
the edge half is older than a day, or one of the two machines' clocks is wrong
goatbuilder federate init --over-ssh <partner> --side edge --half-out <file> (on the edge driver, with the same declaration and init flags; it prints the confirm code)
sha256sum <file> (on both machines: the two must agree)
docs/design/federation-router-discovery.md 1
GB-120 goatbuilder a peering names a partner with a reserved word — A word of goatbuilder federate (init lint compose diff verify apply help status plan deploy test allocate backout rotate) can never name a partner: goatbuilder federate <word> runs that step, so the peering could never be run by name. a peerings entry's with: is one of the reserved words name the partner by its sovereign id in goatbuilder.yaml peerings: with: docs/design/goatbuilder.md 2
GB-121 goatbuilder two peerings would share a value — A per-peering value (an interface, a UDP or TCP port, a link subnet, a forwarder port, a certificate directory, an inventory name) is pinned on one peering and already held by another, by a pin or by the allocation record, or a partner is declared twice on one environment. Two peerings on one edge or core host cannot share one. two peerings pin the same value under values:
a pin takes a value the allocation record holds for another peering
the same partner is declared twice on one environment
pin a different value on one of the two peerings, or remove the pin and run goatbuilder federate allocate to have one allocated
goatbuilder validate (every collision, each naming both peerings)
docs/design/goatbuilder.md 2
GB-122 goatbuilder a pinned port is one the edge host already holds — A pinned WireGuard or carrier listen port is a UDP port the environment's edge hosts already hold (environments[].peering_edge_reserved_udp; absent, NetBird's WireGuard on 51820). A second listener there fails or steals its traffic. values.wg_listen_port or values.carrier_listen_port is pinned to a reserved port
a peering running on the role default 51820, on edges with no mesh client, is being adopted
remove the pin and run goatbuilder federate allocate (it allocates odd ports from 51823 up and never takes a reserved port)
when the environment's edges run no mesh client: set peering_edge_reserved_udp: [] on that environment in goatbuilder.yaml
docs/design/goatbuilder.md 2
GB-123 goatbuilder a peering's pin is not usable — A pin under a peering's values: is malformed (an interface name over 15 characters, a subnet that is not a /31 or /30 on its boundary, a relative certificate directory, a name with spaces), or pins a forwarder value on a peering no forwarder relays. a typing mistake in a pin
public_port or a carrier_* value pinned without forwarder:
correct the pin, or remove it to have the value allocated
declare forwarder: endpoint: on the peering, or remove the forwarder pins
docs/design/goatbuilder.md 2
GB-124 goatbuilder the peering allocation record cannot be used — ops/goatbuilder/peering-values.yaml, the record of the per-peering values goatbuilder federate allocate chose, is unreadable, has another schema, holds a malformed value, gives two peerings one value, or records a port the edge host already holds. Allocation reads it so a re-run never moves a value; it will not guess around a broken one. a hand edit or a merge conflict in the record
a record from a newer goatbuilder
git log -p ops/goatbuilder/peering-values.yaml (restore it from its history)
remove the broken entry and run goatbuilder federate allocate to allocate it afresh (that peering's values may change)
docs/design/goatbuilder.md 2
GB-125 goatbuilder a peering file exists that goatbuilder did not generate — goatbuilder federate allocate writes the peering inventory and each peering's peering-values.yml. It never overwrites a file it did not generate (its first line says who did), and never renders a variable a hand-written file already sets for the same hosts (the group, its parent peering group, or the host_vars of its inventory names), because Ansible's file order would then pick the value. Nothing was written. ops/ansible/inventory/peering-edges.ini or a peering-values.yml was written by hand before the values were allocated
a hand-written group_vars or host_vars file sets a variable allocate renders (a listen port, a certificate path)
move its values into goatbuilder.yaml (pins under the peering's values:, addresses under hosts:), remove the file, and run goatbuilder federate allocate again docs/design/goatbuilder.md 2
GB-126 goatbuilder a peering's inventory group is defined twice — Not every peering declares hosts:, so ops/ansible/inventory/peering-edges.ini stays hand-written and each peering that declares hosts: gets its own rendered inventory file (inventory/peering-.ini), which the driver reads beside it. That peering's group is also defined in the hand-written inventory, so the plays would see it twice. Nothing was written. hosts: added to a peering whose group is still in the hand-written peering-edges.ini remove that peering's groups from ops/ansible/inventory/peering-edges.ini
or remove hosts: from the peering and keep its groups in the hand-written inventory
docs/design/goatbuilder.md 2
GB-127 goatbuilder no free per-peering value is left — Every candidate in a per-peering value's sequence is held by another peering of the same environment (128 interfaces, ports or /31 subnets), or by a pin. many peerings on one environment, or pins scattered over the sequence pin the value under the peering's values: docs/design/goatbuilder.md 2
GB-128 goatbuilder a federate run disagrees with the peering's allocated values — The allocation record holds this peering, and the run's working directory, edge group or core group is not the allocated one. Phase 8 would mint certificates where the edge does not read them, or the plays would run on another peering's hosts. Nothing ran. --work or FED_WORK names a directory whose pki is not the allocated cert_dir
--edge-group names another group than the allocated edge_group
a partner declared on a second environment, or a core_group pin: the driver runs the republisher on _core and does not read the allocated core group yet
goatbuilder federate allocate --dry-run (the allocated values)
pass the --work and --edge-group the refusal names, or pin cert_dir and edge_group under the peering's values: to what you pass
docs/design/goatbuilder.md 2
GB-129 goatbuilder a peering's offer would widen what federation.toml declares — federation.toml owns the release sets this party offers ([data] release). A peering's offer: may only narrow that list for one partner. It names a set federation.toml does not offer, is empty or malformed, or federation.toml cannot be read. Or the peering's federation.toml is hand-written (federation_toml: hand), whose [data] release phase 3 offers whole, and offer: is narrower than it; a peering whose federation.toml goatbuilder federate allocate generates gets the narrowed list. Nothing ran. offer: names a set that is not in federation.toml's [data] release
offer: is narrower than the [data] release of a hand-written federation.toml (federation_toml: hand), which phase 3 offers whole
GOAT_PROFILE_DIR or GOAT_CONTEXT points at another goat context than the one phase 3 runs in
add the set to [data] release in federation.toml, the one owner of the offered sets, or remove it from offer:
declare federation_toml: rendered on the peering so goatbuilder federate allocate generates its federation.toml, or narrow [data] release in the hand-written file to what this partner receives, or remove offer:
goatbuilder validate (the offer's shape)
docs/design/goatbuilder.md 2
GB-210 goatbuilder the host firewall baseline is missing a value it needs — The host firewall baseline (roles/goat-host-firewall, rendered by its goat-host-firewall-render.py) could not build a ruleset, because a value it derives the rules from is missing or malformed. Nothing was applied. goat_host_firewall_management_sources is empty, so SSH would be admitted from nowhere
an edge run reached the role without its peering values (the play was not peering-wg.yml or peering-edge.yml)
a port, address or interface name in the input is not one (a partner endpoint given as a name, an interface name over 15 characters)
a management source is /0, the mesh interface is the host's uplink, or a br-* bridge that is not docker's (a VXLAN segment bridge) exists while container-bridges is kept
twice goat_host_firewall_confirm_timeout plus 15 s is not below goat_host_firewall_rollback_seconds
the switch is false but the host still carries the baseline (/etc/goat-host-firewall/profile exists)
set goat_host_firewall_management_sources in the host's group_vars to the address or CIDR the deployment manages it from
python3 ops/ansible/roles/goat-host-firewall/files/goat-host-firewall-render.py --input <input.json> --list-rules (what the baseline derives)
docs/design/peering-host-firewall.md 2
GB-211 goatbuilder the host firewall override names a rule the baseline does not have, or is malformed — The deployment's goat_host_firewall_override names a rule that is not in the baseline, names a fixed rule (loopback, established, invalid), uses a key other than drop, sources, ports, interface and reason, asks a fixed-expression rule for more than drop, or gives a reason holding a line break or another control character (the reason is written into the ruleset file). The whole override was refused and nothing was applied. a misspelled rule name, or a peering rule named for an interface this host does not carry
an attempt to add a new allow rule; only named baseline rules can be changed, and other-tcp / other-udp are the named rules for a service the host also runs
python3 ops/ansible/roles/goat-host-firewall/files/goat-host-firewall-render.py --input <input.json> --list-rules (the names an override may use) docs/design/peering-host-firewall.md 3
GB-212 goatbuilder the host firewall is loosened without a recorded reason — An override entry widens a baseline rule (more sources, other ports, another interface), or a role input is wider than the profile (goat_host_firewall_mesh_udp_ports not empty, a mesh DNS port other than 5353, a management source broader than goat_host_firewall_management_prefix_floor, or that floor set broader than /24 or /64), and no reason is recorded for that rule. Tightening is free; loosening a named rule needs a reason recorded in the deployment's configuration. Nothing was applied. the entry has no reason: key, or a blank one
an entry meant to narrow widens instead (an empty sources list means any source)
add reason: "<why this host needs it>" to the entry in goat_host_firewall_override
narrow the rule instead, which needs no reason
docs/design/peering-host-firewall.md 3
GB-213 goatbuilder the host firewall would cut the session managing the host — The rendered ruleset admits no SSH from the source of the session running the play (read from SSH_CONNECTION on the host), so loading it would lock the deployment out. Nothing was applied. goat_host_firewall_management_sources does not hold the address this play connects from
the override drops ssh-management, narrows its sources past this address, or binds it to another interface
SSH listens on another port than goat_host_firewall_ssh_port
the session's source could not be read, so the check could not run
add the controller's address to goat_host_firewall_management_sources
run the play from a declared management source
docs/design/peering-host-firewall.md 4
GB-214 goatbuilder the host firewall ruleset was not loaded or not confirmed — Loading the rendered ruleset failed, or a fresh connection to the host did not succeed after it was loaded. A ruleset that was loaded is undone by the rollback timer the role armed before loading it. the kernel refused the ruleset (nft -c failed), for example an nftables too old for a rule
the new ruleset blocks the connection the controller makes, through a path the lockout check could not see (a jump host, a NAT between controller and host)
systemd-run is not available on the host, so no rollback could be armed and nothing was loaded
wait goat_host_firewall_rollback_seconds, then: nft list table inet goat_host_firewall (the previous table is back)
read /etc/goat-host-firewall/ruleset.nft.new on the host and the play output
docs/design/peering-host-firewall.md 1
GB-215 goatbuilder the host firewall would close a port this host listens on — Before loading, the host firewall baseline reads the sockets the host listens on (ss) and found one, on a non-loopback address, that no rule admits and that the deployment has not named as closed on purpose. Loading would cut that service without anyone deciding it. Nothing was applied. the host runs another service beside the peering (a management server, a portal, an exporter)
a port the deployment means to close is not listed in goat_host_firewall_closed_listeners
admit it: goat_host_firewall_override: {other-tcp: {ports: [<port>], reason: "<why>"}}
close it on purpose: goat_host_firewall_closed_listeners: ["tcp/<port>"] (or the process name)
docs/design/peering-host-firewall.md 5
GB-180 goatbuilder another run of this peering is in progress — A real run of goatbuilder federate holds /.lock from start to end, so two runs of one peering never interleave phases or overwrite each other's run journal. This run found the lock held and changed nothing. another terminal or a scheduled job is running goatbuilder federate for the same peering
two peerings were given the same working folder (--work or FED_WORK)
wait for the other run to finish; the lock file names its process id
give each peering its own working folder: --work /var/lib/goat-federation/<partner>
docs/design/goatbuilder.md 1
GB-181 goatbuilder a resource this peering needs is held by another peering on the same host — Before a peering takes a resource on a shared host (a tunnel interface, a port, a link subnet, a certificate folder), goatbuilder reads every other peering's run journal in the same work root under /.allocation.lock. One of them holds the same resource on the same host, or a journal there could not be read, so nothing was claimed and nothing changed. two peerings were given the same per-peering value (a pinned value in peerings: or a hand-written group_vars file)
a journal in the work root is damaged, so the check cannot tell what it holds
give this peering a different value, or let allocation choose one, then make a new plan
read the other peering's journal named in the message: <root>/<peering>/state.json
docs/design/goatbuilder.md 1
GB-182 goatbuilder the run journal cannot be read — /state.json is the only account of what ran for this peering, so a journal that does not parse, has a later schema than this goatbuilder reads, or names another peering is refused rather than overwritten. A journal of the same schema that a later goatbuilder wrote is read: a word or field this one does not know is shown as written and kept. the file was edited by hand or cut short
a later goatbuilder wrote it with a later schema (the message names the schema and the goatbuilder that wrote it)
two peerings share one working folder
use the goatbuilder that wrote it, or move the file aside and run again (the run then starts with no journal, and --resume skips nothing)
give each peering its own working folder: --work /var/lib/goat-federation/<partner>
docs/design/goatbuilder.md 1
GB-183 goatbuilder a federate phase failed — The word the run journal records for a phase that exited non-zero, with its exit code. The run stopped there; no later phase ran. the phase's own error, printed above it (a goat federate exit 20 to 24, a play's failed task, a refused gate) fix the cause, then run the same command again; with --resume, the phases found unchanged are not run again docs/operations/sovereign-federation-runbook.md 1
GB-184 goatbuilder the working folder or its lock cannot be made, taken or written — A real run of goatbuilder federate keeps its lock and its run journal in the peering's working folder, and the allocation lock in the folder above it. One of them could not be created, locked or written, so the run did not start (or, mid-run, its journal is behind; the run says so in a failed journal row). the working folder is not writable by this user (the default /var/lib/goat-federation needs root or a folder made for you)
the disk is full or the folder is on a read-only mount
sudo install -d -o $USER /var/lib/goat-federation (or pass --work <a folder you own>)
check free space where the working folder is
docs/design/goatbuilder.md 1
GB-130 goatbuilder a federation input is missing — federate phase 1 (preflight) found inputs the later phases need that nothing declared and nothing could derive. Every missing input is listed with where to declare it. Nothing was measured, signed or deployed. the deployment does not declare it (the edge group's group_vars, the peering's entry in goatbuilder.yaml peerings:, the inventory)
the script avenue (peering-standup.sh) was run without exporting it; goatbuilder derives what it can, the script does not
declare each listed value where the line names it, then: goatbuilder federate <partner> --from 1 --to 1
export FED_<NAME>=... (one command only; an exported value always overrides what goatbuilder derives)
docs/design/federation-preflight.md 1
GB-131 goatbuilder a port this peering needs is already held on the edge — the tunnel listen port (UDP) or the edge router listen port (TCP) is held on the edge host by something other than this peering's own interface or container. NetBird holds UDP 51820 on a mesh host, which is also the WireGuard default. the tunnel listen port is the WireGuard default 51820 and NetBird's interface holds it
another peering's edge router or interface already uses the port on a shared edge host
pin another port in the peering declaration: goatbuilder.yaml peerings: values: wg_listen_port / edge_listen_port
or set peering_wg_listen_port / goat_zenoh_edge_listen_port in group_vars/<edge group>/, then re-run phase 1
docs/design/federation-preflight.md 1
GB-132 goatbuilder the republisher metrics port is already held on the core — the outbound republisher's loopback metrics port is held on the core host by something other than this peering's republisher. One port per peering on a shared core host. another peering's republisher uses the same metrics port on the core host
another service listens on that loopback port
pin republisher_metrics_port in the peering declaration (goatbuilder.yaml peerings: values:)
or set goat_fed_republisher_metrics_port in group_vars/<peering>_core/
docs/design/federation-preflight.md 1
GB-133 goatbuilder the edge or core host could not be probed for its ports — the read-only probe (ops/ansible/peering-preflight.yml) returned no answer for a host, or ss could not list its sockets, so the ports were not checked. Not a pass. the host is unreachable with the inventory the peering plays use (mesh down, wrong ansible_host, no SSH key)
iproute2 (ss) is not installed on the host
cd ops/ansible && ansible -i inventory/peering-edges.ini <group> -m ping
install iproute2 on the host, then re-run phase 1
docs/design/federation-preflight.md 1
GB-134 goatbuilder the committed slot list differs from the slot registry — ops/scripts/sync-slot-anchors.py --check found live slots with no declared store or declared stores for slots that are not live. A core router restarted on this list drops or misses receive stores. Slots minted between the fix and the restart drift it again, so the restart step runs this check once more. slots were minted or revoked since the list was last regenerated and committed
the list was regenerated, and a slot was minted before the core router restarted
python3 ops/scripts/sync-slot-anchors.py --env <goatnet> --db <registry.db> (writes the list; commit it)
python3 ops/federation/standup/preflight.py <partner> --work <dir> --only slot-anchors (re-check just before the restart)
docs/design/federation-preflight.md 1
GB-135 goatbuilder the slot registry could not be read — sync-slot-anchors.py exited 2: the registry was unreadable, or it reported zero live slots. That is not a verdict on drift, and the check is not passed. FED_SLOT_DB names a file that is not the portal registry, or it cannot be opened read-only
the management host named by FED_SLOT_PG_SSH is unreachable, or its portal database container differs
check FED_SLOT_DB, or FED_SLOT_REGISTRY_STORE=postgres with FED_SLOT_PG_SSH (portal_store in group_vars/<goatnet>_hosts says which) docs/design/federation-preflight.md 1
GB-136 goatbuilder the core has no receive store matching what the edge grants — the core router's storage_manager has no store whose key expression equals an ingress scope the edge admits (from//** by the spec's rule, or the rendered goat_zenoh_edge_ingress_key_exprs). A store wider than the grant, such as from/**, declares a subscriber the edge refuses, so data would cross and not be kept. the core keeps only a historical from/** store
the core's per-partner store names a different partner or lane than the edge grants
add a store on the core whose key_expr equals each admitted scope (the goat-zenoh role renders one per partner); keep a historical from/** store beside it if its data is needed docs/design/federation-ingress-storage-preflight.md 1
GB-137 goatbuilder the core router's configuration could not be read — preflight found the core router but could not read its configuration, so the receive stores were not checked. Reported SKIP, never PASS; FED_PREFLIGHT_STRICT=1 refuses on it. the core router reads its configuration from another path than /etc/zenoh/config.json5
the account behind FED_ROUTER_DOCKER_HOST may list containers but not exec in them
export FED_CORE_CONFIG_PATH=<path inside the core container>
inspect the core's storages through its own approved workflow
docs/design/federation-ingress-storage-preflight.md 1
GB-138 goatbuilder no core router container was found — the selection phase 3 uses (goat federate init) found no running core router on the daemon preflight read: FED_ROUTER_DOCKER_HOST, else the local runtime. Discovery accepts the org.goat.component=zenoh-core label, *-zenoh-router names and legacy goat-zenoh. the core router runs on another host and FED_ROUTER_DOCKER_HOST does not point at it
the core router is stopped, or runs under a custom name without the role label
export FED_ROUTER_DOCKER_HOST=ssh://<user>@<core> (goatbuilder derives it from the peering's hosts.core, else the one host of inventory group <partner>_core)
export FED_ROUTER_CONTAINER=<name or ID>
docs/design/federation-router-discovery.md 1
GB-139 goatbuilder several core router containers were found — more than one running container qualifies as the core router, and preflight, like phase 3, will not choose among them. an old and a new core router run side by side
two stacks on one host both carry a *-zenoh-router name
export FED_ROUTER_CONTAINER=<name or ID of the core router> docs/design/federation-router-discovery.md 1
GB-140 goatbuilder our CA certificate is missing or is not a CA — the certificate phase 8 signs the peering leaves with does not exist, is not a PEM certificate, or is not a CA (basicConstraints CA:TRUE). FED_CA_CERT, goat_federation_ca_cert or the derived ops/ansible/enrollment/ca//root-cert.pem names a missing file
a leaf certificate was named instead of the CA
set goat_federation_ca_cert in group_vars/<edge group>/ to our own CA certificate (never the partner's) docs/design/federation-preflight.md 1
GB-141 goatbuilder our CA signing key cannot be resolved — the CA private key phase 8 needs is missing, does not decrypt with the age identity, is not a private key, or does not match the CA certificate. The deployment keeps it at /-ca/ca.key.age; preflight decrypts it in memory to compare and writes no plaintext. the ca.key.age file is not on this machine
the age identity (~/.config/age/keys.txt, or FED_AGE_IDENTITY) is not the one the key was encrypted to
the key belongs to another CA than the certificate
restore <deploy.state_dir>/<site>-ca/ca.key.age, or set goat_federation_ca_key_age / FED_CA_KEY_AGE
export FED_AGE_IDENTITY=<the identity file the key was encrypted to>
docs/design/federation-preflight.md 1
GB-142 goatbuilder our federation signing key cannot be read — FED_SIGNING_KEY names a file that does not exist, or whose content is neither 64 hex characters (the RFC 8032 seed) nor 44 base64. goat_federation_signing_key or FED_SIGNING_KEY names the wrong path
the file holds a PEM or a public key rather than the seed
set goat_federation_signing_key in group_vars/<edge group>/ to the file holding our Ed25519 seed docs/design/federation-preflight.md 1
GB-143 goatbuilder the peering inventory does not reach the edge or core — inventory/peering-edges.ini is missing, has no group for this peering's edge or core, lists no host in it, or does not list the group under [peering_edges:children] / [peering_cores:children]. A play limited to such a group runs against nothing and reports success. the edge or core group was never added to the deployment's inventory
the committed inventory ships the host commented out
add [<partner>_edge] and [<partner>_core] with their hosts, and list each group under its parent's :children docs/design/federation-preflight.md 1
GB-144 goatbuilder the federation declaration or the profile is missing or wrong — the goat context has no federation.toml, it has no valid sovereign_id or it declares a measured capability, or the profile certificate or CA bundle is missing. goat profile init was not run for this context, or ran as another user (sudo -E keeps HOME)
federation.toml was not written, or declares transports, fabric_version, fabric_fips_image or pqc_capable
goat profile init <bundle.cbor>
goat federate init (prints the declaration template when it is absent)
docs/design/federation-preflight.md 1
GB-145 goatbuilder the core certificate would not name every address the edge dials — FED_CORE_SAN does not cover an address the edge's deployment.yml dials. A core certificate without that SAN fails name verification, and nothing is logged at INFO when it does. an endpoint was added to group_vars//deployment.yml after FED_CORE_SAN was set by hand unset FED_CORE_SAN and let goatbuilder derive it from deployment.yml, or add the missing address docs/design/federation-preflight.md 1
GB-146 goatbuilder docker compose is not available for phase 12 — the edge role deploys with docker compose, and this machine has no docker, no compose plugin, or a compose that reports no version. docker-compose-plugin is not installed install docker-compose-plugin (or drop the plugin binary at /usr/libexec/docker/cli-plugins/docker-compose) docs/design/federation-preflight.md 1
GB-147 goatbuilder a peering directory value is not usable — FED_DIRECTORY_URL is not https, or it, FED_PACKAGE_URL, FED_COUNTERPARTY_ID or FED_CA_CERT holds characters outside a plain URL. Those values reach a command line, so the run refuses them. an http:// directory address, or a value with spaces or quotes set goat_federation_directory_url to the https:// address of the directory docs/design/federation-preflight.md 1
GB-148 goatbuilder the federation working directory is not writable — the run's working directory (--work, FED_WORK, else /var/lib/goat-federation/) cannot be created or written. the default under /var/lib needs root goatbuilder federate <partner> --work <a directory you can write> docs/design/federation-preflight.md 1
GB-149 goatbuilder the peering is not declared — the partner is not in goatbuilder.yaml peerings:. A peering nobody declared is one nobody reviewed (contract C1). the peering was never added to the deployment's goatbuilder.yaml add the peering to ops/goatbuilder/goatbuilder.yaml peerings: (with, environment, site, edge_class) docs/design/federation-preflight.md 1
GB-268 goatbuilder a peering host's clock is not synchronized, is too far from UTC, or could not be read — federate phase 1 (preflight) read each edge and core host's clock from its own time service (chronyc -c tracking, else timedatectl). A Zenoh router refuses a timestamp more than 500 ms ahead of its own clock, and the partner's fabric is synchronized separately, so each side must hold its clock within half that, 250 ms by default (FED_CLOCK_BUDGET_MS), of UTC. A host that is not synchronized to an outside reference, or is over the budget, is a FAIL; a host whose clock could not be read is SKIP, never PASS. the host's time service follows no reachable time source, or only the host's own local clock
the clock is synchronized but still slewing a large step, or the time source itself is wrong
neither chronyd nor systemd answered the probe, so the clock was not read
on the named host: chronyc tracking (or: timedatectl show); repair its time synchronization, then run phase 1 again
goatbuilder federate <partner> --from 1 --to 1
docs/design/federation-preflight.md 1
GB-230 goatbuilder allocate would move a peering that already runs — A peering the allocation record does not hold already runs from hand-written configuration (its edge group has hand group_vars, or the hand inventory names it), and the values allocation chose differ from the ones it runs on (its hand group_vars, else the role defaults). Rendering them would move its tunnel or ports on the next phase 2. Nothing was written. goatbuilder federate allocate run for the first time in a deployment with a peering set up by hand adopt it as it runs: pin the values the refusal names under the peering's values: in goatbuilder.yaml (a pin on 51820 needs that environment's peering_edge_reserved_udp to allow it)
move it on purpose: FED_ALLOCATE_MOVE=<with> goatbuilder federate allocate --yes
docs/design/goatbuilder.md 2
GB-231 goatbuilder the hand-written peering inventory puts one name in two peering groups — No peering declares hosts:, so ops/ansible/inventory/peering-edges.ini is hand-written, and it puts one inventory name in two peering groups. Ansible merges the group_vars of every group a name is in, so each peering would read the other's rendered values. Nothing was written. one machine listed under the same name for two peerings give each peering its own inventory name for the machine, with the same ansible_host
declare hosts: on every peering and let goatbuilder federate allocate render the inventory
docs/design/goatbuilder.md 2
GB-234 goatbuilder the site inventory does not name one edge and one core for a peering without hosts — A peering declares no hosts:, so goatbuilder federate allocate takes its machines from the site inventory (the environment's inventory:, else ops/ansible/inventory/.ini but the peering-.ini files): the group of the peering's edge_class and of the site's other class, each named as the class, , or . More than one file names them, a class has no machine or several, or the two machines are reached as different users. The message names the candidates and where it looked. Nothing was written. two inventory files define the site's class groups
a class group lists several machines, or the edge class group lists none
the edge and the core have different ansible_user values
declare hosts: {edge: <address>, core: <address>, user: <user>} on the peering
declare the environment's inventory: so only that file is read
make the site inventory name one machine per class
docs/design/goatbuilder.md 2
GB-235 goatbuilder a named allocate would change another peering's inventory lines — Every peering declares hosts:, so ops/ansible/inventory/peering-edges.ini is the whole generated peering inventory. goatbuilder federate allocate <peering> renders the named peering's groups into it and leaves every other line as it was; this run would change or drop a section of another peering (its hosts were changed in goatbuilder.yaml since the last allocate, or it is declared but not yet recorded). Nothing was written. another peering's hosts: changed in goatbuilder.yaml since the inventory was last rendered
a peering in the generated inventory is declared but has no entry in the allocation record
goatbuilder federate allocate (every declared peering; review and commit what it changes) docs/design/goatbuilder.md 2
GB-236 goatbuilder our core router's goat-zenoh role declarations do not resolve for admission — Phase 10's admission names no route of its own (FED_CORE_PLAY and FED_CORE_RESTART are unset), and an inventory group of the site inventory (ops/ansible/inventory/.ini, not peering-.ini) has group_vars that configure the goat-zenoh role, so admission restarts the core through the role's own play on that group. It could not resolve that play: several groups configure the role and the peering core's address does not pick one, nothing gives the group's site_id, or no single fleet pin whose image variable the group sets names a deploy play that exists. The message names what was looked for and where. Nothing on the core changed. two inventory groups both configure the goat-zenoh role
the core router group's group_vars declare no site_id and the group is not named _hosts
no component in ops/pins/fleet-pins.yaml has the group's image variable as its var with a deploy play
declare site_id in group_vars/<core router group>/
keep one inventory group per core router, or put the peering's core in the peering inventory with the same address
set FED_CORE_PLAY or FED_CORE_RESTART to name the route by hand
docs/design/goatbuilder.md 2
GB-237 goatbuilder admission cannot write the partner into the role core's zenoh_sovereign_peerings — On a goat-zenoh role core, phase 10's admission writes the partner's zenoh_sovereign_peerings entry (sovereign, and the CN of the agreement's access-control fragment subject) into ops/ansible/group_vars//peerings.yml, the role's own declaration, beside the fragment. It could not: the fragment names no subject CN or several, another of the group's group_vars files already sets zenoh_sovereign_peerings (two owners), an entry's sovereign is a template (it contains {{, {% or {#, so no literal id matches it and the partner would be added a second time), peerings.yml cannot be rewritten without changing something else (admission replaces only the zenoh_sovereign_peerings block and keeps every other line of the file byte for byte, so it refuses a comment inside that list, a flow-style top level it cannot split into variables, or an anchor shared between the list and another variable), or peerings.yml is not YAML or not a mapping of variables. Nothing on the core changed. a comment inside the zenoh_sovereign_peerings list of peerings.yml, a flow-style top level, or an anchor in the list that another variable aliases
zenoh_sovereign_peerings is set by hand in another file of the core router group's group_vars
an entry of zenoh_sovereign_peerings writes its sovereign as a Jinja template, or is not a mapping with a literal sovereign
zenoh_sovereign_peerings itself is a template, a mapping or a scalar, not a literal list: read as empty, it would be written back holding this partner alone
the fragment phase 9 rendered names no subject, or several
move a comment out of the zenoh_sovereign_peerings list (above it, or elsewhere in the file), write the file onevariable: valueblock per variable, and keep anchors out of the list
move the zenoh_sovereign_peerings entries into group_vars/<core router group>/peerings.yml and remove the variable from the other file
write zenoh_sovereign_peerings as a literal list, and each entry's sovereign literally, as the partner's sovereign id
run phase 9 again, then make a new plan and deploy
docs/design/goatbuilder.md 2
GB-238 goatbuilder a federation verb's site flags are given where they do not belong — goatbuilder federate init|lint|compose|diff|verify|apply read this party's context from --context-dir, --cert, --ca-roots, --context and --profile-dir. Those flags belong to the six verbs and come after the verb: given before it they are read as the name of a partner, and given to a peering run or to federate plan they would be ignored. the flags were typed after federate and before the verb
the flags were given to goatbuilder federate <partner> or federate plan, which read the deployment's own declarations
goatbuilder federate lint --context-dir <dir> <package> (the verb first, then its flags)
drop the flags from a peering run or a plan
docs/design/goatbuilder.md 2
GB-239 goatbuilder a flag that does not apply to the federation verb it was given to — The six federation verbs (init, lint, compose, diff, verify, apply) read only the files they are named and the party's context, and run for real. --mock, -C/--repo (except for init --over-ssh) and -o/--output choose things they do not read; --over-ssh and its host flags belong to init alone. A run with one where it does not apply is refused rather than the flag ignored. A flag that belongs to the verb but needs another with it (a host flag on init without --over-ssh) is GB-325. --mock, -C/--repo or -o/--output given to a federation verb
--over-ssh or its host flags given to a verb other than init, to a plan or to a peering run
drop the flag; the verbs take --json for a JSON report
goatbuilder -C <repo> federate init --over-ssh <peering> (the one verb that reads a checkout)
FED_MEASURE=ssh goatbuilder federate <partner> (a peering run measures phase 3 over SSH)
docs/design/goatbuilder.md 2
GB-240 goatbuilder a federation switch names no choice it has — Two variables choose how a peering run works: FED_VERBS who runs the federation verbs (goatbuilder, in process, the default; or goat), and FED_MEASURE where phase 3 measures (ssh, the default; or here). Any other value is refused, as peering-standup.sh refuses it, so a misspelling cannot choose a different path on each avenue. FED_VERBS is set to something other than goat or goatbuilder
FED_MEASURE is set to something other than ssh or here
unset FED_VERBS and FED_MEASURE (the defaults)
FED_VERBS=goat goatbuilder federate <partner> (the verbs run in goat)
FED_MEASURE=here goatbuilder federate <partner> (phase 3 measures the machine the run is on)
docs/design/goatbuilder.md 2
GB-241 goatbuilder the environment named is not declared — A verb was given an environment that goatbuilder.yaml (or goatlite.yaml) does not declare in environments:. Nothing runs against an environment nobody declared. a misspelt environment name
the environment is declared in another checkout's goatbuilder.yaml
the names declared are listed in the message; pass one of them
add the environment to ops/goatbuilder/goatbuilder.yaml environments:
docs/design/goatbuilder.md 2
GB-242 goatbuilder the command line does not parse — The arguments do not make a command goatbuilder knows: a required argument is missing, a value is not one the option takes, two options that exclude each other were both given, or the verb is unknown. Nothing ran. a required argument or subcommand was left out
an option was given a value it does not take
two options that exclude each other were given together
a misspelt verb
goatbuilder <verb> --help (the arguments the verb takes)
goatbuilder --help (the verbs)
docs/design/goatbuilder.md 2
GB-243 goatbuilder a standup names no site — standup stands up one site and needs --site. It may be left out only on a goatlite run, where goatlite.yaml (preset lite) names the one site the environment has. --site was left out on an environment that is not a goatlite goatbuilder standup <environment> --site <site> docs/operations/goatnet-standup.md 2
GB-244 goatbuilder the config file cannot be read — goatbuilder reads the deployment's goatbuilder.yaml (or the file a verb was named) before anything else. It is not there, or it cannot be read. the path is wrong, or -C names another checkout
the checkout has no ops/goatbuilder/goatbuilder.yaml
goatbuilder -C <the deployment's checkout> <verb>
goatbuilder validate <path> (the file you meant)
docs/design/goatbuilder.md 2
GB-245 goatbuilder the config file is not YAML — The config file could be read, but it does not parse as YAML, so nothing in it can be checked. The message names the line and column. a bracket, quote or indentation error near the line named fix the line the message names, then goatbuilder validate <path> docs/design/goatbuilder.md 2
GB-247 goatbuilder the site or class named is not declared — standup and join-node act on a site goatbuilder.yaml declares under the environment, and join-node on a class that site declares and the verb converges (core, obs, breakglass, network-jump). The one named is not there. a misspelt site or class
the site or class is not declared yet
the names declared are listed in the message; pass one of them
declare the site under environments[].sites, or the class under its classes:, in ops/goatbuilder/goatbuilder.yaml
docs/design/goatbuilder.md 2
GB-248 goatbuilder the observability leg cannot run — A site that declares has_obs_host: true stands up its observability host in a leg beside the core. --obs-only asked for that leg alone and there is none to run, or the leg's play (ops/ansible/utility-host-bootstrap.yml) is not in this tree. The message says which. has_obs_host is not true, its group_vars or inventory host is missing (--obs-only)
the checkout has no ops/ansible/utility-host-bootstrap.yml
goatbuilder standup <environment> --site <site> --skip-obs (stand up the site without its obs host)
declare has_obs_host: true, the <site>_obs_hosts group_vars and its inventory host, then run --obs-only
docs/operations/goatnet-standup.md 2
GB-249 goatbuilder the fixture's standup does not fit what was asked — standup --mock rehearses the standup a fixture's converger.yaml declares for one environment and site, against that fixture's goatbuilder.yaml. The environment or site asked for is another, or a phase of the rehearsal names a site or class the declaration does not allow (a joiner may touch only the founder it attaches to; a class needs machines). --mock names a fixture for another environment or site
the fixture's converger.yaml and goatbuilder.yaml disagree
goatbuilder standup <the fixture's environment> --site <its site> --mock <fixture>
correct the fixture's converger.yaml standup: or its goatbuilder.yaml
docs/design/goatbuilder.md 2
GB-250 goatbuilder a joiner site names no founder — A site whose vars.yml says goatnet_founder: false joins another site's goatnet, and seeds from it, so it must name that founder in join_founder_site. join_founder_site is missing from the joiner's group_vars/_hosts/vars.yml set join_founder_site: <founder site> in group_vars/<site>_hosts/vars.yml docs/tutorials/02-first-federated-node.md 2
GB-251 goatbuilder no SSH user is declared for a new machine — join-node reaches a new machine as the project's declared SSH user before anything is installed on it. The checkout declares none, and no name is right for every project's machines, so a real run refuses (a dry run plans with a placeholder). deploy.ssh_user (or lite.ssh_user) is not set in goatbuilder.yaml set deploy.ssh_user in ops/goatbuilder/goatbuilder.yaml
goatbuilder join-node ... --dry-run (plans without it)
docs/design/goatbuilder.md 2
GB-252 goatbuilder the partner is declared on several environments — The same partner is declared as a peering on more than one environment in goatbuilder.yaml peerings:, so the command cannot tell which peering it is about. one partner is peered from two environments, and --env was not given pass --env <environment> (the environments are named in the message) docs/design/goatbuilder.md 2
GB-253 goatbuilder a host's back-out survey answered for another peering — Back-out surveys each host of the peering read-only before it plans what to remove, and every answer names the peering it read. One host's answer names another peering, so nothing is planned from it: removing that peering's resources would be removing someone else's. the inventory groups of two peerings share a host name
the survey ran with another peering's working directory
goatbuilder federate backout <partner> --plan (survey again with this peering's own work directory)
give each peering its own inventory name for the machine (allocate does)
docs/design/goatbuilder.md 2
GB-254 goatbuilder a file goatbuilder reads or writes cannot be read or written — A verb reads or writes a file of the checkout or its state (a pin table, a schema, a record, an inventory, a play) and the operating system refused: the file is missing, unreadable, or its directory is not writable. The message names the file and the system's reason. the file is not in this checkout, or -C names another one
the file or its directory is not readable or writable by this user
ls -l <the file the message names>
goatbuilder -C <the right checkout> <verb>
docs/design/goatbuilder.md 2
GB-255 goatbuilder nothing is declared for the environment — The fixture's (or checkout's) declarations name no component for the environment asked for, or none by the name given, so there is nothing to read, plan or roll back. a misspelt environment or component
the environment declares no component yet
goatbuilder pins --check (what fleet-pins.yaml declares)
pass an environment and component the declarations hold
docs/design/goatbuilder.md 2
GB-256 goatbuilder a deploy play's hosts cannot be resolved — To target a deploy, goatbuilder reads the component's play for its hosts: line, following at most three import_playbook hops. The play has neither, or the chain is longer. the play has no hosts: line and no import_playbook
more than three import_playbook hops
give the play a hosts: line ("{{ target }}" for a play targeted per group)
declare deploy_playbook in fleet-pins.yaml for the play that does
docs/design/goatbuilder.md 2
GB-257 goatbuilder fleet-pins.yaml is not in the shape goatbuilder edits — goatbuilder reads fleet-pins.yaml's components and edits a component's block in place. The file has no components:, or the component's block or its fleet: line is not where the writer looks. fleet-pins.yaml has no components: mapping
the component's block was edited by hand into another shape
goatbuilder pins --check (the table as goatbuilder reads it)
restore the component's block to:with afleet:line, as the others are
docs/design/goatbuilder.md 2
GB-258 goatbuilder a command a verb runs failed — A verb runs git, gh, ansible or one of the repository's own steps and stops when it fails. The message is that command's own error output; it names what went wrong. the command's own error, quoted in the message (a network failure, a missing credential, a refused push) read the quoted output: its tool's own catalog entry may name it (goatbuilder explain '<the line>')
run the command by hand with the same arguments (goatbuilder -v shows it)
docs/design/goatbuilder.md 1
GB-259 goatbuilder the assembled tree's marker is incomplete — An assembled checkout (a goatcore commit plus an overlay) carries .goat-assembled, naming the goatcore_ref and overlay_commit it was built from; fresh-source checks against them. A line is missing. the tree was assembled by hand, or the marker was edited assemble the tree again with the overlay's assembly step
cat .goat-assembled (goatcore_ref= and overlay_commit= lines)
docs/design/goatbuilder.md 2
GB-260 goatbuilder a rollback has no history to go back to, or its target is refused — rollback goes back through the host's deploy ledger and the in-repo history (deploy-history.yml). There is no history for the component, not enough of it, or the ref it would roll to is flagged do_not_roll_to, or a host runs a ref neither history knows. the component has not been adopted into deploy-history.yml
the history holds fewer entries than the rollback asks for
the target ref is flagged do_not_roll_to in the in-repo ledger
a host runs a ref no history records (deployed by hand?)
goatbuilder rollback <component> --status
investigate the host the message names before rolling back
docs/design/universal-deploy-rollback.md 2
GB-263 goatbuilder a site id cannot name the variables its phases read — Phase 12 reads the NetBird token from _NB_API_TOKEN, named after the site that governs the ACL: the site itself, or a joiner's founder (join_founder_site). That site holds a character a variable name cannot, so the standup stops before any phase. join_founder_site in the site's group_vars holds a space, a quote or another character outside letters, digits, - and _
the site was named with such a character
set join_founder_site to the founder's site id (letters, digits, - and _)
goatbuilder validate (the site ids goatbuilder.yaml declares)
docs/design/goatbuilder.md 2
GB-264 goatbuilder no recorded run matches — goatbuilder show and runs diff name a run by its id, a prefix of one, or last, from the run records under /runs. None is recorded there, none starts with the prefix given, or several do. no verb that drives hosts has run with this state directory (a --mock run writes no record)
a mistyped id, or a prefix too short to name one run
goatbuilder runs (the recorded runs, newest first)
give more of the run's id
docs/design/goatbuilder-legibility.md 2
GB-246 goatbuilder a rollback depth outside the ledger — rollback N goes back N deploys in a host's ledger, which keeps the last seven. N outside 1 to 7 names no deploy the ledger holds. N is 0, or more than 7 goatbuilder rollback <component> --env <goatnet> <N from 1 to 7> docs/design/goatbuilder.md 2
GB-185 goatbuilder the partner named to federate status is not a declared peering — goatbuilder federate status <partner> reads a peering declared in goatbuilder.yaml peerings: (on --env, when given). The name given is not one of them, so nothing was read. the name is misspelled, or the peering is declared under another environment's config
the peering has not been declared yet
goatbuilder federate status (every declared peering)
declare it under peerings: in goatbuilder.yaml first (C1)
docs/design/goatbuilder.md 1
GB-186 goatbuilder the plan was not approved — goatbuilder federate deploy executes a plan only when approval names that plan's hash: --approve-plan <hash>, or a person at a terminal typing yes after reading the plan and its hash. --yes never approves a plan. No approval was given, or it named another hash, so nothing changed. there is no terminal and --approve-plan was not passed
--approve-plan names the hash of an older plan (the plan was made again since)
the person answered anything but yes
read the plan, then: goatbuilder federate deploy <partner> --approve-plan <the hash it prints>
goatbuilder federate plan <partner> (a new plan, when the old one is out of date)
docs/design/goatbuilder.md 1
GB-187 goatbuilder the plan is not the one approved — The plan deploy was given is for another peering, or its hash changed between approval and execution (it is read again under the peering's lock, just before anything changes). Nothing changed. A plan that does not match its own hash is GB-201. no plan has been made yet (/plan.json is missing)
the plan was made again, or edited, after it was approved
--plan names another peering's plan
goatbuilder federate plan <partner> (then read it and approve its hash) docs/design/goatbuilder.md 1
GB-188 goatbuilder the plan no longer holds — Immediately before anything changes, deploy runs the planner's live recheck: every live prerequisite is checked again and every input section is compared with a plan built now. Something did not hold (a section changed, a check was not PASS or NOT APPLICABLE or was not answered, or the plan was not READY), so the approved plan no longer describes the change, and a saved plan is not proof the environment stayed the same. Nothing changed. the agreement was composed again (a new package from either side)
apply rendered the files again, or one was edited by hand
the plan recorded a SKIP of a check deploy does not run again (host-allocation): the inventory names no host for this peering, or a value of another peering on a machine this peering shares does not resolve. The message says which, and what clears it
a SKIP the plan accepted (plan --accept) reads differently now: the message names what it reads now and what was accepted
goatbuilder federate plan <partner> (a new plan from what is there now; approve its hash)
for a host-allocation SKIP, do what its line says first (goatbuilder federate allocate <partner>, or name the hosts in ops/ansible/inventory/peering-edges.ini, or set the named value in group_vars), then make a new plan: a new plan alone records the same SKIP
goatbuilder federate plan <partner> --accept host-allocation (when you have judged that SKIP: it is recorded with its exact detail, and deploy goes ahead only while it reads the same)
docs/design/goatbuilder.md 1
GB-189 goatbuilder a stage of goatbuilder federate was asked for the wrong way — The stages status, deploy and test take the partner after them and run their own phases; --plan and --approve-plan belong to deploy, whose start is phase 10 or, once the admission is done by hand, 11. Nothing ran. the partner was left out, or given after a peering name instead of a stage word
--to, --resume or a --from other than 10 or 11 was given to a stage
--plan or --approve-plan was given outside deploy
--bilateral was given outside test
rotate <partner> <key> named a key whose rotation is designed and not built in this goatbuilder (transport or signing; this goatbuilder rotates the leaves and the carrier keys), or a key it does not know; the message says what is missing (for transport, that the partner cannot yet receive our new package)
goatbuilder federate deploy <partner> --approve-plan <hash> (or --from 11 after the admission)
goatbuilder federate test <partner>
goatbuilder federate rotate <partner> leaves (or carrier: the keys this goatbuilder rotates; the message lists them)
docs/design/goatbuilder.md 1
GB-282 goatbuilder rotating our CA needs a decision before it can be built — goatbuilder federate rotate <partner> ca would replace the CA that signs our peering leaves. In package v1 every check accepts exactly one CA per party, the one fingerprint our package carries (federation-package-spec.md §5.1), so the partner could not trust the new CA beside the old one while our leaves change over, and data would stop. The overlap of both CAs is decided (ADR 1242: our next CA is announced in package v2) and designed in the key-rotation design §4, but this goatbuilder does not build it yet. Nothing ran. rotate <partner> ca was asked for, and this goatbuilder does not yet build the CA overlap ADR 1242 decides rotate the leaves instead, under the same CA: goatbuilder federate rotate <partner> leaves
read ADR 1242 and the key-rotation design §4 for the CA overlap a later goatbuilder builds
docs/design/federation-key-rotation.md 1
GB-190 goatbuilder the admission on our core is not recorded as done for what is rendered now — deploy --from 11 skips phase 10, the by-hand admission of the partner's edge on our core, so it runs only when the journal records phase 10 as done for exactly the files rendered now. It does not, so nothing changed: either the admission was never declared, or the rendered files changed after it. deploy has not stopped at phase 10 yet, so the admission was never done for these files
the files were rendered again after the admission was declared
goatbuilder federate deploy <partner> --approve-plan <hash> (it stops at phase 10; do the admission, then run it again with --from 11) docs/design/goatbuilder.md 1
GB-191 goatbuilder a value for confirmation 1 does not match — At phase 5 or 6 a value given by flag or typed is not the one the files hold: their CA fingerprint is not their package's, their CA certificate does not carry that fingerprint, their verifying key is not 32 bytes of base64 or does not verify their half, a sovereign id is not a plain id or is not the declared counterparty, or the agreement digest differs. The run stopped; at phase 5 no keyring was written, and at phase 6 the composed agreement is left for inspection. a value was misheard or mistyped on the call
the partner published a new package or CA after reading theirs aloud
the two sides composed different agreements (their halves differ)
read the values back group by group with the partner, then run the same command again
goat federate diff <ours> <theirs> (names the field where two agreements differ)
docs/design/goatbuilder.md 1
GB-192 goatbuilder confirmation 1 is waiting — Phase 5 or 6 was asked to confirm (by --confirm or a value) and something is not there yet: their half or CA certificate has not arrived, a value read aloud was not given and there is no terminal to ask at, or the person did not type yes. The run stopped there, as a by-hand stop does (exit 0); a stop at phase 5 writes no keyring. the partner has not published yet
one of --partner-key, --partner-ca-fingerprint, --agreement-digest is missing
run the same command again with the values read aloud, or with --confirm at a terminal docs/design/goatbuilder.md 0
GB-193 goatbuilder our verifying key cannot be read, or does not verify our half — Phase 5 shows our verifying key to read aloud, derived from the Ed25519 seed that signs our package (FED_SIGNING_KEY: 64 hex characters or 32 bytes base64). It is unset, unreadable or not a seed, or the key it gives does not verify the signature on our half, so the partner would bind a key that cannot verify our package. FED_SIGNING_KEY is not set in this environment
it names a file that is not the package-signing seed, or a different seed than phase 3 signed with
set FED_SIGNING_KEY to the seed that signed our package (the same one phase 3 uses) docs/design/goatbuilder.md 1
GB-194 goatbuilder the edge cannot be admitted on our core — Phase 10 by the command (confirmation 2) did not change our core: the agreement digest has not been confirmed with the partner for the agreement in place, --approve-admission names another digest (something changed since it was read), the deployment does not say where our core's access control lives or how it restarts, their CA does not match the agreement, or the fragment could not be merged, assembled or placed (what was placed is put back). phase 6 ran without --agreement-digest (or --confirm), so the digest was never compared
the files were rendered again after the summary was read
FED_CORE_ACL_ORG, FED_CORE_PLAY or FED_CORE_RESTART, or FED_SLOT_* is not set
their CA file holds more than the one certificate the agreement names
compare the digest with the partner: goatbuilder federate <partner> --from 6 --to 6 --agreement-digest <digest>
read the summary again and approve the digest it shows now
set the FED_CORE_* settings in the deployment's environment, or run without the admission flags so phase 10 stops, admit by hand, and run --from 11
docs/design/goatbuilder.md 1
GB-195 goatbuilder the core's dry run or the slot check refused the admission — After placing the new access control (nothing loads it before a restart), the core router's dry run failed, or the committed slot list differs from the registry or the registry could not be read. Restarting now could drop or add stores, so the files placed were put back and the core was not restarted. the assembled configuration does not pass the core's play in check mode
the slot list in the repository is behind the registry
run the dry run by hand to read why (the message prints it)
regenerate the slot list with ops/scripts/sync-slot-anchors.py, commit it, deploy the core, then run phase 10 again
docs/design/goatbuilder.md 1
GB-196 goatbuilder the core restart failed after the admission was placed — The new access control is in place on our core, and the restart through the deployment's play or command failed, so it may not be loaded. the play or restart command failed (its output is above) fix the cause, then run the same stage again with --approve-admission <digest> docs/design/goatbuilder.md 1
GB-197 goatbuilder the admission is waiting for approval — Phase 10 was asked to admit the edge (by --confirm, --approve-admission or --review-in-pr) and stopped without changing our core: the summary was written for review in a pull request, the person did not type yes, or there is no terminal and no --approve-admission. Exit 0, as a by-hand stop. --review-in-pr, or pr at the terminal
no approval was given
read the summary (or the review in <work>/admission/review.md), then run the same stage with --approve-admission <digest> docs/design/goatbuilder.md 0
GB-200 goatbuilder the federation plan's inputs changed — an input the saved plan rests on (versions, the agreement, a rendered file, a certificate, a resource or the acceptance policy) differs from what the plan recorded, so the approval given to that plan does not cover this change. a phase rendered, minted or composed again after the plan was made
the checkout, the goat binary or a declared value changed after the plan was made
goatbuilder federate plan <partner>, then review and approve the new plan docs/design/federation-plan.md 1
GB-201 goatbuilder the federation plan is missing or altered — /plan.json is absent, is not a goat-federation-plan/v1 plan, or does not match its own hash, so it cannot stand for what was approved. no plan was made for this working directory
the plan file was edited by hand after it was written
goatbuilder federate plan <partner> --work <dir> docs/design/federation-plan.md 1
GB-202 goatbuilder a live prerequisite of the federation plan no longer holds — rerun immediately before a change, a live phase-1 check (ports, slot list, core router, receive stores) did not pass, so nothing may change. SKIP is not a pass here. another service took a port the plan expected free
a slot was minted after the slot list was fixed
the core router was replaced or stopped
fix the cause the check names (its own code is shown beside GB-202), then goatbuilder federate plan <partner> --recheck docs/design/federation-plan.md 1
GB-232 goatbuilder a rendered deployment half needs a site value no declaration yields — The peering declares deployment: rendered, so goatbuilder federate allocate renders the edge's and the core's deployment halves (peering-deployment.yml). Each required site value is taken from the group_vars of the environment's groups (environments[].groups) or all when one sets it, and is otherwise derived from the declaration that owns it: the core's address from peerings[].hosts.core or its line in the peering inventory, the core router's port from goat_zenoh_listen_port (group_vars, else the goat-zenoh role default), the images from the role defaults and ops/pins/fleet-pins.yaml, the core's certificate name from its name in the site inventory (ops/ansible/inventory/*.ini). One of them could be found in none of these places; the message names each place it read. Nothing is invented and nothing was written. the peering declares no hosts: and the peering inventory has no single host for its core group
no role default and no fleet-pins component pins the image the message names
the site values live in a group the environment does not name in environments[].groups
declare what the message names where it looked (peerings[].hosts, the core group's inventory line, the image pin)
or set the named variable once in the group_vars of one of the environment's groups
or declare deployment: hand on the peering and keep writing its deployment halves by hand
docs/design/goatbuilder.md 2
GB-233 goatbuilder a peering's federation.toml cannot be generated from the environment's — The peering declares federation_toml: rendered, so goatbuilder federate allocate generates its federation.toml from the environment's own ops/federation//federation.toml, the one owner of sovereign_id, term_days, cut_contact, the release sets, the accept limits and the measured path MTU, and fills [transport] from the allocated values. That file is missing, is not TOML, or lacks a required top-level value. Nothing was written. the environment has no ops/federation//federation.toml yet
sovereign_id, term_days or cut_contact is missing from it
write ops/federation/<environment>/federation.toml once for the environment (the template goat federate init prints is the shape)
or declare federation_toml: hand on the peering and keep writing its federation.toml by hand
docs/design/goatbuilder.md 2
GB-160 goatbuilder two peerings on one forwarder would share a value — The peerings one forwarder relays must share no carrier interface, public port, carrier port or carrier address on it: the forwarder sends each packet by its port alone. Allocation keeps them apart across every environment the forwarder relays (by its endpoint); this refusal is the backstop for values that collide anyway. a hand edit of peering-values.yaml gives two relayed peerings one forwarder value
a pin under values: (public_port, carrier_port, carrier_interface, carrier_subnet) repeats another peering's value
pin different public_port, carrier_port, carrier_interface and carrier_subnet values under one peering's values:
or give each environment its own forwarder (forwarder: group:)
docs/design/peering-forwarder.md 2
GB-161 goatbuilder a peering's partner id cannot name a forwarder entry — The forwarder names each peering it relays by the partner id, and its role accepts 2-16 characters of a-z 0-9 and hyphen. This partner id is outside that, so the forwarder would refuse the entry. the peering's with: holds an underscore, or is one character or longer than 16 declare the partner by its sovereign id (lower-case letters, digits, hyphens) docs/design/peering-forwarder.md 2
GB-162 goatbuilder one forwarder group names two forwarders — The forwarder's peerings are rendered into its inventory group's group_vars, so every host in the group would relay every peering. Two peerings name the same group with different forwarder endpoints. two forwarders declared without forwarder: group:, so both took the default peering_forwarders give each forwarder its own group under forwarder: group: docs/design/peering-forwarder.md 2
GB-163 goatbuilder a forwarded peering has no allocated values yet — The peering declares forwarder:, so phase 2 runs the forwarder play and copies the carrier keys, and both read the group_vars goatbuilder federate allocate renders from the allocation record. The record does not hold this peering, so those files do not exist. Nothing ran. forwarder: was added to the peering and goatbuilder federate allocate was not run since goatbuilder federate allocate (then review and commit what it writes) docs/design/peering-forwarder.md 2
GB-203 goatbuilder two peerings would bind the same value on one machine — this peering and another declared peering would bind the same WireGuard interface, tunnel UDP port or republisher metrics port, or overlapping link subnets, on the same machine (by inventory address), so one would break the other. two environments, or two hand-named inventory groups, place their edges or cores on one machine
a value was pinned or declared by hand without looking at the other peerings on that machine
give this peering another value in its edge or core group's vars (or its allocation pin), then goatbuilder federate plan <partner> docs/design/federation-plan.md 1
GB-220 goatbuilder federate backout was not given a partner it can act on — goatbuilder federate backout takes one partner: a sovereign id (2-16 characters of a-z0-9-) that is not a federate subcommand. The working directory and edge group must be plain paths and names, because they go into the play's command line. Nothing ran. the partner was left out, or is not a sovereign id
--work or --edge-group holds a quote, a space or a shell character
goatbuilder federate backout <partner> --dry-run docs/design/goatbuilder.md 1
GB-221 goatbuilder the back-out found a resource it cannot show is this peering's alone — The read-only survey found a resource under this peering's names that another peering holds (a WireGuard interface whose state file or peer names another partner, a carrier another peering's state names), or a core configuration it cannot prove changes by this peering alone (the live file is not seed + org fragment, partner ids overlap, a policy mixes partners). Nothing was changed on any host. two peerings were given the same interface, carrier or stack name in their group values
the core's live configuration was edited by hand after the last assemble
the core keeps no seed/org split (slot-pool mode), so the admission cannot be located
read the REFUSED lines of goatbuilder federate backout <partner> --dry-run; give each peering its own names, or reconcile the core's live file with its seed and org fragment docs/design/goatbuilder.md 1
GB-222 goatbuilder the back-out survey did not complete — ops/ansible/peering-backout.yml failed before every host of the peering answered, or no host answered while the working folder holds the peering's agreement, so there is no plan to show and nothing was changed. a host of _edge, _core or peering_forwarders is unreachable
neither ops/ansible/inventory/peering-edges.ini nor a per-peering peering-.ini names the peering's hosts, so the play matched no host
read the play output above; fix reachability or the inventory, then goatbuilder federate backout <partner> --dry-run docs/design/goatbuilder.md 1
GB-223 goatbuilder the slot recheck did not allow the core router restart — Immediately before the core router restarts without this peering's admission, the committed slot list is checked against the registry, as the admission path does. It drifted, the registry could not be read, or no registry was named, so the core was not restarted. The edge side of the peering is already removed and the partner has no path to the core. slots were minted or revoked since the list was committed
FED_SLOT_DB (or FED_SLOT_REGISTRY_STORE=postgres with FED_SLOT_PG_SSH) is not set for a deployment with lazy slot storage
regenerate and commit the slot list (ops/scripts/sync-slot-anchors.py --env <env> ...), then goatbuilder federate backout <partner> docs/design/goatbuilder.md 3
GB-224 goatbuilder the core router was not restarted on the reviewed candidate — The core step installs only the candidate configuration whose digest the survey printed. The core's files changed since the survey (so the candidate's digest differs), more than one core router was found, or the restart failed. The org fragment kept beside it as .before-backout- is the state before the step. another change reached the core between the survey and the restart
several containers qualify as the core router
goatbuilder federate backout <partner> --dry-run, review the new candidate, then run it again
name the router with peering_backout_core_container or FED_ROUTER_CONTAINER
docs/design/goatbuilder.md 3
GB-225 goatbuilder resources of the peering are still present after the back-out — The survey run after the removal still finds resources of this peering, or the removal play stopped partway. Every step is idempotent: running the back-out again removes what is left and reports the rest as absent. a host became unreachable during the removal
a unit or container would not stop
goatbuilder federate backout <partner> docs/design/goatbuilder.md 3
GB-226 goatbuilder our package for the counterparty could not be withdrawn from the directory — The back-out asks the peering directory to delete /v1/parties//packages/, signed by our bound key. The directory refused or could not be reached. This is a warning: a listing grants nothing, and the peering's hosts are already clean. Our single entry and every other party's entry are never touched. FED_SIGNING_KEY is not set, or is not the key the directory binds to us
the directory is unreachable
python3 ops/federation/standup/directory_withdraw.py --url <directory> --us <us> --them <partner> --key <signing key> docs/design/peering-directory-service.md 0
GB-227 goatbuilder the core still admits the partner through its own configuration — The core router was configured by roles/goat-zenoh from zenoh_sovereign_peerings, which declares the partner's subject and grants as sovereign- ids. The back-out takes the partner's entry out of group_vars//peerings.yml, the file admission writes it into, and runs the role's play, which restarts the router without the partner's store and then deletes that store whole (ADR 1234). After the back-out the core still declares the partner: the entry is in a file admission does not write, the back-out could not rewrite peerings.yml (GB-237), the core was admitted through the deployment's own route (FED_CORE_*), or the role's play failed. The partner has no edge left to reach the core through. the core admits the peering from a zenoh_sovereign_peerings entry the back-out did not write, or could not take out
the role's play failed after the entry was taken out, and the entry was put back
remove the partner's entry from zenoh_sovereign_peerings by review where it is declared and run the core play (a router restart); the role then reports sovereign_from_<partner> in .undeclared-stores
after a failed play: read its output, then goatbuilder federate backout <partner> again
docs/operations/sovereign-federation-runbook.md 3
GB-270 goatbuilder a backed-out partner's store was kept because the router does not yet run without it — On a goat-zenoh role core, the back-out took the partner out of zenoh_sovereign_peerings, ran the role's play to restart the router without the partner's store, then ran ops/ansible/zenoh-drop-partner-store.yml, which deletes /sovereign_from_ only after the router runs without it (ADR 1234). One of the two plays did not get there: the partner is still declared, the live configuration still names the store, the router is not running, or it started before that configuration was written. The store was kept, and the back-out put the partner's entry back in peerings.yml, so running it again repeats the step. the role's play failed before or during the router's restart
the router did not restart after the configuration without the store was written
the partner is declared in another of the core router group's files
read the role play's output above, then goatbuilder federate backout <partner> docs/design/federation-backout-partner-data.md 3
GB-276 goatbuilder --mock was given to a verb that reads the checkout — --mock runs a verb against a fixture directory instead of hosts. Only the verbs that drive or read hosts take one (the message lists them). This verb reads the checkout itself, so a fixture it was named would have been ignored, and the run would have reported on files the person never pointed it at. Nothing ran. --mock given to a verb that reads the checkout, such as pins, validate or doctor drop --mock: the verb reads the checkout (-C <dir> names another one)
goatbuilder --mock <fixture> <verb> with one of the verbs the message lists
docs/design/goatbuilder-conformance.md 2
GB-277 goatbuilder an edge result cannot be imported — On a two-driver peering (drivers: two), the controller imports the file the edge driver wrote for a phase's edge half with goatbuilder federate import <peering> <phase> <file> --sha256 <code>, the confirm code the edge driver printed. The import refused it: the file cannot be read; the code is not hex, gives fewer than the 32 hex digits the edge driver prints, or is not the start of the file's sha256; the file is not that phase's edge result for this peering (another schema, another peering, another side); or this peering's journal here is not waiting on the edge driver at that phase. Nothing was placed or recorded. the file was changed, or not copied whole, on its way from the edge driver
the confirm code was mistyped, or read from another run of the edge driver
the file is another peering's, another phase's, or the core side's result
the controller's run has not reached the phase's hand-off in this working directory
copy the file the edge driver wrote again, and give the confirm code it printed: goatbuilder federate import <peering> <phase> <file> --sha256 <code>
goatbuilder federate <peering> --from <phase> (on the controller, to reach the hand-off first)
docs/design/federation-two-drivers.md 1
GB-286 goatbuilder a joining site has no leaf-cert manifest join-node can read — join-node mints a joining site's router and service leaves from the site's manifest (ops/enrollment/manifests/-certs.yaml, with mint-site-certs.sh) and stages them onto the node before the data router. The manifest is missing, or is not readable YAML, so the plan was refused before any phase ran. the site's manifest was never written
the manifest has a YAML error
cp ops/enrollment/manifests/TEMPLATE-certs.yaml ops/enrollment/manifests/<site>-certs.yaml, then re-thread every value for the site: its URI SANs name this goatnet
ops/scripts/mint-site-certs.sh <site> --dry-run prints what the manifest mints, and names a YAML error
docs/design/goatbuilder.md 1
GB-287 goatbuilder a joining site's leaf-cert manifest has no router leaf — join-node deploys a joining site's data router (zenoh-deploy.yml), whose play reads the router leaf that mint-site-certs.sh mints from the site's manifest. The manifest names no leaf called router, so the plan was refused before any phase ran. the manifest was written without the router entry ops/enrollment/manifests/TEMPLATE-certs.yaml carries add a leaf named router to ops/enrollment/manifests/<site>-certs.yaml, as the template shows, with the site's own SANs docs/design/goatbuilder.md 1
GB-288 goatbuilder a health publisher a joining site declares has no play in the checkout — join-node deploys each health publisher the end-state model marks present at a joining site (endstate.py model , from the pin table) after the site's data router, by running the publisher's play limited to the node. The model marks a publisher present whose play (ops/ansible/-deploy.yml) is not in this checkout, so the plan was refused before any phase ran. the checkout is an older or partial tree that does not carry the publisher's play
the pin table declares a publisher at the site that the site does not mean to run
run join-node from a checkout that carries ops/ansible/<publisher>-deploy.yml
declare the publisher absent at the site in ops/pins/fleet-pins.yaml, with its reason: components: <publisher>: deviations: <site>_hosts: {absent: true}
goatbuilder endstate <env> shows what the model marks present at each site
docs/design/goatbuilder.md 1
GB-296 goatbuilder a site turned on the audit spool, which is not built — A site set audit_subscriber_spool_enabled: true. The spooling audit subscriber a phone site needs (docs/design/island-mode-audit-spool.md) is designed but not built, so there is nothing to deploy for it. The audit-subscriber role stopped before it changed anything. a phone site's group_vars set audit_subscriber_spool_enabled: true before the spool exists set audit_subscriber_spool_enabled: false in that site's group_vars: the end state then reports the site's own statement that its fabric has no audit docs/design/island-mode-audit-spool.md 2
GB-293 goatbuilder the back-out cannot tell what our core router's leaf held before this peering — Our core router's leaf (its certificate and key) holds the leaf this peering's admission placed, and no admission run in the peering's working folder recorded what was there before (both peerings admitted by hand, say), so the back-out can neither put it back nor remove it. No other peering is set up from this machine (no working folder beside this one holds an agreement); when one is, the leaf is kept for it and the back-out goes on. Nothing was changed on any host. the leaf was placed by hand, not by goatbuilder federate admission
the admission folder of the peering's working folder was removed or moved
read the unknown core leaf lines of goatbuilder federate backout <partner> --dry-run; put back what each path held before this peering, or remove it when nothing did, then run the back-out again docs/design/federation-backout-partner-data.md 1
GB-280 goatbuilder a joining site has no mesh DNS zone to serve — join-node deploys a mesh DNS replica on a joining site's core node, so the site still resolves the goatnet's names when it is cut off from its founder (ADR 1235). The replica serves the goatnet's generated zone, an ops/dns//mesh-hosts whose records carry the site's site_dns_zone (the founder's when the site declares none). No such file was found, or neither site declares site_dns_zone, so the plan was refused before any phase ran. the goatnet's zone was never generated or committed (ops/dns/generate-mesh-hosts.py)
neither the joining site's nor the founder's group_vars declare site_dns_zone
python3 ops/dns/generate-mesh-hosts.py, or --init <site_id> for a zone that has no file yet, then commit ops/dns/<dir>/mesh-hosts
declare site_dns_zone in group_vars/<site>_hosts/vars.yml
docs/adr/1235-per-site-mesh-dns-nameserver-groups.md 1
GB-281 goatbuilder the founder's management API token for mesh DNS is unset or unresolved — join-node's mesh DNS step registers the joining site's nameserver group through the founder's management API, with the token in _NB_API_TOKEN. After the secrets injector ran, the variable was empty or still held a secret store's reference, so the step was refused before it registered anything. The message names the variable, never its value. the variable is not stored under the declared secrets posture
the store's tool is not signed in, so with-secrets could not resolve the reference
python3 ops/secrets/posture.py hint <FOUNDER>_NB_API_TOKEN says where to store it
sign in to the declared store's tool, then re-run: goatbuilder join-node <environment> --site <site> --node <node> --class core --address <address> --from 13
docs/adr/1235-per-site-mesh-dns-nameserver-groups.md 1
GB-299 goatbuilder no phone, or no Linux VM on it, answers over the Android debug bridge — join-node --via usb reaches a phone node through the Android debug bridge (adb) on a USB cable: SSH is carried by adb shell nc <the VM's address> 22, and the address is looked up on every connection (ops/scripts/phone-node-ssh.sh). adb is not installed, lists no phone as device, the phone is locked, or the phone's Linux VM has no address. Nothing reached the node. the phone is not attached, USB debugging is not allowed on it, or this machine's key was not accepted on its screen
the phone is locked
the Terminal app is closed, so the VM is not running
more than one device is attached and ANDROID_SERIAL does not name the phone
adb (Android platform tools) is not installed on this machine
adb devices (the phone must be listed as device)
unlock the phone and open its Terminal app, then resume: goatbuilder join-node <environment> --site <site> --node <node> --class phone --via usb --from <phase>
ops/scripts/phone-node-ssh.sh --vm-address (prints the VM's address, or says which of the above it is)
docs/design/phone-node.md 1
GB-300 goatbuilder the phone's Linux VM is not a host a node can be built on — For a phone node, phase 2 of join-node prepares the phone's Linux VM (Debian's kernel with its boot settings, Docker Engine, the deploy user) and then runs the integrity checks in the VM (ops/scripts/phone-node-provision.sh, ops/scripts/phone-node-integrity.sh). The VM could not be prepared, did not come back on Debian's kernel, or failed a check; the line before this one names which. No later phase ran, so nothing is built on the VM. the VM started again on the kernel Android supplies, or without the boot settings
file reads differ between cache drops, a write does not read back the same, or an installed file fails dpkg --verify: the VM returns wrong file data
the kernel log holds a filesystem or I/O error
the WireGuard module does not load, or a container does not run
the phone does not support a VM, or has no Linux Terminal app
read theintegrity:line the phase printed; it names the check
ops/scripts/phone-node-provision.sh --node <node> --check (the checks alone)
a VM that returns wrong file data is reset and prepared again: ops/scripts/phone-node-provision.sh --node <node> --reset
then resume: goatbuilder join-node <environment> --site <site> --node <node> --class phone --via usb --from 2
docs/design/phone-node.md 1
GB-301 goatbuilder a site does not hold what the machine class phone declares — The machine class phone declares once, in ops/ansible/group_vars/class_phone.yml, what a phone node differs in: no LAN address, no public ingress, no disk encryption inside the VM, memory limits for 4 GB, no observability host. The config preflight (standup-config-preflight.py _hosts --class phone, phases 3, 7 and 12 of join-node) found the site at odds with it: its group_vars set one of the class's keys to another value or set a key the class rules out, its node is not under [class_phone] in inventory/mgmt-hosts.ini or has a LAN address there, or the class file is not in the checkout. The same code refuses --class phone at a site that does not declare goatnet_founder: false, before any phase. the site's group_vars carry a value copied from a site of another class
the node's line was not added under [class_phone] in ops/ansible/inventory/mgmt-hosts.ini
the node's line in ops/ansible/inventory/mgmt-hosts.ini carries an ansible_host
the site is a founder: a phone node joins a goatnet
python3 ops/scripts/standup-config-preflight.py <site>_hosts --class phone (each [class phone] line names the key or the inventory line)
remove the key the line names from ops/ansible/group_vars/<site>_hosts/
list the node under [class_phone] in ops/ansible/inventory/mgmt-hosts.ini, as the name alone
docs/design/phone-node.md 1
GB-297 goatbuilder a joining node's clock is not synchronized or is too far from UTC — join-node phase 3 read the joining node's clock from its own time service (chronyc -c tracking, else timedatectl), before anything on the node changed. A Zenoh router refuses a timestamp more than 500 ms ahead of its own clock, and nothing tells the machine's owner, so the node must hold its clock within half that, 250 ms by default (JOIN_NODE_CLOCK_BUDGET_MS), of UTC. A node that is not synchronized to an outside reference, or is over the budget, is refused; a node whose clock could not be read is reported as not verified, never as passed. The rule is the federation preflight's check clock (GB-268). the node's time service follows no reachable time source, or only the node's own local clock
the clock is synchronized but still slewing a large step, or the time source itself is wrong
on the named node: chronyc tracking (or: timedatectl show); repair its time synchronization
goatbuilder join-node <env> --site <site> --node <node> --class <class> --address <address> --from 3
docs/design/federation-preflight.md 1
GB-304 goatbuilder grow was given a file that is not a goatlite — goatbuilder grow hands a goatlite over to the full preset, so it reads goatlite.yaml at the checkout root and needs it to declare preset: lite with a lite: block (ADR 1213). This file declares another preset, or has no lite: block, so there is nothing to grow. A checkout already on the full preset needs no grow. grow changes files in the checkout only, never a host: a finished grow is undone with git before it is committed (restore goatlite.yaml and the files whose generated header it took off, and delete ops/goatbuilder/goatbuilder.yaml), and goatbuilder grow --dry-run shows every write first. goatlite.yaml declares preset: full, or no preset
the lite: block is missing or is not a mapping
goatbuilder validate goatlite.yaml (it says what the file declares)
a full configuration is ops/goatbuilder/goatbuilder.yaml: goatbuilder validate
docs/design/goatlite-growth.md 2
GB-305 goatbuilder grow would replace a configuration that already exists — goatbuilder grow writes the full configuration to ops/goatbuilder/goatbuilder.yaml. That file already exists and holds another configuration (not the one this goatlite grows into, which a run that stopped part way would have left), so grow refused before changing anything: it never replaces a configuration someone wrote. When the file carries grow's own header, the message says it is grow's file from an earlier run, since edited (or goatlite.yaml changed). grow changes files in the checkout only, never a host: a finished grow is undone with git before it is committed (restore goatlite.yaml and the files whose generated header it took off, and delete ops/goatbuilder/goatbuilder.yaml), and goatbuilder grow --dry-run shows every write first. the checkout already declares a full configuration of its own
the file was edited after an earlier grow wrote it, and goatlite.yaml is still there
move ops/goatbuilder/goatbuilder.yaml aside, then goatbuilder grow --dry-run
or merge what it declares into goatlite.yaml, then grow again
grow's own file from an earlier run, edited: merge the edits into goatlite.yaml and delete ops/goatbuilder/goatbuilder.yaml, or delete it to grow afresh
docs/design/goatlite-growth.md 1
GB-306 goatbuilder the full configuration grown from a goatlite is incomplete — goatbuilder grow rewrote goatlite.yaml as a full configuration, and the result fails the schema or a cross-field rule, so it was not written. Most often the goatlite is the four-fact file of the tutorial: the lite preset took repo and components from the checkout, and a full configuration declares its own. The message lists each problem and names the fields to add. grow changes files in the checkout only, never a host: a finished grow is undone with git before it is committed (restore goatlite.yaml and the files whose generated header it took off, and delete ops/goatbuilder/goatbuilder.yaml), and goatbuilder grow --dry-run shows every write first. goatlite.yaml declares no repo or components, and the checkout has no configuration to take them from
a value in goatlite.yaml is valid for the lite preset but not in the full shape (the message names it)
add repo: and components: to goatlite.yaml (the fixture ops/goatbuilder/fixtures/goatnet-from-scratch/00-goatlite shows both), then goatbuilder grow --dry-run docs/design/goatlite-growth.md 1
GB-307 goatbuilder an environment declares converger pull-agent, and the pull converger is not built — goatbuilder.yaml declares converger: pull-agent for this environment, one of its sites or one of their classes. No pull converger is built: no agent on a machine fetches and applies its class's release. The only converger on real hosts is push, and pushing to machines declared to pull would change what the declaration means once pull is built. So doctor refuses the environment, and deploy (its dry run too), release's deploy rungs, fastlane's deploy and rollback refuse it before any step. deploy --verify-only reads and is not refused; validate accepts the value, and --mock converges it on a fixture. the environment, a site or a class was declared pull-agent ahead of the pull converger declare converger: push-ansible for these machines in ops/goatbuilder/goatbuilder.yaml until the pull converger is built
goatbuilder doctor --env <environment> (the env:<name>:converger row names each place pull-agent is declared)
docs/design/goatbuilder-pull-converger-options.md 2
GB-320 goatbuilder a program the run starts could not be started — A goatbuilder verb runs its phases and helpers as programs: bash for a phase's command, python3 for a step written in Python (the preflight, the slot recheck), and the helpers a step names (the core's candidate planner). The operating system refused to start one: it is not installed, not on PATH, or not executable. Nothing of that step ran. bash or python3 is not installed on the machine goatbuilder runs on, or not on PATH
a helper the step names is missing from the checkout, or has lost its executable bit
command -v bash python3 (install what is missing, then run the same command again)
git -C <checkout> status (restore a helper the checkout is missing)
docs/design/goatbuilder.md 1
GB-321 goatbuilder a file a step of this run wrote is missing or is not what it wrote — A step of a federate run leaves its result in the working folder for the next one to read: phase 1's preflight result and the slot recheck's, the agreement, a back-out host record, the rotation id, an edge result on a two-driver peering. The file the next step read was not there, was not JSON, or did not hold what that step writes (another peering's name, a malformed id). Nothing after it ran. the step that writes it stopped before writing it (its own failure is above, or in its log)
the file was edited, replaced or removed by hand between the steps
a different peering's working folder was named (--work, FED_WORK)
run the step that writes it again (the message names the file; its phase is the one before)
remove a file the message says is malformed, then run the same command again
docs/design/goatbuilder.md 1
GB-322 goatbuilder goatbuilder reached a state its own code rules out — goatbuilder checked one of its own assumptions and found it broken: a plan whose shape the code that builds it no longer matches, a verb dispatched to a place that never takes it, a rotation marked built with no run. No input of yours can cause this; it is a defect in this build. Nothing after the check ran. a defect in this goatbuilder build report it with the run record the run printed (goatbuilder runs show <id>) and the exact command docs/design/goatbuilder.md 1
GB-323 goatbuilder a file the run writes outside the working folder cannot be written — Some steps write outside the working folder: the back-out deletes the partner record it generated from the checkout, and init --side edge writes the edge half to the file --half-out names. That file could not be written or removed. the checkout or the folder --half-out names is not writable by this user
the disk is full, or the folder is on a read-only mount
check the folder the message names is writable, then run the same command again docs/design/goatbuilder.md 1
GB-324 goatbuilder a setting in the environment has a value goatbuilder does not take — A goatbuilder setting read from the environment (a FED_* variable) has a value outside the ones it takes; the message names the variable and the values it takes. Nothing ran. the variable was set by hand, or by an older script, to a value this build does not know unset the variable, or set it to one of the values the message names docs/design/goatbuilder.md 2
GB-325 goatbuilder flags given to a federate stage or verb do not go together, or one they need is missing — A federate stage was given flags it cannot take together (--accept with --recheck, --recheck with --mock or --no-live, --side without the files that side reads or writes), a flag without the one it needs (--work for status without the partner, init --over-ssh without -C, a host flag such as --edge-host or --side on init without --over-ssh), or a flag naming something the saved plan does not hold (--accept ), or a value the saved plan was made with differently (--edge-group or --as on a recheck). Nothing ran. a command line put together from two different runs' flags run the stage with the flags the message names, apart where it says to run them apart
goatbuilder federate <stage or verb> --help
docs/design/goatbuilder.md 2
GB-315 goatbuilder a program a step ran gave output or an exit goatbuilder cannot use — A step of a federate run started a program and reads its answer: the core's candidate planner prints a plan on stdout, and phase 1's preflight exits and writes a result whose ready verdict must agree with its exit. The program ran, but what it gave cannot be used: no plan on stdout, or an exit and a result that disagree or cannot be read. Its own words are in the message. Nothing after it ran. the program failed before it gave its answer (its stderr is in the message)
the program is a different version from the one this goatbuilder expects
read the program's own words in the message, fix what it names, then run the same command again
git -C <checkout> status (a helper the checkout changed or is missing)
docs/design/goatbuilder.md 1
GB-316 goatbuilder a confirmation could not be read from the terminal — A federate stage asked a person to type a confirmation (the back-out asks for the partner id) and reading the answer from the terminal failed. Nothing was changed. the terminal closed, or stdin is not readable run the same command again at a terminal
pass --yes where the stage takes it, for a run without a terminal
docs/design/goatbuilder.md 1
GB-278 goatbuilder the edge driver's checkout or carried files are not the controller's — On a two-driver peering (drivers: two), the edge driver runs a phase's edge half with the command the controller's hand-off names: goatbuilder federate <peering> --side edge --from <N> --to <N> --expect-commit <commit>, with one --expect <file>=<sha256> per file the controller carried. The edge driver refused to run: --expect-commit was not given; its checkout is not a git checkout, has no commit, has changes, or is at another commit than the one named; or a carried file is not under its or in its checkout, or has another sha256. Nothing ran on the edge. the edge driver's checkout was not updated to the controller's commit, or has local changes
the command was typed without the hand-off's --expect-commit
a carried file was not copied from the controller's , or was changed on the way
git -C <checkout> fetch && git -C <checkout> checkout <commit> (the commit the hand-off names), and commit or remove every changed file
copy each carried file from the controller's <work> to the same path under the edge driver's <work>, then run the hand-off's command again
docs/design/federation-two-drivers.md 1
GB-289 goatbuilder the IPsec rotation's staged material is missing or unreadable — A credential rotation of an ipsec-fips peering (goat-peering-edge with peering_edge_rotation set) switches the edge to a new leaf that the offline ceremony issued and an operator staged beside the live files: -cert.next.pem in the swanctl x509 directory, -key.next.pem in its private directory at mode 0600, and for an ipsec-ca rotation our new CA as -local-ca.next.pem in x509ca. The rotation also needs our current IPsec CA certificate at peering_edge_local_ca_path to check the new leaf's chain. One of them is missing, unreadable, or the key is not mode 0600. The role refused before anything moved. the new leaf, its key, or (for ipsec-ca) our new CA was not staged on this edge
peering_edge_local_ca_path is not set, or names no file on the edge
the staged key was copied without mode 0600
stage what the ceremony issued at the paths the message names, with the key at mode 0600, then run the rotation again
set peering_edge_local_ca_path in the edge's group_vars to our current per-peering IPsec CA certificate
docs/design/federation-key-rotation.md 2
GB-290 goatbuilder the staged IPsec certificate is not right for this link — Before an IPsec credential rotation moves anything, goat-peering-edge checks the staged certificate: its key is the staged key, it chains to the CA it must chain to (our current CA for ipsec-leaf, the staged new CA for ipsec-ca), its subject CN is peering_edge_local_id, its key is ECDSA P-384 (secp384r1) as the proposals require, it is valid now, and it stays valid for peering_edge_min_valid_days more days (default 30). The message names each check that failed. Nothing was moved. the leaf was issued for another peering or another identity
the leaf was issued by another CA, or the staged key belongs to another certificate
the leaf uses another key type, or expires too soon
have the ceremony issue the leaf again with Subject CN = peering_edge_local_id and an ECDSA P-384 key, from the right CA, then stage it and run the rotation again docs/design/federation-key-rotation.md 2
GB-302 goatbuilder the IPsec link did not come up on the new leaf; the old pair was put back — After switching to the staged leaf, goat-peering-edge loaded the credentials, re-authenticated the IKE SA and waited up to peering_edge_rotation_confirm_seconds (the DPD timeout by default) for an established IKE SA with an installed CHILD SA. None came up, so it put the set-aside pair back from .previous-/, loaded it and re-authenticated: the old leaf is live. It marks the rotation rolled back (.previous-/rolled-back), so every later run of the same id, a resume or a retire, reports GB-302 again, moves nothing and never replaces our CA. For ipsec-ca with --no-overlap, a partner that no longer trusts our old CA cannot carry the link on the old pair either. the partner does not trust the CA that issued the new leaf
the partner's edge expects another identity, or is not reachable
this rotation id was rolled back by an earlier run
read swanctl --list-sas and the charon log on both edges, then stage the new pair again and start a rotation with a new id docs/design/federation-key-rotation.md 2
GB-303 goatbuilder the partner has not read back our new IPsec CA — An ipsec-ca rotation switches the edge to a leaf issued by our new per-peering CA only once the partner trusts that CA. Nothing on our side can read the partner's trust store, so the switch needs the sha256 fingerprint the partner read back after placing our new CA (peering_edge_rotation_ca_readback, or goatbuilder federate rotate ipsec-ca --partner-holds-ipsec-ca ). It was not given, or it is not the staged CA's fingerprint. Nothing was moved. the partner has not placed our new CA yet, or has not read its fingerprint back on the call
the fingerprint read back belongs to another file
have the partner place our new CA beside their current one and read back its sha256 fingerprint, then run the rotation again with that fingerprint docs/design/federation-key-rotation.md 2
GB-298 goatbuilder an IPsec rotation under way, or not yet retired, is not the one asked for — goatbuilder federate rotate ipsec-leaf|ipsec-ca records the rotation it starts in /.rotation-ipsec (its id, its kind and, for ipsec-ca, the partner's attestation: the fingerprint it read back of our new CA) before phase 11 runs, and removes the record once phase 11 passed. Each rotation is also kept in /ipsec-rotations.json, with when it switched, passed phase 14 and was retired. A run that finds a record resumes that rotation with the same id, so goat-peering-edge sets nothing aside twice. This run was refused before anything ran: it asked for another kind or another read-back than the record names (fingerprints are compared in one form, sha256: and lowercase), the record is empty or not a rotation record, a retire was asked while a rotation is under way or of the other kind than the rotation retired, or a new rotation was asked while an earlier one switched and is not retired. an earlier rotation stopped before phase 11 passed, and a rotation of the other kind was asked for
the partner's read-back changed between the run that started the rotation and this one
an earlier rotation switched and its retire step has not run
the record was emptied or edited by hand
finish the rotation the record names with the command that started it (the message names its id and kind)
retire the earlier rotation once phase 14 passed here after the switch (and the partner says theirs did): goatbuilder federate rotate <partner> ipsec-leaf\|ipsec-ca --retire <id>
removing <work>/.rotation-ipsec starts a new rotation id; the edge keeps /etc/swanctl/.previous-<id>/ as it is (after a GB-302 rollback, with the old pair live), and nothing removes it yet
docs/design/federation-key-rotation.md 1
GB-309 goatbuilder the retire step finds no confirmed rotation to retire — The retire step of an IPsec credential rotation (goat-peering-edge, peering_edge_rotation_retire) removes the set-aside key in .previous-/ on the edge, and nothing else: it never moves the live pair, the staged pair or our CA. It removes the key only once the new leaf is live: the confirm recorded it in .previous-/confirmed (the unique id of the IKE SA the re-authentication made, and the time), the live pair is not the set-aside one, the live key is the live certificate's, and that SA, or one established after the confirm's time, has an installed CHILD SA. The confirm is recorded only once the SAs from before the re-authentication are gone, and a roll-back removes the record. This edge has no .previous-/ for the id given, or the rotation is not confirmed live, so nothing was moved or removed. the id is mistyped, or names a rotation of another edge
the rotation stopped before its switch, so the live pair is still the set-aside one
the rotation was not confirmed: it stopped after the switch and before its confirm wrote .previous-/confirmed
the confirm's record is unreadable: it lacks the id= or at= line the confirm writes
neither the IKE SA the confirm recorded nor one established since is up, or the live key is not the live certificate's
goatbuilder federate rotate --retire: this side's ledger (/ipsec-rotations.json) shows no such rotation whose phase 11 switched and whose phase 14 passed here after the switch, or it is already retired
check the id the rotation printed (ls /etc/swanctl/.previous-*, or <work>/ipsec-rotations.json)
run the rotation again with the same id: it re-authenticates, confirms and writes .previous-<id>/confirmed; retire once phase 14 passed here after the switch, and the partner says theirs did
docs/design/federation-key-rotation.md 2
GB-311 goatbuilder an IPsec rotation stopped half way through its switch, and what is left does not finish it — The switch of an IPsec credential rotation (goat-peering-edge) moves the staged key, then the staged certificate, over the live names. A run killed between the two moves leaves one moved in and the other still staged. The next run finishes the remaining move when what is left is the moved one's match: the moved file is not the set-aside one, and the two halves are one key pair (each parsed by openssl; a file that does not parse matches nothing). Here they are not, so the live pair would be mixed: nothing was moved. the staged file left behind was replaced after the run stopped
someone moved or edited the live or set-aside files by hand
restore the staged file the ceremony issued for this rotation, then run the rotation again with the same id
or put the set-aside pair in /etc/swanctl/.previous-<id>/ back by hand, check it with openssl, and stage the new pair again
docs/design/federation-key-rotation.md 2
GB-328 goatbuilder a rotation's values reach no play in the phases this run built — A key rotation of goatbuilder federate rotate passes its values (the rotation id, and for an IPsec rotation its kind and read-back) to the plays of one phase: phase 11 for ipsec-leaf and ipsec-ca, phase 2 for the peering key. It puts them before each play's -e fed_work=. The phase this run built has no such play, so the values would reach nothing and the phase would run as a plain one and report the rotation done. Nothing ran. The phases are built from the agreement each run, and the rotation already checked that the agreement chose its transport, so this is goatbuilder's own fault, not the peering's. the phase's command in this goatbuilder no longer runs its play with -e fed_work= goatbuilder federate rotate <partner> <key> --dry-run (it prints the phase's command)
report the command and this goatbuilder's version (goatbuilder --version); do not run the phase by hand in its place
docs/design/federation-key-rotation.md 1
GB-351 goatbuilder a Spock subscription has forward_origins set, which in a mesh replicates in a loop — A Spock subscription, or the setting that creates them, has forward_origins other than empty or 'none'. In a mesh of three or more sites such a subscription passes each change on to the next site, which passes it back, without end, while every subscription still reads replicating and nothing is logged. The spock-forward-origins-check role stopped the run before spock-mesh-peering or ory-narrow-repset changed anything. The message names the setting, or each subscription as .. a site's group_vars set spock_mesh_forward_origins or ory_spock_forward_origins
a subscription was created by hand with forward_origins := '{all}'
remove spock_mesh_forward_origins and ory_spock_forward_origins from the site's group_vars (both default to [])
on the host and database named, run SELECT spock.sub_drop('<subscription>'); then run the playbook again, which creates it with forward_origins empty
docs/design/spock-mesh-peering.md 2

tools goat and goatbuilder call

Code Tool What it means Why What to do Read Exit
X-ANSIBLE-001 ansible a host was unreachable — ansible could not open a connection to the host over the transport it was given (ssh over the mesh, or the door). this laptop is not on the goatnet's mesh profile and no door was opened
the host is down
the SSH key is not accepted (see the ssh entries)
goatbuilder doctor --env <env> --all (the transport row)
ssh -o ConnectTimeout=5 <ssh user>@<mesh ip> true
docs/design/env-doors.md
X-ANSIBLE-002 ansible sudo on the host wants a password — a task needed privilege escalation and the host's operator user is not NOPASSWD. the host was not baselined (mgmt-host-baseline.yml sets NOPASSWD sudo)
the wrong remote user
run ops/ansible/mgmt-host-baseline.yml against the host once docs/operations/goatnet-standup.md
X-ANSIBLE-003 ansible a play assertion failed on the host — a task asserted a precondition, usually that a templated secret is non-empty, and it was false on the host. a secret the play templates was empty in the deployer's environment (see GB-007)
a group_vars fact missing for this goatnet
goatbuilder deploy --env <env> --component <c> --dry-run (names every variable the play needs) docs/operations/goatnet-standup.md
X-ANSIBLE-004 ansible a task failed on a host — ansible marks the task fatal; its msg field says what the module could not do. see the msg in the fatal line grep -n -A3 'fatal:' <log> docs/operations/goatnet-standup.md
X-ANSIBLE-005 ansible-preflight the preflight refused a stale or unsafe checkout — ops/scripts/ansible-preflight.sh refuses to run a play from a tree that differs from origin/main in a safety-critical path, or from a stale trunk; ansible reads role defaults from the working tree, so a stale tree templates stale pins. local main behind origin
a feature branch that touches group_vars or roles
git pull --ff-only origin main (the message prints the exact command) docs/operations/goatnet-standup.md
X-SSH-001 ssh the host refused the key — the connection reached the host and SSH rejected every key offered. your public key is not on the host's operator user (ssh-key-self-service)
the wrong user (hosts take the deploy.ssh_user account)
the key is not loaded in the agent
ssh -v <ssh user>@<host> true 2>&1 \| grep -i 'offering\\|denied'
goat admin ssh-key … (register the key through the goatnet)
docs/design/ssh-key-self-service.md
X-SSH-002 ssh no route to the host — the connection never reached the host: timed out, no route, or refused on port 22. the address is on a network this laptop is not on (a mesh IP without the mesh, a Tailscale IP without the tailnet)
the host is down
netbird status \| head -5
goatbuilder doctor --env <env>
docs/design/env-doors.md
X-SSH-003 ssh the host key changed — the host presented a different key than ~/.ssh/known_hosts holds for that address. the host was rebuilt (expected after a re-image)
an address was reused by another machine
confirm the rebuild with the operator, then: ssh-keygen -R <address> docs/operations/site-node-troubleshooting.md
X-OP-001 op 1Password CLI is not signed in — op needs a session for the account before it can resolve references. no op signin in this shell
OP_ACCOUNT names a different account than the one signed in
eval $(op signin) (OP_ACCOUNT=<account> for the team store)
goatbuilder doctor --credentials --all (the tool:op-session row)
docs/design/secret-store-requirements.md
X-OP-002 op an op:// reference is malformed — op run validates every op:// value in the environment and refuses the whole run when one has fewer than three parts (vault/item/field). The variable is often one from your shell profile, not the refs file. a variable in your shell holds a two-part reference (op://Vault/Item) env \| grep 'op://' (find the variable)
env -u <THAT_VAR> op run --env-file=… -- <command>
or complete it to op://<vault>/<item>/<field>
docs/design/secret-store-requirements.md
X-OP-003 op a reference does not resolve to an item — the vault, item or field named in an op:// reference does not exist, or this account cannot see it. the item was renamed or moved to another vault
the account lacks access to the vault
op item get '<item>' --vault <vault>
check refs/goat.refs in <owner>/goat-credentials for the current line
docs/design/secret-store-requirements.md
X-AWS-001 aws AWS rejected the credentials — the access key is unknown to AWS, or a session token was sent where none is accepted. aws-vault run without --no-session (STS tokens are rejected by this account)
a rotated key still in the keychain
aws-vault exec <profile> --no-session -- aws sts get-caller-identity
aws-vault add <profile> (re-enter the current key)
docs/design/cloud-infrastructure-discipline.md
X-AWS-002 aws-vault aws-vault has no credentials for the profile — the keychain holds no credentials under the profile name, or the keychain could not be opened (headless). a fresh machine
a headless session cannot prompt the keychain
aws-vault add <profile>
aws-vault list
docs/design/cloud-infrastructure-discipline.md
X-AWS-003 aws the SSM target is not connected — Session Manager has no live agent for the instance, so the door cannot open. the host is down or has no route to the SSM endpoint
the instance id in doors.yaml is stale
aws-vault exec <profile> --no-session -- aws ssm describe-instance-information --region <region> docs/design/env-doors.md
X-AWS-004 aws the AWS CLI fell back to an expired aws login session — the aws-vault credential for the profile is empty or malformed, so the CLI used an expired aws login session instead and asked to reauthenticate. the aws-vault credential is empty or malformed, so the CLI fell back to an expired aws login session
AWS_VAULT_KEYCHAIN_NAME in the environment (a direnv export) points aws-vault at the wrong keychain
aws login (when the error names it)
env -u AWS_VAULT_KEYCHAIN_NAME aws-vault exec <profile> --no-session -- aws sts get-caller-identity (when the keychain name is set)
aws-vault add <profile> (when the key really is bad; it replaces the stored key without deleting it first)
docs/design/cloud-infrastructure-discipline.md
X-GIT-001 git not a git repository — the working directory is not inside a checkout. run from the wrong directory cd /path/to/goat (or pass -C) CONTRIBUTING.md
X-GIT-002 git the branch cannot fast-forward — local and remote histories diverged, so a --ff-only pull refuses. local commits on main git log --oneline origin/main..HEAD (what is local); move it to a branch, then git pull --ff-only CONTRIBUTING.md
X-GIT-003 git the remote refused access — the push or fetch was refused: the SSH key is not authorised for the repository, or the remote does not exist. the wrong GitHub account for this remote
the key is not loaded
ssh -T git@github.com
gh auth status
CONTRIBUTING.md
X-GH-001 gh GitHub returned 404 — the repository, run, release or API path does not exist, or this account cannot see it. a typo in the slug or id
a private resource the account is not a member of
gh auth status
gh repo view <owner>/<repo>
CONTRIBUTING.md
X-GH-002 gh the GitHub CLI is not logged in — gh has no token for github.com in this environment. a fresh machine
GH_TOKEN set to an expired token
gh auth login CONTRIBUTING.md
X-GH-003 gh GitHub rate-limited or forbade the call — HTTP 403 from the API: a rate limit, or a token without the scope. many unauthenticated calls in a short time
a token missing read:packages or repo
gh api rate_limit
gh auth refresh -s read:packages
CONTRIBUTING.md
X-COSIGN-001 cosign the signature did not verify — cosign found no signature for the artifact under the expected identity, or the signature does not match. the artifact was rebuilt without signing
the wrong public key or identity
cosign verify --key <org key> <image@digest> docs/reference/goatbuilder/02-install.md
X-DOCKER-001 docker no permission on the Docker socket — the user is not in the docker group on that host, so every docker call is refused. running docker without sudo on a host where the operator user is not in the docker group sudo docker … (hosts run docker as root by baseline) docs/operations/site-node-troubleshooting.md
X-DOCKER-002 docker the image could not be pulled — the registry answered that the manifest is unknown or the pull is unauthorised. a tag that was never pushed, or a digest from the other GHCR org
the host's registry login has expired
crane digest <repo.image_namespace>/<image>:<tag>
on the host: sudo docker pull <ref>
docs/design/base-images.md
X-DOCKER-003 docker the Docker daemon is not running — the socket exists but nothing answers, or does not exist at all. docker.service stopped on the host
a local Docker Desktop that is not launched (never launch it for goat work)
on the host: sudo systemctl status docker docs/operations/site-node-troubleshooting.md
X-NETBIRD-001 netbird no such mesh profile — the profile named does not exist on this laptop; profiles are per goatnet. the profile was never added
the hyphen/underscore form differs from the profile name
netbird profile list docs/design/env-doors.md
X-NETBIRD-002 netbird the mesh daemon is not running — the netbird client could not reach its own daemon. the service is stopped sudo netbird service start (or restart the desktop client) docs/operations/battle-drills/02-mesh-client.md
X-HTTP-001 http the service rejected the credential (401) — the request reached the service and the token or cookie was not accepted. an expired or wrong token check which token the command is using; for goat, E_TOKEN_INVALID says how to rotate it docs/design/goat-cli-error-codes.md
X-HTTP-002 http the service refused the identity (403) — the credential was accepted and the identity is not allowed to do this. the wrong one of two credentials (participant vs operator)
no grant for this surface
for goat admin verbs: GOAT_PORTAL_TOKEN=op_… goat … docs/design/identity-access-and-onboarding-reference.md
X-HTTP-003 http the service is up but its backend is not (5xx) — the front door answered with a gateway error: Traefik or Caddy is up and the service behind it is not. the container is restarting or down on the host: sudo docker ps --format '{{.Names}} {{.Status}}' \| grep -v healthy docs/operations/site-node-troubleshooting.md
X-HTTP-004 http an onboarding call went to the portal — goat sent an onboarding call (invite, approve) to the portal instead of the onboarding endpoint: the portal answers 405 Method Not Allowed, or the call names a host that is only portal. the context's profile has no onboard_endpoint, so onboarding calls went to the portal GOAT_ONBOARD_ENDPOINT=<onboard url> goat … (for this run)
goat --context <name> profile init <bundle.cbor> from a current bundle (the profile then carries onboard_endpoint); goatbuilder doctor's tool:goat-contexts row names the contexts
docs/design/three-stage-candidate-onboarding.md
X-DIRENV-001 direnv direnv exec was given an assignment, not a command — direnv exec <dir> VAR=value cmd looks VAR=value up on PATH as the command and fails; direnv exec takes a command. an environment assignment placed where direnv exec expects the command direnv exec . env VAR=value cmd docs/operations/admin-operator-guide.md
X-DNS-001 dns the name did not resolve — the resolver has no answer for the name; mesh names resolve only while the mesh is up. off the mesh (mesh names are served by mesh CoreDNS)
a typo in the name
netbird status \| head -5
dig +short <name> (then the same against the mesh resolver the profile names: goat names)
docs/design/mesh-dns-redundancy.md
X-TLS-001 tls the TLS handshake failed — the certificate presented was not trusted, did not match, or the handshake ended early. a CA this machine does not trust (goat ca add)
a clock far enough off that the certificate reads as invalid
goat ca ls
date -u (compare with a known-good clock)
docs/design/internal-tls-ca.md
X-SMOKE-001 onboard-intake-smoke the live rows' browser toolchain is not on this machine — the live acceptance rows (intake-form-browser-submit, review-gate-accepts-open-field-invite) run ops/scripts/onboard-intake-smoke.sh from the deployer's laptop, and it exits 2 before touching the goatnet when ops/goat-id-console/web/e2e has no node_modules. The release did not fail on the goatnet; it failed here. a laptop that never installed the console's Playwright suite (cd ops/goat-id-console/web/e2e && npm ci && npx playwright install chromium)
goatbuilder doctor --browser-tests (the tool:live-rows row fails until this machine is ready)
docs/design/release-candidate-validation.md
X-PY-001 python a Python module is missing — python3 could not import a module a script needs. a fresh machine or a venv not on PATH pip install <module> (pyyaml, jsonschema are the usual two)
a script Ansible runs on the control node uses ansible_playbook_python, which carries PyYAML; a task that still calls bare python3 should too
docs/design/goatbuilder.md
X-SECRETS-001 with-secrets the declared secrets store is unknown — GOAT_SECRET_STORE or secrets: store: in goatbuilder.yaml names a store goat does not have, so no command can get its secrets. a typo in GOAT_SECRET_STORE or in goatbuilder.yaml
a custom store whose ops/secrets//run is not on this checkout
goatbuilder secrets store (prints the store in force and where it came from)
use one of: 1password, openbao, custom, sops, git-crypt, env-file, env
docs/design/pluggable-dependencies.md
X-SECRETS-002 with-secrets the env-file store has no file — the declared store is env-file, and its file of values does not exist on this laptop. the quickstart's env file was never generated
a goatlite whose phase 1 has not run yet (phase 1 renders the goatnet, then creates this file)
GOAT_SECRET_ENV_FILE points somewhere else
on a goatlite: goatbuilder standup <name> --to 1 (renders the goatnet, then creates the file)
on any other goatnet: goatbuilder secrets init --env <goatnet> (strong random values, 0600, gitignored)
docs/design/pluggable-dependencies.md
X-SECRETS-003 goatbuilder refusing to write secret values to a file git would commit — secrets init writes plaintext values, so it refuses a path that is not gitignored. --file names a path inside the repository that .gitignore does not cover add the path to .gitignore, or use the default .envrc.local.d/secrets.env docs/design/pluggable-dependencies.md
X-SECRETS-004 with-secrets bao:// references need the OpenBao injector — the environment holds bao:// references, and ops/secrets/openbao/run (the OpenBao store's injector, ADR 1203) is not on this checkout. a .refs file holds bao:// lines on a checkout that predates the OpenBao store git pull --ff-only origin main (the injector lands with the OpenBao store)
or remove the bao:// lines and use op:// references meanwhile
docs/design/pluggable-dependencies.md
X-SECRETS-005 with-secrets a git-crypt file is still encrypted — the declared store is git-crypt and its file has not been unlocked on this laptop, so its values cannot be read. this clone was never unlocked git-crypt unlock docs/design/pluggable-dependencies.md
X-ACCEPT-001 acceptance.sh acceptance could not run a required check, so the peering is not accepted — ops/federation/standup/acceptance.sh (phase 14) found no failure, but a required check was BLOCKED (its input or evidence was unavailable) or SKIPPED (not exercised). Such a check proves nothing, so the verdict in /evidence/acceptance/result.json is INCOMPLETE and the run exits 3, not 0. a surface the check reads is not configured (for example the partner's REST endpoint or the SPIRE servers)
the agreement offers nothing in one direction, so that direction's checks had nothing to send
a probe session the check needs was not started
read the BLOCKED and skipped lines above the acceptance-counts line, or the why field of result.json; set what each names, then run phase 14 again
goatbuilder federate <peering> --from 14
docs/design/federation-acceptance.md
X-DIGEST-001 image-digest no tool could read the image's digest — ops/scripts/image-digest tried crane, skopeo and docker in turn; none is installed, or each refused, and each tool's own error is printed beneath. none of crane, skopeo or docker is installed
the registry refused this laptop's credentials (each tool's error says which)
the tag does not exist yet (its image build has not finished)
brew install crane
gh auth token \| crane auth login ghcr.io -u <user> --password-stdin
docs/design/pluggable-dependencies.md