Skip to content

03 — Lab phase plan (CONUS)

This document walks you through standing up GOAT-Net at home station, week by week, from "decision made" through "operators trained, ready for Phase 2 planning."

Target elapsed time: ~6 weeks with weekly concierge sync. Some weeks compress; some expand. Hardware lead time is the most likely scheduling driver.

Pre-conditions before week 1

You can start week 1 once these are in place. Most of them are work you'd already do for any platform stand-up:

  • Procurement order placed for Phase 1 hardware (02-procurement.md)
  • AWS account (or equivalent cloud) provisioned
  • Domain registered (<your-unit>.<tld> or sub-delegation)
  • Cloudflare account (or equivalent DNS provider)
  • Lab space allocated (rack, power, cooling, network drop)
  • LAN segment defined (CIDR, gateway, internet uplink)
  • Technical lead identified, has 0.5 FTE for 6 weeks
  • 1-2 operators identified, available weeks 5-6 for training
  • AO informed; not yet engaged in formal review
  • Concierge engagement opened (see 06-asks-and-commitments.md)

If you're not sure about the network or LAN segment, that's fine — week 1 includes a 1-on-1 design call where we work through the per-site shape with you.

Week 1 — Design + dependencies

Goal: All decisions made, all dependencies ordered, all reading done.

Activity Lead Outcome
Design call — per-site shape (3-Orin vs single-Orin lab? FIPS at lab y/n? cloud provider?) Tech lead + us Decisions captured in your engagement notes
Network design — LAN CIDR, switch model, VRRP plan, transport Tech lead LAN diagram, IP plan
Cloud account hygiene — IAM, MFA, billing alerts Tech lead Account ready for Terraform
Domain DNS configured Tech lead NS records pointing at Cloudflare or chosen provider
Hardware shipped (procurement ordered week 0) Procurement Tracking numbers
Read 00-executive-overview.md + 01-what-you-get.md end-to-end Tech lead, AO Shared mental model
Skim relevant ADRs we send you Tech lead Familiarity with substrate decisions

Concierge load: ~4-6 hours of sync calls + ~10-15 hours of async questions.

Stuck-points typical at week 1: cloud-provider choice (commercial vs GovCloud), TAA constraints surfacing late, network gear that doesn't support VRRP. We work through these together.

Week 2 — Hardware receiving + bench setup

Goal: Hardware on the bench, OS installed, network reachable.

Activity Lead Outcome
Hardware received and inventoried Operator Asset tags, serial numbers recorded
Initial OS install — Ubuntu 22.04 LTS Tegra (Orin) or 24.04 LTS (x86) Operator Hosts boot to Linux
LAN connected, switch configured, VLANs (if any) defined Operator Hosts ping each other on LAN
Initial passwords + SSH key setup Operator Operator can SSH into all hosts
Cloud account: AWS CLI configured, aws-vault profile set up Tech lead aws sts get-caller-identity works
Terraform installed locally on tech lead's workstation Tech lead terraform version works
Ansible installed locally on tech lead's workstation Tech lead ansible --version works
Repository cloned (we provide the URL — sanitization-clean version) Tech lead Local checkout
Direnv + secret-management baseline (.envrc.local template) Tech lead Secrets pattern in place; values to fill in week 3

Concierge load: ~2-3 hours of sync; ~5-10 hours of async.

Stuck-points typical at week 2: Tegra OS quirks (we have a runbook); BIOS/firmware updates that take longer than expected; disk geometry surprises (LUKS partitioning).

Week 3 — Offline CA + relay tier

Goal: Your offline CA exists, your relays are deployed.

Activity Lead Outcome
YubiKey-PIV ceremony — generate offline CA root inside YubiKey slot 9c Tech lead CA root exists; private key never touches host disk
Backup YubiKey ceremony (per the same protocol) Tech lead 1-of-N backup stored offsite or in safe
ca-init --mode prod run; root cert published Tech lead Public root cert at known path
Relay tier Terraform applied Tech lead 3 relay VPSes running in 3 regions
Relay tier Ansible applied (haproxy ingress dispatch + rathole tunnels + STUN + relay binary) Tech lead Relays operational; DNS round-robin resolves
Relay smoke test — TLS handshake succeeds from external workstation Tech lead Relays reachable
wg-cp0 outer wrap deployed on each relay Tech lead Silent control plane substrate in place

Concierge load: ~3-4 hours of sync (the YubiKey ceremony is sensitive; we walk through it live).

Stuck-points typical at week 3: YubiKey firmware version (need 5.7+ for Ed25519 PIV); cloud-region TAA restrictions; cloud-provider quota limits on instance types.

Week 4 — First site bringup

Goal: Your three mgmt-hosts come up running GOAT-Net. Or, if you chose single-Orin lab, your one mgmt-host comes up.

Activity Lead Outcome
LUKS-encrypt NVMe drives on each mgmt-host Operator Data partitions encrypted
FIDO2 + LUKS keyslot enrollment (multiple modes per Block 65) Operator Auto-unseal keypath in place
seed-ssh-key.sh for each mgmt-host Tech lead Operator pubkey on each host
mgmt-host-baseline.yml Ansible play Tech lead OS baselined, hostname set, reboot if needed
mgmt-host-luks.yml Ansible play Tech lead LUKS volumes mounted at known paths
Bootstrap bundle generation — first peer's bundle Tech lead Signed CBOR bundle exists
site-bootstrap.yml -e site_id=<your-site> Ansible play (first run) Tech lead Mgmt-plane stack up: Postgres + signal-server + netbird-mgmt + Caddy/Traefik + identity stack
First peer enrollment via bundle Operator First peer (your tech lead's laptop, probably) is on the mesh
Smoke tests — mesh reachable, services healthy, observability dashboards green Operator Reference observability board shows green
Optional: 4th host as obs-host C (full three-Orin) Operator Observability + storage + utility tier

Concierge load: ~8-12 hours of sync (week 4 is the densest week).

Stuck-points typical at week 4: Postgres init script timing; Spock extension load order; Traefik certificate issuance; the first bundle's signature verification (matching the offline CA's root cert correctly).

By the end of week 4, you have a working GOAT-Net site at home station, with your tech lead enrolled as the first peer.

Week 5 — Apps + operators

Goal: At least one of your apps runs on GOAT-Net. Operators start hands-on.

Activity Lead Outcome
Pick first app to run on GOAT-Net (TAK federation, sensor ingest, dashboard, custom) Tech lead App identified
Deploy first app — Ansible role or docker-compose, your choice Operator + Tech lead App running, reachable from mesh peers
Enroll 5-10 more peers (operator workstations, lab Orins, etc.) Operator Mesh has population; peer-to-peer paths exercised
Operator runbook walkthrough — daily checks, alerts, common stuck-points Operator + us Operator can describe how to read the dashboard, what alerts mean, what to do when something breaks
Operator tooling walkthrough — the goat CLI, the live fabric dashboard, the identity view, the inspection surfaces Operator + us Operator can answer "is the fabric healthy", "who is this peer", and "what can it reach" without asking us
Backup verification — run pgbackrest stanza-create + first backup Operator Backup working; test restore on a separate host
Audit log integrity check Operator First daily check succeeds
Continuous chaos probes (subset) — run a partition probe in lab Operator + us Substrate behaves correctly under partition

Concierge load: ~4-6 hours of sync; lots of async during operator hands-on.

Stuck-points typical at week 5: app integration with mesh DNS (your app may assume direct IP routing); TAK-shim configuration if running TAK; backup-restore on a non-canonical host.

Week 6 — Operations baseline + Phase 2 planning

Goal: You're operating GOAT-Net independently. You have a written plan for Phase 2.

Activity Lead Outcome
Independent stand-up rehearsal — operator stands up a fresh lab site from clean hardware in ≤1 day with no concierge support Operator "If we lose this site, we can rebuild it in a day" — proven
Drift-check sweep — what's deployed differs from documentation? Tech lead Closeout list
Phase 2 design call — division HQs? Brigade CPs? Transport? Drill cadence? Tech lead + us Phase 2 plan draft
Phase 2 procurement plan Procurement officer + Tech lead Hardware order, transport plan
AO formal engagement begins — show them the deployment, walk through compliance posture Tech lead + AO AO has a mental model; ATO journey starts
Concierge engagement transition — to "as-needed" instead of weekly sync Tech lead + us Reduced cadence; you have our number

Concierge load: ~3-4 hours of sync; mostly transition.

By the end of week 6, you have:

  • A working GOAT-Net lab at home station
  • Operators trained, including on the operator CLI and inspection surfaces
  • One+ of your apps running on it
  • A backup + restore proven
  • A Phase 2 plan
  • AO engaged

What can go wrong (and what we do about it)

Risk Likelihood Mitigation
Hardware doesn't arrive on time High Use AWS EC2 mgmt-hosts as evaluation stand-in; switch to Orin/Karbon when arrives
YubiKey 5.7+ firmware not available in your supply chain Low We have a 5-operator dev-mode fallback
Cloud account constraints (TAA, GovCloud quirks) Medium We surface in week 1 design call; some unknowns will emerge
Tech lead loses time to higher-priority work Medium We can pause + resume; concierge engagement is flexible
AO has unexpected concerns Medium We support compliance-posture conversation; you own the AO relationship
App integration is harder than expected Medium Most apps need ~half-day per integration; some need more
Substrate bug surfaces in your environment Low-medium Treat as F-NNN finding; we fix in flight; you may be the first to find it

Definition of done — Phase 1

You're done with Phase 1 when:

  • Lab site is operational; observability green
  • Tech lead has stood up a fresh site from clean hardware in ≤1 day independently
  • At least one of your apps runs on GOAT-Net
  • Operators have completed runbook walkthrough; can describe daily checks
  • Backup-restore proven on a separate host
  • Phase 2 plan draft exists
  • AO has reviewed compliance posture (informally; formal ATO comes later)
  • You have a written list of what's open / blocked / next

If any of these are not true, Phase 1 isn't done. We'd rather extend Phase 1 by 1-2 weeks than push you to Phase 2 with a half-baked lab.

What's the bare minimum if 6 weeks is too long?

If you have 2-3 weeks, not 6, the compressed shape is:

  • Week 1: Design + procurement + cloud + read packet (compressed; go-faster path means we accept more open questions)
  • Week 2: Hardware on bench + offline CA + relay tier + first mgmt-host
  • Week 3: First peer enrolled + first app running + operator handoff

Tradeoffs of compressed Phase 1:

  • No drift-check sweep
  • Operators less trained
  • AO not yet engaged
  • Less time for substrate to settle / for surprises to surface
  • Phase 2 planning happens during Phase 2 instead of before it

The compressed shape is real, but you carry more risk into Phase 2. We recommend the 6-week shape unless your timeline is firm.