Architecture.

§0 · what it is

A self-hostable, open-source, dual-protocol VPN fleet platform.

One codebase serves two postures from the same binaries:

• Personal — "I want my own VPN," one operator, a handful of nodes.
• Enterprise — a team managing many users across many regions.

Defaults differ; the engine is identical.

The data plane is AmneziaWG (obfuscated WireGuard) and XRay (VLESS + REALITY), both terminating end-user tunnels on UDP/TCP 443. The platform is the control plane, account system, and clients around that data plane.

§1 · goals

Five hard constraints the design defers to.

• Self-hostable in under 30 minutes. Clone, follow the README, get a working fleet. No lock-in beyond the chosen cloud provider.
• Defense-in-depth control plane. The controller issues credentials and rotates server config — the highest-value target. It assumes it will be attacked: no inbound ports, no public DNS, no public IP.
• Dumb nodes. A compromised VPN node must not yield control of the fleet. Nodes act only on cryptographically validated instructions.
• Survive a controller outage. The controller is always-on and continuously reconciles the fleet, but it is a control-plane, not a data-plane, dependency: if it is briefly unavailable, every node keeps serving existing tunnels indefinitely. The same applies to clients — a client connects from cached profiles when the account service is unreachable.
• The controller never holds usable user secrets. User profiles are end-to-end encrypted; a controller compromise yields ciphertext, not profiles.

§3 · component responsibilities

What each piece is on the hook for.

coxswain — controller

• Source of truth. SQLite by default (optional pure-Go Postgres backend, selected by DSN) holding fleet inventory, profiles, users, devices, peers, admins, sessions, the CA, the audit log, session history, and alerts.
• Always-on, self-healing. A reconcile sweep checks every node on an interval and re-applies config when it drifts or a data plane goes stale; creating or removing a profile/device pushes to the affected nodes automatically. A controller restart re-reconciles the whole fleet.
• Admin Web UI — embedded SPA served on localhost: fleet, paths, profiles, plus live sessions, alerts, the audit log, and API tokens.
• Outbound control loop — holds a long-lived mTLS/gRPC connection to each node; pushes config, pushes/revokes peers, and receives a live event stream.
• Token-authenticated API + audit. A management API secured by scoped, expiring tokens (hashed at rest) alongside session login; every management action lands in a hash-chained, tamper-evident audit log.
• Monitoring & analytics. Persists connect/disconnect session history with per-session byte counts; an in-process engine sweeps it and raises alerts (best-effort, experimental). A gRPC event stream is available for SIEM ingestion.
• Node onboarding over SSH — installs and updates the node agent on operator-provided VMs; all node control is gRPC.
• Issues node certs, the controller's own client cert, relay certs, and per-user/device certs.
• Account & sync service — authenticates users/admins, serves E2E-encrypted profile bundles. This surface is reached only via a relay.

node — VPN node agent

• Stateless except for what coxswain gave it. All config is written to disk only after coxswain pushes it over mTLS.
• Data plane: awg-quick@awg0 on UDP 443, xray.service on TCP 443.
• Control port (mTLS-only, gRPC). Status, metrics, push config, live peer add/remove, handshake stats, service restart — and a server-stream of live events back to coxswain.
• SSH is install-only. Every operational instruction is gRPC.
• Cold-start resilient. Comes up from disk every boot. Controller offline ⇒ existing peers keep working.
• Network policy. Three independent toggles per node — forwarding, masquerade, client-isolation — that coxswain compiles into a canonical PostUp/PostDown rule set and pushes over the control channel. The rule set is the contract: coxswain's preview and the node's application must not drift.
• Multi-IP / port-range endpoints. A node binds a set of public IPs and a UDP port range, yielding a large pool of (ip, port) endpoints from a small config — and the profile carries that pool as an array, never a single endpoint. See the §3 callout below on what this is for.

relay

• Stateless public proxy. Terminates client mTLS, forwards gRPC streams to coxswain. No database; every lookup is delegated.
• Strips spoofable client metadata; injects exactly one trusted value — the verified device fingerprint.
• Two transports to coxswain: embedded (in-process) or remote reverse tunnel.
• Carries only ciphertext profile bundles — see §8 — so a compromised remote relay host cannot read user profiles.

caravel — mobile client

• Two decoupled layers: a VPN engine (multi-node, multi-protocol) and a set of pluggable profile sources.
• Posture-aware: personal (account login, QR, file import; admin section if the logged-in account is an admin) vs managed (MDM config present — account login and admin hidden, profiles locked). One app, one store listing.

§4 · trust model & PKI

One root, two intermediates, no third party.

A single in-repo root CA, generated on coxswain's first run, stored in coxswain's SQLite, never copied off the controller. Two intermediates under it:

• Fleet CA — issues node certs, the controller's client cert, and relay certs.
• Device CA — issues per-user / per-device leaf certs for caravel and the admin browser.

Certificate	Issued by	Held by	Validity
Root CA	self-signed	coxswain only	10 years
Fleet / Device intermediates	Root CA	coxswain only	5 years
Controller client cert	Fleet CA	coxswain	1 year, auto-rotated
Node server cert	Fleet CA	each node	1 year, auto-rotated
Relay cert	Fleet CA	each relay	1 year, auto-rotated
Device leaf	Device CA	each caravel / browser	1 year

Compromise containment

• Compromised node → attacker gets that node's key + the CA cert (not key). Cannot impersonate coxswain or other nodes. Operator revokes the node cert.
• Compromised remote relay → attacker can see traffic metadata but profile bundles are E2E-encrypted ciphertext. Cannot mint certs.
• Compromised coxswain → attacker gets the CA key. Fleet fully compromised — but user profiles remain encrypted (the controller never holds users' private keys in usable form). Hence coxswain's "no inbound ports, behind NAT" posture.

Post-quantum hardening

Every AmneziaWG peer is issued a unique 256-bit preshared key, mixed into the WireGuard handshake. WireGuard's ECDH (Curve25519) is quantum-vulnerable; the symmetric PSK is not — so recorded tunnel traffic stays confidential against a future harvest-now-decrypt-later attacker. coxswain generates the PSK per peer, ships it inside the E2E profile bundle (§8), and pushes it to node. This is a pragmatic interim measure, not a full post-quantum handshake.

§7 · real-time & multi-admin

The admin UI feels live, multi-admin safe.

A client connecting to a node appears immediately, not on a thirty-second poll.

• node → coxswain: coxswain holds its outbound mTLS connection open and the node streams events (handshake up/down, peer connect/disconnect, errors) over a gRPC server-stream. Polling remains only as a fallback heartbeat.
• coxswain → browser: every open admin page holds a WebSocket. coxswain pushes state changes to all of them — open the dashboard on three machines, all three update together.

Optimistic concurrency

Every mutable record carries a version integer. A mutation must send the version the admin loaded. If coxswain's current version is higher, it rejects with HTTP 409 Conflict — "changed by someone else, reload." Live WebSocket replication makes conflicts rare; the version check is the hard safety net.

§8 · control-plane operations

A controller that watches itself and the fleet.

The controller is always-on and treats correctness as a continuous property, not a one-shot push. This part of the platform has shipped and is exercised on a live fleet; the analytics rules are best-effort and experimental.

Self-healing reconciliation

• Reconcile sweep. The controller checks every node on an interval and re-applies config when a node serves stale config, or when peers exist but nothing is handshaking. Per-node rate-limited.
• Zero-touch provisioning. Creating or removing a profile or device pushes to the affected nodes automatically; if a node is unreachable, the sweep delivers when it returns.
• Drift-aware status. cox nodes status surfaces applied-vs-intended config revision and live handshake liveness — a silently broken node shows red, not green.

Access, tokens & audit

• Scoped API tokens. Least-privilege scopes (readonly / monitor / admin), optional expiry, hashed at rest (plaintext shown once), revocable.
• Tamper-evident audit log. Every management action — API and CLI — records who, what, to what, from where, and success/failure. Rows are hash-chained, so edits or deletions are detectable (cox audit verify).
• Hardened access. The session cookie is Secure over TLS; the dashboard is reached only over TLS or an SSH-forwarded loopback port.

Monitoring & analytics

• Live + persisted sessions. Connect/disconnect events stream off each node over gRPC, carrying the per-session source IP and resolved device/user, and are persisted with per-session rx/tx byte counts.
• Anomaly detection (experimental). An in-process engine sweeps session history and raises alerts — leaked profile, impossible travel, off-hours access, auth-failure spikes, revoked-profile-active, and a best-effort data-volume / exfil rule — each with severity and evidence, ack/resolve-able.
• SIEM stream. A first-class gRPC event stream (monitor-scope token, off by default) carries the same events to enterprise ingestion, alongside SSE/WebSocket for dashboards.