The v1 agent wire contract
- Status
- Accepted
Context
ADR-0002 fixes the transport (one persistent stream per agent, desired-state deltas), ADR-0004 fixes the identity model, ADR-0010 fixes the enforcement scope, and ADR-0012 reserves the multi-region seams. None of them fixes the shape of the messages. The first proto definitions under proto/innerwall/v1/ make those shapes concrete, and from the moment they land, the breaking-change gate holds every later change to them. The decisions below are the ones the shapes encode and that a later contributor could plausibly undo one field at a time.
Decision
-
Two services, split on the authentication boundary.
EnrollmentServiceis reachable with a provisioning token and nothing else; it is the only surface a workload can reach before it has an identity.AgentServiceis reachable only with the credential issued at enrollment, over mutual TLS. Authentication policy is therefore declared per service, not per RPC, and no RPC can be accidentally exposed on the wrong side of the boundary. WithinAgentServiceone thing is decided per method, and it moves no method across the boundary: how long after its expiry a credential is still accepted, which is for renewal alone, within the grace ADR-0016 as amended sets. (Amended 2026-10-14; see Amendments.) -
Snapshot on connect; deltas are intra-session only. Every new sync stream begins with the server sending a full
PolicySnapshot, regardless of the version the agent claims inHello. Deltas are sent only after that snapshot, on the same stream. Rendered per-workload policy is small (a workload's own inbound rules) and reconnects are rare, so the cost is a few kilobytes per reconnect. What it buys is the elimination of cross-session divergence as a failure class: there is no state in which the server and agent disagree about the base a delta applies to. -
Atomic apply-or-fail acknowledgement. The server pipelines
PolicyUpdatemessages without waiting for acknowledgements; the agent applies them strictly in order and acknowledges each withACK_STATUS_APPLIEDorACK_STATUS_FAILED. Partial application is not representable: a version number is only meaningful if it describes the complete installed state. A failed apply leaves the agent on its last good version, and the server's recovery move is a fresh snapshot, never a retried delta. An acknowledgement says what was installed when it was sent; whether the owned firewall table still holds it is a separate fact every heartbeat carries, because the local kill switch removes the table without changing any version (ADR-0011). (Amended 2026-10-13; see Amendments.) -
Per-workload monotonic versions. The wire contract knows only a per-workload version, opaque to everything except ordering. Any estate-wide "generation" that the UI or audit trail wants is a server-internal mapping and is deliberately absent from the agent contract, so that recompiling one workload never forces a version bump on every other.
-
Authored and rendered models are separate layers. Operators write
Rulesets whoseLabelSelectors express intent over labels; the control plane resolves that intent intoWorkloadPolicy, a fully concrete set ofResolvedRules (peer CIDRs, one protocol, port ranges). Agents never evaluate selectors. Agent behavior is deterministic and debuggable from the rendered artifact alone, selector semantics can evolve without an agent upgrade, and deltas operate on the rendered model, where the high-churn quantity, peer membership in a resolved rule, is cheap to express (PeerChange). -
Direction is defined; outbound is rejected at admission.
Directioncarries bothINBOUNDandOUTBOUNDfrom the first version, and aRuleorients its scope and peers by direction rather than carrying separate source and destination fields. In v1 the control plane rejects any rule withDIRECTION_OUTBOUNDat admission (ADR-0010); enabling outbound later is a validation change, not a schema change. Flow telemetry (ReportFlows) is a separate RPC fromSync, so telemetry volume can never head-of-line-block a policy update. A report also carries the intervals in which the agent knows its evidence is incomplete, so a loss travels with the evidence it qualifies, on the same loss-tolerant path, and is never inferred from silence. (Amended 2026-10-04, twice on 2026-10-11, and 2026-10-13; see Amendments.) -
Scoped provisioning tokens with server-assigned labels. A token is minted for a label scope and may enroll many workloads, so it can be baked into an image or a provisioning pipeline. The control plane assigns labels from the token's scope; agents never self-assign labels, because an agent-supplied label would let a compromised host place itself into a more permissive policy scope. Identity is granted at enrollment and is thereafter derived only from the connection credential, never from request payloads (ADR-0004). Enrollment and renewal carry a PKCS#10 certificate signing request, so the private key never leaves the host.
The message that each decision lives in, and the region_id reservations required by ADR-0012, are in proto/innerwall/v1/. Comments in those files carry the same reasoning at the point of use.
Consequences
- The proto files are now under the breaking-change gate; undoing any decision above is a visible, reviewed proto diff and requires a superseding ADR.
- The agent is simple by construction: no selector evaluation, no delta reconciliation across sessions, no partial state. Complexity concentrates in the control plane, where it is observable and testable.
- Reconnection costs one snapshot per stream. At the sizes involved this is negligible; if a future estate proves otherwise, the fix is a superseding ADR, not a silent reintroduction of cross-session deltas.
- Outbound policy, when it arrives (ADR-0010), changes admission validation and the rendered model, not the authored
Ruleshape. - Enrollment tokens are scoped secrets. Their lifecycle (expiry, usage limits, revocation) is enrollment-policy behavior, not wire-contract behavior, and is specified with the enrollment implementation.
Amendments
(PR #22, flow-collection gaps). Additive fields under decision 6, no field renumbered or retyped. ReportFlowsRequest gains repeated EvidenceGap gaps: each gap is an interval [from, to) with a kind (source overrun, source restart, buffer overflow, truncated table dump), the source that lost the evidence (unspecified for a loss that is not one source's), and a count where one is known. Every loss the agent can detect, the buffer's dropped windows included, is expressed this one way. Gaps ride the next window rather than a schedule of their own, and a window may carry gaps and no records; delivery is at least once, like the records', and the reporting workload is the connection credential's. Heartbeat gains source_overruns, the live count beside dropped_flow_records. FlowRecord.connection_count is documented as the connections opened, or found live when the source subscribed, in the window, which is what the agent's table dump at every subscribe produces. The core decisions stand.
(PR #23, conformance-audit absorption). One additive enum value under decision 6, nothing renumbered or retyped: EvidenceGapKind gains EVIDENCE_GAP_KIND_WINDOW_OVERFLOW, the agent's open aggregation window held as many distinct keys as it allows and observations for new keys were dropped, with the observations dropped as its count (ADR-0019 as amended). The core decisions stand.
(PR #81, clock-skew detection). One additive field under decision 6, nothing renumbered or retyped: ReportFlowsRequest gains sent_at, the sender's clock when it sent the request, stamped on every request immediately before it is sent, a retried window included. It carries the agent's clock for skew detection: the control plane compares it with the instant it received the request, and a difference beyond twice the flow aggregation window (a named constant in internal/ingest, not configuration) is skew. An agent whose clock has drifted stamps every window it reports with that clock, so its flows land at the wrong times without anything saying so; the skew is recorded on the workload in Postgres (ADR-0017) and surfaced, a WARN log line naming the workload and the offset and a note beside the workload's flows, and the window is stored as reported, never refused. The window bounds cannot carry this: the agent holds closed windows in its buffer while the control plane is unreachable and delivers them when it returns (ADR-0019), so late delivery of buffered windows is expected and is not skew. A request without sent_at, from an agent that predates the field, is judged by its window's end and only for a clock running ahead, since a window cannot legitimately arrive before it ends. The core decisions stand.
(PR #90, rejected windows). One additive enum value under decision 6, nothing renumbered or retyped: EvidenceGapKind gains EVIDENCE_GAP_KIND_WINDOW_REJECTED, the control plane refused a closed window as malformed (InvalidArgument) and the agent dropped it, over the window's bounds with its records as the count (ADR-0019 as amended). A refused window can never be accepted, and retried as the oldest window it held every later window behind it until the agent restarted: one malformed window cost all telemetry after it, with nothing saying so (issue #86). Every other failure is still retried with backoff, because a window delivered late is legitimate (ADR-0019). The control plane reads a report's clock before it validates the window's bounds, so a report refused for its bounds still records the skew that likely inverted them; sent_at is independent of the bounds. The core decisions stand.
(PR #95, owned-table presence, issue #64). One additive field under decision 3 and one additive enum value under decision 6, nothing renumbered or retyped. Heartbeat gains owned_table, an OwnedTableState: whether the Innerwall-owned firewall table is in the kernel, read when the heartbeat is composed (PRESENT, ABSENT, or UNSPECIFIED from an agent that predates the field, one that has installed nothing, or a read that failed). The local kill switch deletes the table without the daemon or the control plane (ADR-0011) and changes no version, so an acknowledged version kept saying a host enforced what it no longer held: the console showed a kill-switched host enforced and synced until an operator happened to push a policy. The read is part of composing the heartbeat, on the heartbeat's own schedule, the way an inventory report reads the host's listening services; it adds no timer and asks the control plane nothing, so ADR-0002's no-polling rule and its one recorded host-local exception are untouched, and the removal is reported within one heartbeat interval. The agent never reinstalls the table because it found it gone, since removing it is the kill switch; its next apply or restart reinstalls it, as before. The control plane records the state on the workload in Postgres (ADR-0017), keeps the last known state when a heartbeat does not know, and treats an applied acknowledgement as the table installed. EvidenceGapKind gains EVIDENCE_GAP_KIND_TABLE_REMOVED: observation and the classification its connection marks carry die with the table, so the interval from the last instant the agent knew the table present to its reinstallation is an evidence gap, not one source's and without a count, shipped in pieces with every window that closes while it lasts so that a long removal is never read as a quiet host (ADR-0019 as amended). The core decisions stand.
(PR #102, issue #93): an expired credential may renew. Clarification of decision 1, corrected in place. The boundary between the two services stands as decided: EnrollmentService takes a token and nothing else, and every AgentService method requires a verified workload credential. What is new is one per-method distinction inside AgentService, on the credential's validity period alone: RenewCredential accepts a credential expired within the renewal grace, and every other method requires one within its validity. Default-deny holds, since a method added later requires a valid credential unless it is named as renewal is. ADR-0016 as amended carries the rule and its reasoning.
(PR #125, a silent agent, issue #114). One additive enum value under decision 6, nothing renumbered or retyped: EvidenceGapKind gains EVIDENCE_GAP_KIND_AGENT_SILENT, the control plane received no flow report from the workload for longer than the telemetry-liveness allowance and no window delivered afterwards covered the interval (ADR-0019 as amended). Unlike every earlier kind it is never sent by an agent — an agent cannot observe its own absence — and ingest refuses a report that claims it; the value lives in the wire enum because the stored gaps, the read model, and the operator surface speak this one enum, and reserving the number keeps a future agent from minting it to mean something else. Decision 6's rule that a loss "is never inferred from silence" stands for the agent's own reporting: the agent still infers nothing; the control plane states only what it directly observed — its own receipts — and the interval is bounded by them. The core decisions stand.