Persistent gRPC streams with desired-state delta sync
- Status
- Accepted
Context
A central plane serving thousands of endpoints fails predictably when the protocol is chatty: agents polling for policy, reporting via repeated REST calls, and sharing a rate-limited front door with human and automation traffic. Poll intervals force a trade between convergence latency and load; reconnection after outages produces thundering herds; imperative APIs make missed calls a correctness problem.
Decision
- Each agent holds one persistent bidirectional gRPC stream over mTLS for policy: deltas flow down; heartbeats, inventory, and ACKs flow up. Aggregated flow telemetry travels on its own RPC,
ReportFlows, over a connection of its own with its own jittered backoff, so telemetry volume can never head-of-line-block a policy update (ADR-0015, ADR-0019). Nothing polls the control plane in steady state on either. (Amended 2026-09-13 and 2026-10-06; see Amendments.) - Policy distribution is desired-state: the control plane versions each agent's compiled ruleset; agents reconcile to the latest version and report the version they run. An agent offline for an hour reconnects and receives one snapshot of its current policy (ADR-0015), not a backlog of missed calls. (Amended 2026-10-06; see Amendments.)
- Reconnection after failure uses exponential backoff with full jitter (
wait = random(0, min(cap, base × 2^attempt))) because a control-plane restart otherwise triggers a synchronized storm of CPU-expensive mTLS handshakes; directed reconnects redial immediately, and the attempt counter resets upon successful stream establishment. (Amended 2026-10-10; see Amendments.)
Consequences
- The dominant failure mode of chatty central APIs is designed out rather than rate-limited.
- Desired-vs-actual version is queryable; drift detection and rollback are structural features.
- Long-lived connections make the gateway stateful for the life of a stream: a render's announcement reaches the replica holding a workload's stream through the database notification channel, so any replica can push to any agent it holds without a presence table (see ADR-0017, as amended).
- Firewalls between agents and the control plane must permit long-lived outbound TLS; documentation must cover keepalives and middlebox timeouts, and the deploy guide does. (Amended 2026-10-06; see Amendments.)
Amendments
(PR #5). The consequence that required a presence table for routing pushes is corrected in place: pushes reach the replica holding a stream through the database notification channel, per ADR-0017's amended propagation design. The core decision, one persistent stream per agent carrying desired-state deltas with full-jitter backoff and nothing polling, stands.
(PR #7, ADR-0015, ADR-0019). The decision bullet that listed aggregated flows among what flows up the policy stream was written before the wire contract existed and is corrected in place: ADR-0015 fixed flow telemetry as a separate RPC so that its volume can never head-of-line-block a policy update, and ADR-0019 built it on a connection of its own with the same full-jitter backoff. The core decision stands: one persistent policy stream per agent, desired-state deltas, jittered reconnects, and nothing polling anywhere in steady state.
(PR #25, conformance-audit absorption). Two subsidiary points are corrected in place, and the core decision, one persistent policy stream per agent, desired-state distribution, full-jitter reconnects, stands. The sentence that an agent offline for an hour "fetches one delta" predates ADR-0015, which fixed that every new stream begins with a full snapshot and deltas exist only inside a stream; the gateway does exactly that. The no-polling rule is scoped to what it was written against, interaction with the control plane: nothing asks the control plane whether anything changed. Scheduled work the records name (heartbeats, inventory, window rotation, renewal, retention) is not polling the control plane, and one host-local read is recorded as an exception: the conntrack source re-reads the host's own addresses every 30 seconds, because that is how it tells inbound connections from others; a netlink address-change watch that replaces the timer is recorded backlog. The keepalive consequence now names where the guidance lives.
(PR #53). Reconnection after a dial or stream failure retains exponential backoff with full jitter, but a directed reconnect redials immediately because the control plane is known reachable. The attempt counter resets to zero upon stream establishment (after HelloAck) so backoff does not accumulate across unrelated outages or directed handoffs. Resetting at HelloAck rather than after a session survival threshold means a control plane that crash-loops after HelloAck is redialed by every agent within random(0, 2 × base) on each cycle instead of backing off exponentially; that trade-off is accepted to keep the reconnect loop simple, with a survival threshold (such as one heartbeat interval) as the recorded path if post-handshake crash loops require it. The core decision stands.