Rendered from innerwall v0.4.1ADR-0019

Flow storage as windows plus totals, peers resolved at ingest, bounded retention, and the query shapes the schema serves

Status
Accepted

Context

ADR-0009 fixed that agents aggregate flows before shipping, that version 1 stores them in Postgres only, and that every read and write goes through a FlowStore interface over a column-shaped schema. ADR-0015 fixed the wire record: one FlowRecord per (source, destination, port, protocol, direction, decision) per reporting window, shipped on a telemetry stream of its own. Implementing the first end-to-end flow path, from the host's connection tracker to a queryable table, made four things concrete that neither record settled, and each is the kind of decision a later change could undo one plausible pull request at a time: what the rows are, when a source address is turned into a peer, how rows age out, and which queries the schema is shaped to answer.

The operator console's design produced the queries. It needs a workload's flows over a time range; a rollup over a label scope, filtered by decision and grouped by peer and service, to answer "what would this policy have blocked, and for whom"; and a "since first seen" verdict per flow key that does not depend on how much history is retained. A schema that served the first two by scanning and the third by scanning everything ever stored would be correct and slow in exactly the query that makes simulation trustworthy.

Decision

  1. Two tables: windows and totals. flow_windows holds one row per aggregated record per reporting window, as the agent shipped it plus what ingestion resolved. flow_totals holds one row per (workload, resolved peer, destination port, protocol, direction, decision), upserted in the same transaction as the window's rows: the counters add, the first-seen instant is the earliest, the last-seen the latest, and the label snapshot and matched rule are the latest window's. Windows serve time-bounded questions and are pruned; totals serve "since first seen" questions from one row and are never pruned. Both are column-shaped, with no nested structure beyond the label snapshot, so the same rows port to a columnar store behind the interface when an estate's volume demands it (ADR-0009). Both reserve region_id (ADR-0012). Beside them, flow_gaps holds the intervals in which a workload's agent knows its evidence is incomplete, one row per reported interval, keyed so that a repeated delivery is stored once; it is column-shaped, reserves region_id, takes the workload from the connection credential like the windows, and is read and written only through the same interface. A gap is not a flow record: it qualifies the windows of its workload over its interval, and nothing aggregates it into them. (Amended 2026-10-04; see Amendments.)

  2. Peers are resolved at ingest and stored with the row. When a window arrives, each record's source address is resolved once, against the registry as it stands at that instant: a workload's current address wins, then the most specific address group containing the address, then nothing, in which case the address itself is the peer. A workload peer is stored with its identity and a snapshot of its labels at that moment. Resolution happens at ingest and not at query time because addresses are reassigned and labels change: a join at query time would show today's owner of an address as the peer of last week's flow, and today's labels as the scope it was in. The stored row is what the operator would have seen had they looked at the time. The reporting workload's identity comes from the connection credential and from nowhere in the payload (ADR-0016).

  3. Retention is a bounded, periodic delete of windows; totals are kept. A scheduled job on every control-plane replica deletes windows that started before a configurable horizon (30 days by default), in bounded batches, each batch under a transaction-scoped advisory lock so that one replica prunes at a time and the others skip the round (ADR-0017). The job has a fixed period and runs whether or not there is anything to delete; nothing waits on it, so it is maintenance, not the polling ADR-0002 forbids. The same batches, under the same lock, delete gaps that ended before the horizon, so a gap is kept at least as long as any window it could describe. (Amended 2026-10-04; see Amendments.) Declarative time partitioning with retention by partition drop, which ADR-0009 named, is the recorded path when volume makes deletes expensive: the windows table is keyed by window start so the partition key is already in place, and the change is a migration behind the same interface. ADR-0009 is amended to say so.

  4. The schema serves three query shapes and the command line issues them first. (a) A workload's windows in a time range, optionally filtered by decision, newest first. (b) A rollup over a set of workloads (a label scope, resolved by the caller to the workloads currently in it) in a time range, optionally filtered by decision, grouped by resolved peer and service (destination port and protocol), with the distinct workload count, summed counters, and the range of instants seen. (c) A workload's totals since first seen, optionally filtered by decision. The indexes exist for these and nothing else: (workload, window start) for (a), (decision, window start) for (b), window start for retention, and the totals primary key for (c). The innerwall flows commands issue exactly these queries, so the storage shape is proven by the only read surface that exists before the API façade (ADR-0007) lands. The operator surface's read model adds five named groupings of shape (b) over the same windows and indexes, each its own statement (by matched rule, with the distinct workload count; by rule and resolved peer; by resolved peer and reporting workload; by reporting workload and service; by resolved peer and service, with the distinct workload count, which is (b) itself), whose workload set selects the records its workloads reported or, when the caller asks for either endpoint, also the records whose source resolved at ingest to one of them, each record once, bounded to a documented number of groups with the totals of the whole, and a form of (a) paged by a cursor on (window start, row id); there is no free-form grouping. The evidence gaps have one read of their own, not a grouping: the gaps of a workload set (one workload, a label scope resolved as for (b), or every workload) that intersect a half-open range, newest first and bounded, as one named statement served by a (workload, gap end) index, with gap end also serving retention. (Amended 2026-09-13, 2026-09-28, 2026-10-04 twice, and 2026-10-09; see Amendments.)

  5. The agent's side of the pipeline is bounded and independent of the sync stream. The collector aggregates observations from the host's connection tracker on the wire record's key over the window the control plane configures in SyncConfig; a closed window enters a buffer bounded in records; when a window does not fit, the oldest windows are dropped first and every dropped record is counted into the heartbeat's dropped_flow_records, so the operator knows the map is incomplete before trusting it (ADR-0011). Each drop is also an evidence gap over the dropped window's bounds with the count dropped, beside the gaps the sources record (a kernel overrun, a restart, a table dump cut at its bound) and the gap the open window records when it is full: the window holds a bounded number of distinct keys, the closed buffer's figure by default, an observation for a new key past the bound is dropped, and the window reports one gap from the first drop to its close with the observations dropped; the reporter ships the gaps held with the next window and puts them back when its stream fails. A window's length is measured on the monotonic clock as well as the wall clock, and a window the wall clock stepped back under is cut at the step, ending at its start plus its monotonic length with the next window opening on the stepped clock, so no window closes with its end before its start. A window the control plane refuses as malformed is dropped rather than retried, and the drop is a gap over its bounds with its records as the count. The interval in which the owned firewall table is gone from the kernel is a gap too, from the last instant the agent knew the table present to its reinstallation, shipped in pieces with each window that closes while it lasts. (Amended 2026-10-04, 2026-10-05, 2026-10-11, and 2026-10-13; see Amendments.) The reporter ships one window per ReportFlows stream, in requests capped at SyncConfig.flow_batch_max_records, over a connection of its own with its own jittered backoff, so telemetry never shares the sync stream's fate (ADR-0015). Records that fail validation at ingest are skipped and counted; the rest of the window lands.

Consequences

  • A flow record is immutable history: the peer it names and the labels it carries are the ones in force when it was observed. Renaming, relabeling, or re-addressing a workload changes nothing already stored. A display that wants the peer's current name looks it up by the stored identity; the snapshot is for the scope the flow was in.
  • "Since first seen" is one row per key regardless of the retention horizon, at the cost of one upsert per key per window, which the batch write makes cheap.
  • Retention is a delete, not a partition drop, in this version. It is bounded and serialized, and it is the first thing to revisit when window volume makes it visible. The migration to partitioning changes nothing above the FlowStore interface.
  • The rollup resolves its label scope at query time, deliberately: it answers what the workloads in scope now have seen, or, matching either endpoint, what they have seen and sent. Either way the scope's membership is today's; a record's source is matched by the workload identity stored with it, never by its label snapshot. The stored label snapshot answers the other question, what scope a flow was in when it happened, and the two are not conflated.
  • The buffer is in memory and bounded in records. Buffering to local disk across an agent restart, which ADR-0011 anticipates, is additive behind the same buffer and lands with the resource budgets it belongs to.
  • Byte counts depend on the kernel's per-connection accounting being enabled; where it is off, connections are counted and bytes are zero, and the agent says so once at start. The agent does not change kernel settings on the host's behalf.

Amendments

Amendment2026-09-13

(PR #10, M3.2 operator read model). Decision 4 gains the groupings the operator surface's rollup endpoint serves and the cursor-paged form of the windows list, as additive detail: four named groupings over the same windows and the same indexes, each one hand-written statement chosen by name, never assembled; a rollup bounded to a documented group cap that returns the top groups in its order with the totals and group count of the whole and a marker when truncated; and pages keyed by (window start, row id) so a window landing between pages neither repeats nor hides a row. The peer in every read is the one stored at ingest (decision 2); the read model looks up only the current display name of the stored identity. No schema or index changes; the core decision stands.

Amendment2026-09-28

(PR #17, M3.7 simulation review). Decision 4's operator-surface groupings gain a fifth, by resolved peer and service with the distinct workload count per group: the shape (b) already names, served on the rollup endpoint as group_by=peer,service with workload_count on each group, as its own named statement over the same windows and indexes. The simulation review's rows, matrix, and recency ordering are keyed by peer and service; the four groupings could produce them only by assembling rows client-side, which ADR-0007 as amended rules out. Additive detail; no schema or index changes; the core decision stands.

Amendment2026-10-04

(PR #19, M3.8 policy editor). Decision 4's grouping by matched rule gains the distinct workload count per group, workload_count on the rollup endpoint, in the shape the fifth grouping carries it: the policy editor states each rule's hit counters with the number of workloads in scope that reported them, and no grouping could give that number without assembling it client-side, which ADR-0007 as amended rules out. The same statement over the same windows and indexes gains one aggregate; the set of groupings stays closed at five. Additive detail; no schema or index changes; the core decision stands.

Amendment2026-10-04

(PR #22, flow-collection gaps). Evidence gaps join the schema, the pipeline, and the reads, as additive detail. Decision 1 gains flow_gaps beside the windows and totals: intervals in which a workload's evidence is known to be incomplete, stored once per reported interval, with region_id reserved and the workload taken from the credential. Decision 3's retention deletes gaps that ended before the horizon in the same locked batches, so gaps outlive every window they describe. Decision 4 gains the gaps read, one named statement over a workload set and a half-open range, which is not a grouping; the set of groupings stays closed at five. Decision 5's buffer records every drop as a gap as well as counting it, so the one loss model covers the buffer and the sources alike. A range whose evidence has a gap in it is not complete evidence, and the readers that draw a verdict from flows say so (ADR-0001). The windows and totals, their indexes, and the core decision stand.

Amendment2026-10-05

(PR #23, conformance-audit absorption). Decision 5 gains a bound on the open window, as additive detail: the aggregator holds at most a configured number of distinct keys (--flow-window-keys, the closed buffer's 50,000 by default), drops an observation for a new key past it while still folding observations for keys it holds, and records the loss as a fifth gap kind, window_overflow, from the first drop to the window's close with the count of observations dropped. flow_gaps admits the fifth kind (migration 00009). The pipeline was bounded only after a window closed; an audit showed the open window growing with every distinct key. The core decision stands.

Amendment2026-10-09

(PR #36, flow map label scope). Decision 4's operator-surface groupings gain a choice of endpoint for their workload set, as additive detail: by default the set selects the records its workloads reported, the traffic that reached them; asked for either endpoint (endpoint=either on the rollup endpoint, --endpoint either on innerwall flows rollup --group-by), it also selects the records whose source resolved at ingest to a workload in the set, a record with both ends in it counted once. A label scope on the flow map selected only destinations, so a workload that only sends traffic drew an empty map; the map now asks for either endpoint, while the simulation review, the policy editor's hit counters, and a workload's own reads keep the destination, because they answer what reached the workloads in scope. The source is matched by the identity stored at ingest (decision 2) against the scope's current membership, resolved at read time as before. Each grouping stays one named statement over the same windows; no schema or index changes; the core decision stands.

Amendment2026-10-11

(PR #90, a backwards clock step mid-window). Decision 5 gains two rules for the agent's side, as additive detail. The collector measures each window on the monotonic clock beside the wall clock, and a window whose wall-clock length falls short of its monotonic length by more than slewing explains (a second plus 500 parts per million) is cut at the step: it ends at its start plus its monotonic length, its records' instants are held within its bounds, and the next window opens on the stepped clock. Nothing observed is lost, so a cut records no gap. Stamping the end from the stepped clock produced a window ending before it started, which the control plane refuses (issue #86). The reporter drops a window the control plane refuses as malformed instead of requeueing it, and records the loss as a sixth gap kind, window_rejected, over the window's bounds with its records as the count; flow_gaps admits the sixth kind (migration 00011). Requeued as the oldest window, a refused window was resent and refused indefinitely, holding every later window behind it. Every other failure is still requeued and retried with backoff. The core decision stands.

Amendment2026-10-13

(PR #95, a removed owned table, issue #64). Decision 5 gains a seventh gap kind, table_removed, as additive detail. The local kill switch deletes the owned table, and the observation that depends on it and the classification its connection marks carry go with it; the agent then shipped empty windows and nothing qualified them, so a kill-switched host read as a quiet one with no evidence gaps. Every heartbeat now reads whether the table is present (ADR-0015 as amended); the first that finds it gone opens a gap from the last instant the table was known present, the next apply or restart that reinstalls it closes the gap, and while it is open each closing window cuts the gap at its own end and carries the piece, so the loss reaches the control plane while it lasts rather than when it ends. A source's restart interval is still shipped whole, since it ends within the restart backoff. flow_gaps admits the seventh kind (migration 00012). The core decision stands.

Amendment2026-10-13

(PR #95, telemetry liveness, issue #61). Ingest gains one write beside decision 2's resolution, as additive detail: the instant the control plane received each report is stamped on its workload in Postgres (ADR-0017), before the window is validated and never moved backwards, so a report refused for its bounds still counts. The sync stream and the telemetry connection back off independently (ADR-0002, decision 5), and during a control-plane restart the console called a workload Offline, with a counter that read the sync path's last contact, while its windows kept landing every minute. The read model judges a workload's telemetry live while a report arrived within two flow windows of the read, against the same configured window ingest judges skew by, and the console names an offline stream with live telemetry as reconnecting and otherwise counts from the latest contact on either channel. The write is one row update per report. The windows, totals, gaps, and the core decision stand.

Amendment2026-10-16

(PR #125, a silent agent, issue #114). The gaps gain an eighth kind, agent_silent, the one kind the control plane derives itself, as additive detail to decisions 1 and 2's ingest write. An agent process that is down leaves the owned table enforcing (ADR-0011) but observes nothing, and no gap said so: the agent cannot report its own absence, and ADR-0011 keeps its on-disk state to the persisted policy, so there is no recorded instant for a new process to reopen a gap from. The control plane is the party that saw the silence. Flow ingest already stamps every report's receipt on the workload (the 2026-10-13 liveness amendment); the stamping statement now returns, under a row lock, the receipt it replaces, so of two concurrent reports exactly one sees each silence. A report that ends a silence longer than two flow windows — the same allowance after which the read model stops calling the telemetry live, pinned by test — writes one gap: from the previous receipt, the last instant evidence is known to have reached the control plane, to the arriving window's start when that lies within the silence (a restarted agent opens its first window at process start), and to this report's receipt when the report carries no usable window start. A window that reaches back to or before the previous receipt is buffered evidence delivered late, not evidence lost, and derives nothing — so a telemetry-connection outage the agent rode out and a quick agent restart record no gap, and what an overflowed buffer dropped stays the agent's own buffer_overflow gap, never stored twice under two kinds. The sync stream's lifecycle is not a bound: an agent can outlive its last report and then die with buffered windows, so instants the stream was alive past the last report are not instants whose evidence survived. The kind is not one source's loss and counts nothing; a reported gap claiming it is refused at ingest. The write goes through the same store interface (one WriteGaps beside the window write), rides the same key, reads, and retention, and flow_gaps admits the eighth kind (migration 00015). An outage still running has no row; the liveness marker carries it until it ends. The core decision stands.

Amendment2026-10-17

(a quiet workload, issue #144). The 2026-10-16 derivation gains one exception, as a correction of a subsidiary point: no agent_silent gap is derived for a silence one agent process demonstrably ran through. The agent ships only windows that hold something, so a running agent on a workload that receives little goes many windows between reports having lost nothing, and the derivation read every such stretch as an absent agent. Every heartbeat carries the agent process's uptime on its monotonic clock; heartbeat ingest records the heartbeat's receipt and that receipt minus the uptime, the process start on the control plane's clock, together and from heartbeats alone (migration 00016), and the receipt statement reads the pair under its row lock. When the process started at or before the previous receipt and its heartbeat arrived within one flow window of the arriving report, the process that ran at the previous report is the one sending this one — the arriving window was open a window length before it was sent, and two of one agent's processes never overlap — so no restart lies in the silence and whatever that process lost it reports itself. Otherwise the derivation and its bounds stand unchanged: a restarted process's start lies after the silence began, a dead process's last heartbeat is older than a window before the restarted one's first report, and a sync stream that is down leaves no recent heartbeat. This is the process's lifetime, not the sync stream's: the earlier point that the stream's lifecycle is no bound stands, and an agent that dies with buffered windows still opens its gap at the last receipt. Rows the earlier rule wrote for quiet stretches are left to retention, since nothing stored distinguishes them from an outage. The core decision stands.