Credential lifecycle

Renewal

• Lifetime. A credential lives 24 hours (innerwall serve --leaf-ttl).
• When it renews. The daemon renews at two thirds of the lifetime, less a random jitter of up to a twelfth of it, so a 24-hour credential renews between about 14 and 16 hours after issue and a fleet's renewals spread out instead of arriving together.
• Two clocks. The renewal is scheduled on the wall clock and on the time actually elapsed since the agent obtained the credential, whichever comes first. A host whose wall clock is wrong or steps backwards still renews on time; when elapsed time wins by more than a minute the daemon logs the wall clock is behind the credential's age; renewing on elapsed time instead.
• When it fails. A failed renewal is logged (credential renewal failed; will retry), retried with backoff, never past the credential's expiry, and reported on the agent's heartbeat, so the workload page and innerwall workload status show the renewal error.

Renewal needs nothing from the operator. The rest of this guide is about what happens when it cannot reach the control plane in time.

Expiry is never fatal

A credential expires when the control plane was unreachable for the whole stretch between the renewal point and expiry (roughly eight to ten hours), or the agent was stopped across it. The agent does not stop for it:

• Enforcement continues. The daemon keeps enforcing its persisted policy, at start or while running (fail static, ADR-0011), and logs that the credential has expired.
• Renewal keeps trying: at once, then with jittered backoff, never more than about five minutes between attempts.
• Flow reports are refused while the credential is expired, because each report opens a new connection that must authenticate. The daemon holds the closed windows in its bounded buffer (--flow-buffer-records, 50,000 by default) and delivers them once it has renewed; beyond the buffer, the oldest are dropped and recorded as a buffer_overflow evidence gap.
• A sync stream already open survives. A stream opened on a valid credential is authenticated when it opens and stays up past the credential's expiry for as long as its connection lasts. A new stream after expiry is refused until the credential is renewed.

The renewal grace

The control plane accepts an expired credential for one thing only, renewal, for --renewal-grace after its expiry: 72 hours by default, set on innerwall serve. So an outage of up to about three days heals by itself when the control plane returns: each agent renews on its next attempt and carries on.

• Within the grace, any other call is refused with workload credential has expired: it expired at …, … ago; an expired credential may only renew, which it may for … more.
• A late renewal is recorded. The control plane logs credential renewed late, within the renewal grace at warning level with how late it was; the agent logs credential renewed after it had expired. The workload keeps the record (health.credential.last_late_renewal in the API): the console shows how late the renewal was, and innerwall workload status prints renewed late.
• After a late renewal, give it a few minutes. The sync stream and the flow reporter take the renewed credential on their next connection, which comes when their current reconnect backoff ends: up to about five minutes. Until then the workload may still read offline or show stale state; nothing on the host is affected.

Beyond the grace: re-enrollment

Past the grace, the renewal is refused too: workload credential has expired: it expired at …, … ago, beyond the 72h0m0s renewal grace; re-enrollment with a new provisioning token is required. The daemon still enforces from disk and still retries, and logs that refusal on every attempt; it does not stop by itself. The workload can only be re-enrolled:

1. Mint a provisioning token with the labels the host should carry (the console's Mint token, or innerwall token mint). The new workload takes the token's labels, not the old workload's.
2. Enroll over the expired credential. On the host, run innerwall-agent enroll with the new token, the same --server and --bootstrap-ca as the first enrollment, and the daemon's --state-dir. Enrollment overwrites an expired credential, and says whose: replaced the expired credential of innerwall://workload/… (expired …). It refuses a valid one.
3. Restart the daemon. A daemon still running over the state directory notices the new key the next time its renewer wakes (within about five minutes while expired), logs that the state directory was enrolled again underneath it, and exits so that its restart runs as the new workload. Restart it, or let the service manager do so.
4. Set the mode again. The new workload starts in visibility, like every new workload: on its first snapshot the host's table becomes the observe-only table, and nothing is enforced until you move it to simulation and promote it again.

Rotating a credential by hand

innerwall-agent renew renews the credential in a state directory once, from the command line. Use it only on a host whose daemon is not running.

Related

• What the console shows for each credential state: states.
• Why identity lives only in the certificate, and why labels never do: ADR-0016.
• Agents on hosts whose clocks drift: troubleshooting.