HA qualification
The HA qualification suite in test/hachaos injects faults into a complete HA deployment.
It checks the safety contract described in High availability.
It’s the release gate for HA, tracked in #352.
Run it
Section titled “Run it”The suite writes client-go’s in-cluster service-account files, which needs root, so it runs in a disposable container:
Each run writes report.md and results.json to .hachaos/artifacts.
For each scenario it also writes every process log, a timeline of faults and phase changes, and the client ledger there.
The report counts the HTTP 503 responses the client workload received between the fault and the end of recovery.
.crow/ha-qualification.yaml runs the ci tier on changes to HA code, split into five parallel groups.
A shared build step downloads the control-plane tools and compiles Zendrite, the upgrade binaries, and the test suite once.
The groups then run in separate containers against those binaries.
The upgrade binaries are the same source with another version string and HA upgrade contract:
zendrite-nextis the adjacent level, which a rolling upgrade admits.zendrite-unadmittedis a level that needs the offline upgrade.
The build sets their contracts at link time through setup/ha.contractOverride, which configuration can’t change.
HACHAOS_PHASE=build prepares the shared tools, and HACHAOS_PHASE=run runs them without compiling or downloading Go modules.
The default all phase does both for local runs.
The script logs tool preparation and build durations, and the suite reports each scenario’s duration.
Manual runs and the ha-qualification cron can select the full or reference tier with HACHAOS_TIER, and publish their artifacts to the package registry.
What runs
Section titled “What runs”Every scenario gets a fresh deployment of real processes on one host:
- A PostgreSQL primary with a synchronous standby behind a stable address. Candidates see promotion the way they see it through an operator’s read-write Service.
- A three-node JetStream cluster with file storage and
sync_interval: always. - An S3 server for media.
- etcd and a kube-apiserver whose RBAC grants candidates only
get,create, andupdateon Leases. - Three Zendrite candidates with HA enabled. A supervisor restarts a process that exits, as the kubelet restarts a container.
- A router that checks the active-only endpoint every 500 ms and cuts in-flight requests when a backend fails, like the chart’s HAProxy routing layer. A backend that answers 404 gets no new requests but keeps the ones in flight. As in the chart’s router, a request that arrives while no backend is up waits for up to 30 seconds.
- A standalone homeserver joined to the test room, so federation runs in both directions.
Candidates connect to every dependency through fault proxies. A proxy attributes each connection to the candidate that opened it. So a partition can apply to one candidate while the others keep working, and it can drop traffic in one direction only. Federation requests pass through a gateway that terminates TLS, so the suite can record every request with the process that sent it.
Workload
Section titled “Workload”While a fault is active, clients keep working the way a careful client does:
- A writer sends messages and retries each one with its original transaction ID until the server acknowledges it.
- A syncing client long-polls with its existing token across takeover.
- To-device messages, read receipts, media uploads and downloads, and search run alongside.
- A user on the remote server writes to the same room and syncs to receive local messages over federation. The remote user also sends to-device messages and read receipts to the local user. These reach the HA server only as EDUs in federation transactions.
- Every server runs with the HA-qualified MSCs
msc2444,msc2753,msc2836, andmsc4194enabled. The HA candidates also enablemsc3861.HACHAOS_MSCS=noneturns them off. - With
msc3861, the candidates delegate authentication to a mock MAS in the suite. The HA server’s users get their tokens from it instead of registering. - An MSC probe sends thread replies into world-readable room version 11 and 12 rooms.
A local user peeks into them through
/sync, and a remote user peeks into them over federation. The local user also peeks into a room on the remote server, where the remote user writes. - An OIDC probe signs new users in and rotates signed-in users’ tokens. It provisions half of them through the MAS API first, and the other half on demand by their first request.
- An MSC4194 probe has a local user write into a room of its own.
Meanwhile the room’s creator keeps redacting that user’s events with the MSC4194 endpoint, two per request, paging while
is_more_eventsis true. So faults hit the moderator halfway through a batch.
What the suite checks
Section titled “What the suite checks”After the fault heals, the suite compares what clients were told with what the servers store:
- Every acknowledged write is in the room exactly once. A request that failed without an answer may have committed, which is allowed, but it must not produce two events.
- Every acknowledged write reaches the remote server, and reaches the syncing client with the token it had before the fault.
/event_relationshipsreturns every acknowledged thread reply. Each reply reaches the local and remote peekers with the tokens they had before the fault. Every acknowledged write in the remote room reaches the local peeker.- Once the moderator has redacted what’s left, every acknowledged message of the MSC4194 probe’s user is redacted, and the moderator’s own message isn’t. The report lists events redacted more than once by a request retried after its answer was lost.
- Every acknowledged OIDC sign-in still authenticates as its user and device. It left exactly one account, subject mapping, and device, with the display name MAS provisioned.
- Every acknowledged to-device message is delivered. The newest acknowledged receipt (or one still in flight when the workload stopped) is visible. Media uploaded before the fault is intact, and search finds messages from before and after it.
- The same holds for the remote user’s to-device messages and receipts, which arrive over federation. Receipts coalesce, so only the newest acknowledged one must be visible. Delivery is at least once, so the report counts and lists to-device messages delivered twice.
- Every stream and durable consumer has a leader at the end. A scenario that replaced a broker reports how long its replicas took to catch up.
- A follower that is still behind, while its stream keeps a leader and a current majority, is noted with the broker’s store errors rather than failed. nats-server can leave one behind after a broker rejoins on its old storage (#473), and Zendrite can’t repair that. When that broker logged the store error, the suite restarts it on its storage, as the runbook prescribes. The scenario fails if its replicas don’t catch up then.
The gateway records each federation transaction from the remote server with its contents and what the sender was told. The report counts:
- transaction attempts and distinct transactions
- the transactions, distinct to-device messages, and receipts affected by a failed attempt
- the candidates’ logged publication failures, with the distinct items they concerned
So a burst of retries of one transaction doesn’t read as many lost items.
When an item is missing, missing-federated-*.txt lists every attempt that included it.
The suite also samples the publish, consumer, and transaction metrics of the serving candidate and reports how much they moved.
The suite also checks ownership at every protected boundary:
- Database: A trigger records the activation marker of every transaction that writes an application table, and commit timestamps order them. The commits of one generation must all precede the first commit of any later generation.
- Broker, media, and federation: The suite attributes publications, S3 writes, and outbound federation requests to a generation. A retired generation may act only until it can notice its successor, within one heartbeat interval.
- Routing: No two candidates may pass the active-only check at the same time.
- Resumed owners: A candidate resumed after a successor took over must refuse writes sent to it directly.
The suite measures recovery time from fault injection to the first successful client write, sync, inbound federation request, and outbound federation request. It compares that with the 120 second objective.
Planned operations, rolling-restart and rolling-upgrade, must meet a stricter claim than recovery.
Every client request and every inbound federation request that overlapped the operation must succeed.
A request that succeeded only when retried counts as a failure.
The report lists these operations separately, with:
- the maximum and 99th percentile write latency
- the outbound federation requests the successor had to retry
- how long search answered 503 while the successor rebuilt its index, which is the documented exception
Scenarios
Section titled “Scenarios”| Scenario | Fault |
|---|---|
leader-sigkill |
The active candidate is killed. |
leader-graceful-shutdown |
The active candidate receives SIGTERM. |
leader-paused-beyond-lease |
The active candidate is frozen until a successor serves, then resumed. |
rolling-restart |
Every candidate restarts in turn, the active one last, and no request may fail. |
leader-partitioned-from-kubernetes |
The active candidate loses only the Kubernetes API. |
leader-kubernetes-responses-lost |
Lease renewals arrive but their responses are lost. |
leader-partitioned-from-database |
The active candidate loses only PostgreSQL. |
leader-partitioned-from-everything |
The active candidate is isolated from every dependency. |
kubernetes-api-outage |
The kube-apiserver is frozen for longer than the Lease. |
postgres-primary-failover |
The PostgreSQL primary is killed and its standby promoted. |
nats-stream-leader-loss |
The broker leading the room input stream is killed. |
media-store-outage |
The S3 server is frozen. |
fencing-cannot-be-proven |
A PostgreSQL backend of the old owner can’t be terminated, so promotion must stay blocked. |
leadership-lost-during-startup |
The successor loses the Kubernetes API while fencing or recovering. |
queued-federation-delivery |
Federation is cut, events queue, and the active candidate is killed. |
coordinated-upgrade |
A build the upgrade contract doesn’t admit must not become prepared or take the released Lease. The maintenance-window upgrade must keep every write. |
rolling-upgrade |
Every candidate is replaced by the next level, the active one last, and no request may fail. |
rolling-upgrade-successor-fails |
The candidate taking over during a rolling upgrade is killed while it activates. |
rolling-upgrade-broker-leader-lost |
The broker leading the room input stream is killed while the owner hands over during a rolling upgrade. |
msc-relationship-index-failure |
PostgreSQL rejects MSC2836 relationship index writes, and the active candidate is replaced before they’re allowed again. |
msc3861-takeover-intermediates |
With delegated authentication, the active candidate is frozen and replaced while a confirmed cross-signing reset, an unredeemed SSO login token, and MAS provisioning are in flight. The successor must challenge the reset and refuse the login token, retried flows must succeed, and the resumed old owner must act on neither. |
nats-broker-host-frozen |
The broker the active candidate uses stops responding without closing connections, as on a host that goes NotReady. After an extended absence, it’s replaced on its storage. |
nats-broker-replaced-empty |
The receipt stream leader is killed. After an extended absence, a broker with empty storage replaces it and must catch up. |
nats-publish-acks-lost |
Broker traffic towards the active candidate is held for 15 seconds, so every publication is ambiguous until it’s retried. |
The broker scenarios keep each broker away for 90 seconds in the ci tier and four minutes otherwise.
During the first two, the remote server abandons a transaction after 20 seconds, as a sender’s request timeout does.
This cancels the request while the HA server is still publishing.
Losing one broker of three needs no takeover, so these scenarios have a 60 second recovery objective.
Limitations
Section titled “Limitations”The suite trades fidelity for reproducibility in a few places:
- Every process shares one host and one clock, and proxies emulate partitions.
- The routing layer reimplements the chart’s HAProxy check and hold rather than running HAProxy.
- Both upgrade builds share the release’s schema. So a rolling upgrade exercises the handover, the upgrade contract, and the successor’s recovery, but no migration. Also try an upgrade between real releases on a staging deployment.
- The S3 server is gofakes3, so the suite doesn’t exercise S3 failover itself.
- The PostgreSQL operator is emulated: the suite promotes the standby and relaxes synchronous commit while no standby exists.
- Power loss and correlated storage failure are out of scope for a process-level suite.