Backups and disaster recovery

Recovery boundaries

A backup is a logical export of the managed auth:* and fleet:* SableDB keyspaces. It is not a SableDB volume image, continuous replication or a transaction log. Realm and Fleet deployments produce separate objects because they have separate owners, datastores and failure domains.

Realm workspace

Fleet workspace

The exporter scans both managed prefixes for either deployment role. A Fleet backup therefore also contains every applicable central auth:* record, including its signing and operator identity state, alongside the Fleet-specific families above.

Excluded and external state

Short-lived WebAuthn ceremonies, agent handoffs, Fleet pairing/connection attempts, maintenance locks, backup health, leases and restore markers are deliberately excluded so recovery does not replay an in-progress operation. An unknown durable key family fails snapshot creation instead of being silently omitted.

Browser navigation and filters are transient. The Dioxus dashboard assets come from the pinned release image; durable dashboard settings are the organization, operator, service-account or Fleet records above. Retain the image digest, validated YAML, active and previous master/backup keys, bucket IaC and downstream application data in separate recovery systems.

Creation pipeline

1. Startup, the interval scheduler or backup create takes the in-process operation lock and a one-hour SableDB lease.
2. The exporter takes the process-local mutation gate, records capture time, scans managed keys and reads each value with its TTL.
3. Records are sorted and validated; the signing keyset must also be recoverable with the configured master-key ring.
4. A stable Postcard DTO is compressed with Zstandard level 3 and sealed with a random-nonce AES-256-GCM envelope.
5. RustyAuth uploads with an SHA-256 transport checksum under the tenant's v3 prefix.
6. GetObject must return the same decryptable snapshot plus the required version, retention and server-side-encryption posture.
7. The scheduler persists the successful object key and resets its failure count.

Manifest validation

The authenticated snapshot stores its UUID, tenant, capture time, sorted records, record count, SHA-256 content digest, key-family counts and ordered-event sequence. Validation runs before upload and after every download.

RustyAuth rejects tenant crossing, duplicate or unknown keys, digest/count disagreement, missing signing state, malformed expiries, event gaps, orphaned identity indexes, unknown session/operator owners, broken service credentials, inconsistent Realm-to-Fleet locators, cross-workspace Fleet relationships, invalid role-binding indexes and audit/idempotency disagreement.

Binary .rauth format

New objects use RAUTHBK3. Protobuf/gRPC remains the application transport; it is not the durable backup format. Postcard removes repeated JSON field names, Zstandard compresses the structured data, and encryption happens last because ciphertext is intentionally incompressible.

The key ID selects the active or a previous key without storing key material. Header or payload tampering, truncation and use of the wrong key all fail authentication before decompression. Existing RAUTHBK2 compressed-JSON objects remain restorable; pre-restorable PAUTHBK1 payloads are rejected.

S3 object and immutability contract

rustyauth-backups/v3/<tenant-id>/<capture-time>-<snapshot-uuid>.rauth

The v3 object uses content type application/vnd.rustyauth.backup.v3 and records non-secret snapshot ID, derived key ID, format version and complete-server-workspace scope metadata. Reads are confined to the configured tenant's v2/v3 prefixes.

The default immutable profile requires every new object to return a version ID, COMPLIANCE Object Lock, retention through at least capture time plus the configured policy, and the configured SSE mode. With aws-kms, RustyAuth can also require the exact KMS key ID returned by the provider. provider encryption skips only the exact SSE-header comparison.

The explicit portable profile supports providers without Versioning or Object Lock. It retains unique keys, the authenticated application envelope, checksum, manifest validation and decrypting read-back, but it does not claim WORM retention.

Deploy the repository's immutable AWS bucket stack, or reproduce the same guarantees with a compatible provider. Legacy v2 compatibility does not prove that an old object had these storage controls.

Configuration and keys

backups:
  enabled: true
  destination:
    endpoint: https://s3.eu-west-2.amazonaws.com
    region: eu-west-2
    bucket: example-rustyauth-backups
    urlStyle: virtual
    storageProfile: immutable
    serverSideEncryption:
      mode: aws-kms
      kmsKeyId: arn:aws:kms:eu-west-2:123456789012:key/<key-id>
  schedule:
    interval: 6h
    recoveryPointObjective: 6h
  retention: 90d
  alertAfterFailures: 2

Supply AUTH_BACKUP_ACCESS_KEY_ID, AUTH_BACKUP_SECRET_ACCESS_KEY and an independent 32-byte AUTH_BACKUP_ENCRYPTION_KEY_HEX through the secret store or their _FILE inputs. Put former application keys in AUTH_BACKUP_PREVIOUS_KEYS_HEX. Partial configuration and repeated-byte placeholder keys fail startup.

The defaults are a six-hour interval/RPO, 90 days immutable retention, alert after two failures and aws-kms provider encryption. The interval may be 5 minutes–7 days; the RPO may be the interval–30 days; retention may be 1–3,650 whole days.

Operator commands and receipts

Create and verify return JSON containing format version, snapshot/object identifiers, capture time, record and envelope sizes, derived encryption-key ID, storage profile, provider version ID, retain-until time and reported SSE mode. Retain these receipts with deployment and drill evidence.

Scheduler health and alerting

The first scheduler tick runs after startup and later ticks use delay semantics, so a stalled provider does not create a burst of catch-up uploads. Health survives restarts in an operational SableDB record excluded from snapshots. It tracks running state, last attempt/success, last object, consecutive failures, RPO, retention, overdue and alerting.

Alerting begins when the last success exceeds the RPO or the failure threshold is reached. backup status and doctor print JSON and exit non-zero while alerting. Schedule one of them from the host and page on failure; backup_health_alert=true is supporting telemetry, not the only alert path. Public discovery and readiness deliberately omit backup posture.

Clean-room restore

Prepare a pinned compatible image, exact tenant/role configuration, the selected object receipt, bucket/KMS access, active and previous backup/master keys, and a new private SableDB volume with no managed records. Never restore over a live namespace.

rustyauth backup list
rustyauth backup verify <object-key>
rustyauth backup restore <object-key>
rustyauth doctor
rustyauth keys status
rustyauth operator list

1. verify proves storage posture, envelope authentication, tenant, manifest relationships and signing-key recoverability without writing.
2. restore writes an in-progress sentinel, then restores records in atomic batches of at most 250 and skips already expired TTL records.
3. By default it skips every stored session and increments each user's session generation.
4. RustyAuth loads the recovered signing state, forces a fresh signing-key rotation and appends recovery.restored.
5. The sentinel is removed only when every security step succeeds. A remaining sentinel blocks normal startup.

Before promotion, verify readiness, discovery/JWKS, a real synthetic passkey sign-in and the role-specific workspace: Realm identities/settings/operators/events or Fleet hierarchy/connections/roles/audit. Record the restored count, new signing key and test result.

Rotation and recovery drills

To rotate the portable backup key, make a new random key active, retain the former key in AUTH_BACKUP_PREVIOUS_KEYS_HEX, restart every instance, create and verify a new object, then complete a clean-room restore. Keep old keys until every object they protect has expired and the drill evidence is retained.

Run a monthly external drill against a real retained provider object. A passing drill downloads both encryption layers, restores a new volume, passes doctor and completes an end-to-end passkey operation. Page on the job's non-zero exit—the production process cannot attest that an isolated job ran elsewhere.

Normative repository reference

The exhaustive contract, including the complete inclusion/exclusion boundary, envelope layout, S3 metadata, limits, IAM policy, key rotation, command receipts, compatibility rules and integration-test command, lives in Backups and disaster recovery. Configuration source precedence is in Configuration; service placement is in Deployment.