Design an access management system

Decide who can do what to which resource across many services: roles, groups, fine-grained permissions, fast checks on every request, and auditable changes.

We are building a central authorization service: many internal services ask it "can principal P do action A on resource R?" on every request, and admins manage roles, groups and grants through it with a full audit trail. Authentication (who you are) is out of scope; it arrives as a verified token.

Functional requirements

• Principals are users and service accounts; users belong to groups, and groups can nest (a team inside a department).
• Permissions are service:action strings (payments:refund, warehouse:query); roles are named bundles of permissions.
• Bind a role to a principal or group on a scope: the whole org, a project, or one resource (account/acct_42). Grants on a scope apply to everything under it.
• check(principal, permission, resource) returns allow/deny, plus the reason (which binding granted it) for debugging.
• List effective permissions for a principal, and list who can do X on resource R (for access reviews).
• Optional explicit deny bindings that override any allow; optional expiry on a binding for temporary access.
• Every mutation is audited: who changed what, when, why, and the before/after state.

Non-functional requirements

• Check latency under 5 ms p99 from the caller's view, under 1 ms when served from a local cache. It sits on every request.
• 99.99% availability for checks; management writes can tolerate brief unavailability.
• Revocation propagates everywhere in seconds (target p99 < 5 s); a fired employee must lose access fast.
• No stale "allow" survives longer than a bounded window, and callers can ask for a fresh-as-of-my-write answer.
• Audit log is append-only, tamper-evident and kept for 7 years (SOC 2, PCI).

Back-of-the-envelope estimation

• 50k orgs, 2M principals, 50k groups, 500 permissions, 5k roles (built-in plus custom), 20M bindings.
• Checks: 200k/s average, 1M/s peak across ~300 calling services. Writes: ~50/s, bursts of 5k/s during SCIM syncs.
• Read:write ratio is about 10,000:1, so precompute on write and keep checks cheap.
• Effective permissions per principal: ~50 roles × ~40 permissions, deduped to ~300 entries × ~64 B ≈ 20 KB. 2M principals × 20 KB ≈ 40 GB materialized; fits in a sharded Redis or a KV store with replicas.
• Audit: 50 writes/s × 2 KB ≈ 9 GB/day ≈ 3 TB/year before compression.

Check (hot path, gRPC):

Check(principal="user_123", permission="payments:refund",
      resource="org_9/account/acct_42", consistency_token?)
→ {allowed: true, reason: {binding_id: "b_77", role: "refund_agent", scope: "org_9/account"},
   decision_version: 81234}
BatchCheck([...up to 100 tuples]) → [{allowed, reason}]

Management (REST, admin-authenticated, all writes take an Idempotency-Key and a required reason):

• PUT /v1/roles/{role_id}/permissions/{permission} binds a permission to a role → {role_version, zookie}.
• POST /v1/bindings {principal|group, role_id, scope, effect: allow|deny, expires_at?} → {binding_id, zookie}.
• DELETE /v1/bindings/{id} → {zookie}.
• PUT /v1/groups/{id}/members/{principal} and DELETE for membership.
• GET /v1/principals/{id}/effective-permissions?scope=... → [{permission, scope, via: [binding_id, group_path]}].
• GET /v1/resources/{r}/access?permission=... lists principals who can act (access reviews).
• GET /v1/audit?actor=&target=&from=&to= pages through the audit log.

Recovery contract: a write returns a zookie (an opaque change sequence number). A caller that just granted access and wants to use it passes the zookie to Check; the server answers only from a cache at least that fresh, otherwise it falls through to the source of truth. Without a zookie the answer may be up to 5 s stale. On authz-service failure, callers get a typed UNAVAILABLE, and each service applies its declared fail policy (below).

Source of truth in a relational database (Postgres or MySQL, sharded by org_id): transactions keep a binding and its audit row atomic, and the graph is small per org.

Scopes are materialized paths (org_9/project_3/account/acct_42), so "grants on ancestors" is a prefix check.

The naive effective-permission query joins bindings → role_permissions, unioned over the principal's transitive groups (a recursive CTE over group_members), filtered to scopes that are prefixes of the target resource. That is 3–5 joins plus a recursion: fine for admin pages, far too slow for 1M checks/s.

Materialized view (the hot path), in a sharded KV (Redis Cluster or DynamoDB), keyed by principal:

ep:{org_id}:{principal_id} → {version: 81234,
  allow: {"payments:refund": ["org_9/account"], "payments:read": ["org_9"]},
  deny:  {"payments:refund": ["org_9/account/acct_13"]}}

Storage trade-off: the relational store gives correct joins and transactions, the KV gives O(1) reads. We keep both and accept that the KV is a derived, rebuildable cache.

flowchart LR
  SVC[Calling service] --> SDK[Authz SDK with local cache]
  SDK -->|miss| CHK[Check service]
  CHK --> EP[(Effective permission KV)]
  CHK -.->|fallback| DB[(Relational source of truth)]
  ADM[Admin console and SCIM] --> MGMT[Management API]
  MGMT --> DB
  DB -->|outbox| LOG[Change stream]
  LOG --> MAT[Materializer]
  MAT --> EP
  LOG --> INV[Invalidation fanout]
  INV -.-> SDK
  MGMT --> AUD[(Audit log store)]

1. Check. The service calls the SDK in-process. The SDK looks up the principal's effective-permission entry in a local LRU (TTL 30 s, bounded by invalidations). On a hit it walks the resource path prefixes against allow and deny in microseconds. On a miss it calls the Check service, which reads one KV key (~1 ms) and returns the decision plus the entry to cache.
2. Fresh check. If the caller passes a zookie newer than the cached version, the SDK skips the local cache; the Check service skips the KV if it is behind, and computes from the database directly.
3. Management write. The Management API authorizes the admin (the service checks itself with iam:manage_bindings), then in one transaction writes the binding, a changelog row and an audit_events row. It returns the changelog seq as the zookie.
4. Propagation. A CDC reader turns changelog rows into events on a partitioned stream (Kafka, keyed by org_id). The Materializer recomputes affected principals and writes their new KV entries; the invalidation fanout tells SDK caches to drop those principals.
5. Audit. Audit rows are also shipped to WORM object storage and a search index for the compliance UI.

Materialization and the check hot path. Precompute on write, because writes are 10,000× rarer than reads. The work is the fan-out: which principals does a change touch?

To keep the big fan-outs cheap, store two layers: per-principal {roles by scope, groups} (small, changes rarely) and per-role permission sets (shared, cached separately, versioned). The check becomes "principal's role list ∩ role's permission set", two lookups instead of one, but adding a permission to a role updates one role entry, not 2M principal entries. Group-to-role bindings are expanded per principal because group trees are usually shallow (cap nesting depth at 5). Large groups (>10k members) keep a single group entry that member lookups reference instead of being copied into each principal.

The check itself: for payments:refund on org_9/account/acct_42, walk the 3 prefixes, return deny if any deny matches, else allow if any allow matches, else default deny. Expiring bindings carry expires_at in the entry and are evaluated at check time, so no job has to race the clock.

Cache invalidation on permission changes. Three cache layers (SDK in-process, Check-service, KV) all must forget a revoked grant within 5 s.

• Event-driven: the invalidation fanout publishes {org_id, principal_ids | role_id, seq} over a pub/sub channel each SDK instance subscribes to (or a long-poll stream). SDKs drop the matching entries and record last_seen_seq.
• Safety net: SDK TTL of 30 s, so a missed message bounds staleness. If an SDK detects a gap in seq or loses its subscription for more than 10 s, it flushes its whole cache.
• Ordering: the Materializer writes the KV with a compare-and-set on version so an older recompute never overwrites a newer one; invalidation is sent after the KV write, so a refill never reloads the old value.
• New-enemy problem: if Alice is removed from a doc and then the doc gets new content, a stale check could leak it. Zookies fix this for callers that care: content writes record the authz seq they were checked at, and reads demand at least that seq.
• Emergency revocation (fired employee, leaked key): disable the principal, which writes a tombstone the SDK checks before anything else and pushes a high-priority invalidation.

Fail-open vs fail-closed. Each service declares its policy. Default is fail closed: deny on error, because a breach is worse than an outage for payments or data warehouses. To keep that from becoming an outage, the SDK serves stale entries for up to 15 minutes when the Check service is unreachable ("stale-if-error"), but never for principals with a tombstone. Fail open is allowed only for read-only, low-sensitivity endpoints (status pages, docs), and every fail-open decision is logged. Run the Check service in every region with its own KV replica so a regional outage doesn't cascade.

Audit logging. The audit row commits in the same transaction as the change, so no change escapes the log. Each row stores prev_hash = H(prev row) per org, and a daily root hash is written to WORM storage (S3 Object Lock), so tampering is detectable. Checks themselves are sampled (1%) plus all denies and all admin-scope checks, since logging 1M/s in full costs ~170 TB/day.

From RBAC to relationship-based access (Zanzibar). RBAC breaks down when permissions follow data relationships: "editors of a folder can edit its docs", "members of the account's team can view its charges". Zanzibar stores tuples object#relation@subject (doc:7#parent@folder:3, folder:3#editor@group:eng#member) and a per-namespace schema that says doc.edit = editor + parent->edit. A check becomes a bounded graph walk with caching of sub-results, and the zookie model above is exactly Zanzibar's. Our RBAC fits as a special case: a binding is scope#role@principal. Move to it when resource-level sharing is common; until then, roles over scope paths are simpler to reason about and audit.

• Materializing effective permissions makes checks O(1) but turns some writes into large fan-outs and adds seconds of propagation; the two-layer layout limits the blast radius at the cost of a second lookup per check.
• In-process SDK caches give sub-millisecond checks and survive outages, but every cache is a place a revoked grant can linger; we pay with an invalidation stream and a 30 s TTL ceiling.
• Fail closed protects data but can turn an authz outage into a full outage; stale-if-error softens that at the cost of up to 15 minutes of stale allows for non-revoked principals.
• Explicit deny makes emergency lockouts easy but makes "why can't I?" harder to explain; the reason field on every decision is how we pay that back.
• Relational source of truth keeps writes and audit transactional but limits write throughput per org shard; acceptable at ~50 writes/s.
• Pure RBAC over scope paths is easy to audit; Zanzibar-style relationships are more expressive but harder to review ("who can see this?" needs a reverse graph walk).

Mid-level (L4)

• Clear RBAC model: principals, groups, roles, permissions, bindings, with a working schema.
• A check API and a correct evaluation rule (deny overrides, default deny).
• Notices the join-heavy query is too slow and adds a cache.

Senior (L5)

• Precomputes effective permissions and reasons about write fan-out by change type.
• Designs invalidation with a change stream, versioned entries and a TTL backstop, and quantifies staleness.
• Makes the fail-open/fail-closed decision per service and keeps audit writes transactional.

Staff (L6+)

• Brings in consistency tokens (zookies) and the new-enemy problem, and shows how callers opt into fresh checks.
• Plans for SCIM bursts, built-in role edits across all orgs, emergency revocation and multi-region.
• Knows when RBAC stops fitting and explains how the design evolves to relationship-based access without a rewrite.