Fault-tolerant work queue with leases and retries
Implement an in-memory work queue that keeps working when workers crash. Workers take a task, process it, and report success or failure. A worker that dies silently must not lose the task: every reservation is a lease that runs out.
class WorkQueue:
def __init__(self, lease_seconds: int, max_attempts: int): ...
def add(self, task_id: str, now: int) -> bool: ...
def reserve(self, now: int) -> str | None: ...
def complete(self, task_id: str, now: int) -> bool: ...
def fail(self, task_id: str, now: int) -> str | None: ...
def status(self, task_id: str, now: int) -> str | None: ...
Every call passes the current time now. Times never go down between calls.
Rules
- A task is in one of four states:
AVAILABLE,RESERVED,COMPLETED,DEAD. -
addputs a new task at the back of the available line and returnsTrue. Adding an id that already exists returnsFalseand changes nothing. -
reservetakes the task at the front of the available line, marks itRESERVED, counts one attempt, and returns its id. The lease ends atnow + lease_seconds. ReturnNoneif no task is available. -
completesucceeds (returnsTrue) only if the task is currentlyRESERVED; it becomesCOMPLETED. Otherwise returnFalse. -
failworks only on aRESERVEDtask (otherwise returnNone). If the task has used fewer thanmax_attemptsattempts, it goes to the back of the available line andfailreturns"AVAILABLE". If it has used all of them, it becomesDEADandfailreturns"DEAD". -
Lease expiry: before handling any call at time
now, every lease whose end time is<= nowexpires. An expired lease is handled exactly likefail(retry or dead-letter). Leases that end at the same time expire in the order they were reserved. -
statusreturns the task's state, orNonefor an unknown id.
Aim for O(log n) or better per call. Talk through how you find expired leases without scanning every task.
Constraints: up to 2 * 10^5 calls; times up to 10^9; 1 <= max_attempts <= 10^5; 0 <= lease_seconds <= 10^9.
Example 1
q = WorkQueue(lease_seconds=30, max_attempts=3)
q.add("resize-7", 0) -> True
q.add("resize-8", 0) -> True
q.reserve(5) -> "resize-7"
q.fail("resize-7", 9) -> "AVAILABLE" # retried, now behind resize-8
q.reserve(10) -> "resize-8"
q.reserve(11) -> "resize-7"
q.complete("resize-8", 20) -> True
Example 2
q = WorkQueue(lease_seconds=4, max_attempts=2)
q.add("emb-1", 0) -> True
q.reserve(1) -> "emb-1" # lease ends at 5
q.status("emb-1", 5) -> "AVAILABLE" # the worker went silent
q.reserve(6) -> "emb-1" # second attempt, ends at 10
q.complete("emb-1", 11) -> False # too late: the lease expired at 10
q.status("emb-1", 11) -> "DEAD"
Sign in to see the worked answer
The full answer, chapter by chapter, with what interviewers expect at each level. Free with an account: one full answer per company.
