Primary and secondary circuit breakers with failover
A service reads from a primary backend and can fail over to a secondary backend (for example a leader database and a follower). Put a circuit breaker in front of each backend so that a backend that keeps failing is skipped instead of being called again and again.
Implement two classes.
CircuitBreaker(threshold: int, cooldown: float) guards one backend and has three states:
-
CLOSED: calls are allowed. Each failure adds 1 to a consecutive-failure count, and a success resets the count to 0. When the count reachesthreshold, the breaker becomesOPEN. -
OPEN: calls are refused without calling the backend. Oncecooldownseconds have passed since it opened, the breaker becomesHALF_OPEN. -
HALF_OPEN: one trial call is allowed. If it succeeds, the breaker becomesCLOSED. If it fails, the breaker becomesOPENagain and the cooldown restarts.
DualCircuitBreaker(primary, secondary, threshold, cooldown), where each backend is a function request -> response that raises an exception on failure:
-
call(request, now) -> (backend_name, response): try the primary if its breaker allows it. If the primary is refused or fails, try the secondary the same way. Record every real call's outcome on that backend's breaker. If neither backend returns a response, raiseAllBackendsUnavailable. When both breakers are open, raise it without calling either backend. -
states() -> dictreturns each breaker's current state.
Time is passed in as now (seconds) so the behaviour is deterministic.
Constraints: threshold >= 1 and cooldown > 0. Calls arrive one at a time with non-decreasing now.
Example 1 (threshold=2, cooldown=10). The primary is down and the secondary is up.
call("x", 0) -> ("secondary", ...) primary: 1 failure, closed
call("y", 1) -> ("secondary", ...) primary: 2 failures, now open
call("z", 2) -> ("secondary", ...) primary is not called
Example 2. Continue from Example 1. The primary recovers at t = 5.
call("w", 8) -> ("secondary", ...) primary still open (opened at 1, cooldown 10)
call("v", 11) -> ("primary", ...) trial call succeeds, primary closed again
Reconstructed from the reported question; details may differ from what interviewers ask.
Sign in to see the worked answer
The full answer, chapter by chapter, with what interviewers expect at each level. Free with an account: one full answer per company.
