One thing I have been paying more attention to in backend systems is how they behave when something they depend on stops behaving normally.
\nAn API call to an external service can fail for many reasons. The dependency might be temporarily unavailable, slow to respond, returning errors, or failing consistently.
\nThe first instinct is often simple:
\n“Try again.”
\nSometimes that is exactly what the system needs.
\nBut retrying is not a complete failure strategy.
\nIf the dependency is already struggling, repeatedly sending the same request can create even more traffic. A temporary failure can turn into a larger problem if every caller keeps retrying without any limits.
\nThat is where I started thinking about retries, circuit breakers, and fallbacks as three different tools rather than three interchangeable patterns.
\nA retry asks:
\n“Should I try this operation again?”
\nA circuit breaker asks:
\n“Should I keep calling this dependency at all right now?”
\nA fallback asks:
\n“If the dependency cannot complete this operation, what can I do instead?”
\nThose are different questions.
\nRetries are useful when a failure is likely to be temporary.
\nFor example, a network request might fail because of a short-lived connection problem or a temporary interruption in the dependency.
\nInstead of immediately failing the request, the application can try the operation again.
\nConceptually:
\nRetry (conceptual)
\nThe important part is that retries need boundaries.
\nA retry policy should consider things such as:
\nThe last question is particularly important.
\nRetrying a read is generally a different problem from retrying an operation that changes state. If an operation creates something, charges something, sends something, or otherwise has side effects, blindly repeating it can create a second operation rather than simply recovering the first one.
\nSo the lesson for me is not “add retries.”
\nIt is:
\nRetry when the failure looks transient and the operation can be safely retried.
\nImagine an external dependency is consistently failing.
\nWithout a limit, every incoming request can trigger multiple attempts.
\nOne request becomes several requests to the failing service.
\nAt scale, that can look something like:
\nUnbounded retries (conceptual)
\nThe application is spending more work trying to reach something that is already unavailable.
\nThis is where retries can become part of the problem.
\nA retry mechanism should therefore be designed with the possibility that the dependency may remain unavailable for longer than expected.
\nThat led me to the next question:
\nWhat happens when continuing to call the dependency is no longer useful?
\nA circuit breaker introduces another decision.
\nInstead of allowing every request to keep calling a failing dependency, the application can temporarily stop making those calls after failures cross a defined threshold.
\nConceptually:
\nCircuit breaker (conceptual)
\nYes
\nNo
\nThe useful part is not the name “circuit breaker.”
\nThe useful part is the change in behavior.
\nInstead of repeatedly discovering that the dependency is unavailable, the application can recognize the failure pattern and stop sending unnecessary traffic for a period of time.
\nThat gives the dependency some breathing room and prevents the application from spending resources repeatedly waiting for failures.
\nEventually, the circuit can allow requests through again to determine whether the dependency has recovered.
\nThe important idea is that a circuit breaker is about controlling calls to an unhealthy dependency.
\nA retry is about giving an individual operation another chance.
\nThey solve different problems.
\nI think about the distinction like this:
\n“Maybe this particular failure is temporary.”
\nThe system makes another attempt.
\n“This dependency appears unhealthy right now.”
\nThe system stops repeatedly calling it.
\nThat means they can work together.
\nA request might retry a small number of times for a transient failure, while the circuit breaker prevents the service from continuing to hammer the dependency when failures become persistent.
\nThe exact thresholds and timing depend on the system. There is no universal number that makes a circuit breaker correct.
\nThe important part is understanding what decision the mechanism is making.
\nRetries and circuit breakers answer questions about calling the dependency.
\nA fallback answers a different question:
\n“What should the application do if that dependency cannot provide the result?”
\nSometimes the answer is to return a controlled error.
\nSometimes the application can use previously available data.
\nSometimes a default response is acceptable.
\nSometimes the operation simply cannot continue and the system should communicate that clearly to the caller.
\nConceptually:
\nFallback (conceptual)
\nA fallback should not pretend that the dependency succeeded when it did not.
\nThat distinction matters.
\nReturning stale or partial information might be acceptable for one type of operation and completely wrong for another.
\nThe fallback has to match the business meaning of the operation.
\nThe most useful mental model for me is not:
\n“Which one should I use?”
\nIt is:
\n“What kind of failure am I dealing with, and what should happen next?”
\nA simplified flow looks like this:
\nFailure path (conceptual)
\nThis is only a conceptual model. Real systems can have more states and more detailed policies.
\nThe important part is that each mechanism has a different responsibility.
\nIt is tempting to think of reliability patterns as things you simply add to a service.
\nAdd retries.
\nAdd a circuit breaker.
\nAdd a fallback.
\nDone.
\nBut each one changes system behavior.
\nA retry changes how many times an operation can execute.
\nA circuit breaker changes whether a dependency is called at all.
\nA fallback changes what the user or calling service receives when something fails.
\nThose decisions should therefore be connected to the actual operation.
\nFor example, retrying a request that is safe to repeat is a different problem from retrying an operation with side effects.
\nSimilarly, returning cached data as a fallback might be useful for a read-heavy endpoint but unacceptable when the caller requires the latest authoritative state.
\nReliability is therefore not only about keeping requests alive.
\nIt is also about deciding which behavior is safe when the normal path stops working.
\nOne thing I increasingly appreciate about these patterns is that they protect more than the individual request.
\nA retry can help recover from a short-lived problem.
\nA circuit breaker can prevent a struggling dependency from receiving an increasing number of repeated requests.
\nA fallback can allow the rest of the system to degrade in a controlled way instead of failing unpredictably.
\nThat means failure handling is partly about protecting the dependency and partly about protecting the application itself.
\nThe goal is not necessarily:
\n“Never show an error.”
\nSometimes the correct behavior is a clear, controlled failure.
\nA reliable system is not one where failures never happen.
\nIt is one where failures have predictable behavior.
\nUseful for:
\nTrade-offs:
\nUseful for:
\nTrade-offs:
\nUseful for:
\nTrade-offs:
\nNone of these patterns is universally required.
\nThe architecture should reflect the dependency, the operation being performed, and what failure means to the application.
\nThe biggest shift for me was moving away from thinking about retries as the default answer to every external-service failure.
\nSometimes a retry is exactly right.
\nSometimes continuing to retry is the wrong thing to do.
\nSometimes the system should stop calling the dependency for a while.
\nSometimes the right answer is to return something different.
\nRetries, circuit breakers, and fallbacks are therefore not competing solutions. They are different decisions in the failure path.
\nThe useful question is not:
\n“Which pattern should I add?”
\nIt is:
\n“What should this system do when the normal path stops working?”
\nThat question leads to much better backend designs.
","headings":[{"level":2,"text":"The problem","id":"the-problem"},{"level":2,"text":"Retry: dealing with transient failures","id":"retry-dealing-with-transient-failures"},{"level":2,"text":"Why retries alone are not enough","id":"why-retries-alone-are-not-enough"},{"level":2,"text":"Circuit breaker: stopping repeated failure","id":"circuit-breaker-stopping-repeated-failure"},{"level":2,"text":"Retry vs Circuit Breaker","id":"retry-vs-circuit-breaker"},{"level":3,"text":"Retry","id":"retry"},{"level":3,"text":"Circuit breaker","id":"circuit-breaker"},{"level":2,"text":"Fallback: doing something useful when the dependency fails","id":"fallback-doing-something-useful-when-the-dependency-fails"},{"level":2,"text":"Putting the three together","id":"putting-the-three-together"},{"level":2,"text":"The part I find easy to get wrong","id":"the-part-i-find-easy-to-get-wrong"},{"level":2,"text":"Failure handling is also about protecting the system","id":"failure-handling-is-also-about-protecting-the-system"},{"level":2,"text":"Trade-offs","id":"trade-offs"},{"level":3,"text":"Retries","id":"retries"},{"level":3,"text":"Circuit breakers","id":"circuit-breakers"},{"level":3,"text":"Fallbacks","id":"fallbacks"},{"level":2,"text":"What I took away","id":"what-i-took-away"}]}}