Once a system is split into services, every interaction between them crosses a network that can be slow, drop requests or deliver them twice. How services talk to each other therefore decides how the system behaves when something goes wrong. This article describes the main communication styles, when each one fits, and the failure-handling patterns that go with them.
It assumes you already have, or are planning, several services. Splitting a system into services, scaling them and managing their data are covered separately in Building scalable microservices APIs.
Synchronous and asynchronous communication
In synchronous communication the caller sends a request and waits for the response. In asynchronous communication the sender hands a message to a broker and carries on; the receiver processes it later. The choice affects coupling more than speed.
| Synchronous call | Asynchronous message | |
|---|---|---|
| Caller needs the result to continue | Yes | No |
| Receiver must be up at that moment | Yes | No, the broker holds the message |
| Failure is visible to the caller | Immediately | Only through monitoring or a later event |
| Consistency | Immediate | Eventual |
| Extra infrastructure | None | A message broker |
Asynchronous messaging is not better by default. It removes the dependency on the receiver being available, at the price of a broker to operate, eventual consistency and harder debugging. Use a synchronous call when the caller cannot proceed without the answer, such as checking a permission or reading data to render a page. Use a message when the work can happen after the response, such as sending an email, updating a search index or notifying other services that something changed.
Synchronous calls
Most service-to-service calls are HTTP with JSON, which every language and tool supports. gRPC is a common alternative for internal traffic: it uses HTTP/2 and Protocol Buffers, generates typed clients from a schema, and supports streaming. It is a reasonable choice when services exchange many small messages or need streaming, and less convenient when the API must also be called from browsers or by third parties.
The main risk of synchronous calls is the chain. If a request passes through five services in sequence, its latency is the sum of all five and it fails if any one of them fails. Keep chains short, make independent calls in parallel, and question any design in which a service cannot answer without calling several others.
Every synchronous call needs a timeout. Without one, a slow dependency holds connections and memory in the caller until the caller is also unresponsive. The function below sets a two-second timeout and retries failures that may be temporary, waiting longer after each attempt.
async function getUser(id, attempts = 3) {
for (let attempt = 1; ; attempt++) {
try {
const res = await fetch(`http://user-service/users/${id}`, {
signal: AbortSignal.timeout(2000),
})
if (res.ok) return await res.json()
const error = new Error(`user-service responded ${res.status}`)
error.retryable = res.status >= 500 || res.status === 429
throw error
} catch (err) {
if (err.retryable === false || attempt === attempts) throw err
const delay = Math.random() * 100 * 2 ** attempt
await new Promise((resolve) => setTimeout(resolve, delay))
}
}
}Three rules are built into it. Only errors that might succeed on a second try are retried: timeouts, network errors, 5xx and 429, but not other 4xx responses. The delay grows exponentially and is randomized, so that many callers recovering from the same outage do not retry in step. And the number of attempts is small and fixed.
Retries also multiply. If three layers each retry three times, one user request can become 27 calls to the service at the bottom, at the moment it is least able to cope. Retry at one layer, usually the one closest to the failure, and let the others fail fast.
Queues and publish-subscribe
Brokers offer two basic shapes. With a work queue, each message goes to exactly one of a pool of competing consumers; it suits jobs such as resizing an image. With publish-subscribe, each subscribing service gets its own copy of every message; it suits events, where the publisher states that something happened and does not know or care who reacts.
The following examples use RabbitMQ through the amqplib package. The publisher sends a user.registered event to a topic exchange and waits for the broker to confirm it has stored the message.
import amqp from 'amqplib'
import { randomUUID } from 'node:crypto'
const connection = await amqp.connect(process.env.AMQP_URL)
const channel = await connection.createConfirmChannel()
await channel.assertExchange('user.events', 'topic', { durable: true })
export async function publishUserRegistered(user) {
const event = {
id: randomUUID(),
type: 'user.registered',
occurredAt: new Date().toISOString(),
data: { userId: user.id, email: user.email },
}
channel.publish('user.events', event.type, Buffer.from(JSON.stringify(event)), {
persistent: true,
messageId: event.id,
contentType: 'application/json',
})
await channel.waitForConfirms()
}Each consuming service has its own queue bound to the exchange, so the email service and the profile service each receive the event and fail independently. This is the email service's consumer, using a channel opened in the same way.
const queue = 'email.user-registered'
await channel.assertExchange('user.events.dead', 'fanout', { durable: true })
await channel.assertQueue(queue, { durable: true, deadLetterExchange: 'user.events.dead' })
await channel.bindQueue(queue, 'user.events', 'user.registered')
await channel.prefetch(10)
await channel.consume(queue, async (message) => {
if (!message) return
const event = JSON.parse(message.content.toString())
try {
if (!(await processedEvents.has(event.id))) {
await sendWelcomeEmail(event.data.email)
await processedEvents.add(event.id)
}
channel.ack(message)
} catch (err) {
console.error('user.registered failed', event.id, err)
channel.nack(message, false, false)
}
})The message is acknowledged only after the work is done, so a consumer that crashes halfway gets the message again. That is at-least-once delivery, and it means consumers will sometimes see the same message twice. The consumer therefore records the IDs of events it has handled, in its own database, and skips repeats. A crash between sending the email and recording the ID can still produce a second email; where that is unacceptable, the downstream operation itself must be idempotent.
A message that fails is rejected without requeueing, which sends it to the dead-letter exchange configured on the queue. Bind a queue to that exchange and alert on its depth. Requeueing a message that always fails would otherwise block the consumer in a loop.
Ordering is the other property to check. A queue with several competing consumers does not process messages in order. If order matters, for example for successive updates to one account, use a broker that preserves order per key, such as Kafka partitions, or include a version number in each event so that consumers can discard stale ones.
Publish events reliably
A service that updates its database and then publishes an event performs two writes, and the process can crash between them. The database then says the user exists while no event was ever sent, and nothing will repair it.
The transactional outbox pattern closes this gap. The service writes the event into an outbox table in the same database transaction as the business change, so that both are committed or neither is. A separate relay process reads unsent rows, publishes them to the broker and marks them sent. The relay may publish an event twice if it crashes after publishing, which the idempotent consumers described above already handle. Change data capture tools such as Debezium can act as the relay by reading the database's log.
Event sourcing and CQRS
Two further patterns are often mentioned alongside event-driven systems. Both are optional, and neither is needed in order to publish events between services.
Event sourcing stores every change to an entity as an immutable event and derives the current state by replaying them. It gives a complete history and the ability to rebuild state as of any point, which suits ledgers and audit-heavy domains. It also makes simple queries indirect, and old events must remain readable as their schemas change.
Command query responsibility segregation (CQRS) uses separate models for writing and reading: commands go to a model built to enforce rules, while queries are served from views shaped for them and updated from events. It helps when read and write loads differ greatly or queries need data in a very different shape. The read side lags behind the write side, so a user may not immediately see their own change unless the interface allows for it.
Adopt either for a specific service that has the problem it solves, not as a system-wide default.
Contain failures
Timeouts and bounded retries deal with brief faults. Two more patterns stop a longer outage from spreading.
A circuit breaker tracks failures of calls to one dependency. After a threshold of consecutive failures it opens, and further calls fail immediately without touching the network. After a cool-down it lets calls through again; success closes it and failure reopens it. Callers get a fast error they can handle, and the struggling service gets room to recover. A minimal version is shown here.
class CircuitBreaker {
constructor(fn, { threshold = 5, cooldownMs = 10_000 } = {}) {
Object.assign(this, { fn, threshold, cooldownMs, failures: 0, openedAt: 0 })
}
async call(...args) {
const open = this.failures >= this.threshold
if (open && Date.now() - this.openedAt < this.cooldownMs) {
throw new Error('Circuit open')
}
try {
const result = await this.fn(...args)
this.failures = 0
return result
} catch (err) {
this.failures += 1
if (this.failures >= this.threshold) this.openedAt = Date.now()
throw err
}
}
}
const users = new CircuitBreaker(getUser)
const user = await users.call('123')This version lets every waiting call through once the cool-down ends. A library such as opossum limits that to a single trial call and adds error-rate thresholds and metrics.
A breaker is only useful with a plan for when it is open. Decide per call: serve a cached or default value, leave an optional part of the response out, or return an error for the part that is truly required. A product page can render without recommendations; checkout cannot proceed without payment.
A bulkhead limits how much of the caller's capacity any one dependency can consume, by giving each dependency its own connection pool or a cap on concurrent calls. When one dependency hangs, calls to it queue against its own limit and the rest of the service keeps working.
A service mesh such as Istio or Linkerd can apply timeouts, retries, circuit breaking and mutual TLS in a proxy beside each service, configured centrally. That gives uniform behaviour across languages, at the cost of another system to run. The mesh cannot know which operations are safe to retry or what a sensible fallback is, so those decisions remain in application code.
Summary
- Call synchronously when the caller needs the answer to proceed; send a message when the work can happen later and the receiver need not be available.
- Give every synchronous call a timeout and retry only repeatable operations, a few times, with randomized exponential backoff, at one layer.
- Assume at-least-once delivery: acknowledge messages after processing, make consumers idempotent and route failures to a dead-letter queue.
- Use a transactional outbox so that a database change and its event cannot diverge.
- Add circuit breakers and bulkheads around dependencies, and decide in advance what each caller does when a dependency is unavailable.