Saga Pattern and Distributed Transactions — How to Avoid Disaster in Microservices

Your order went through, but the payment failed. Or the payment succeeded, and the warehouse didn't update stock. In microservices, a transaction that spans multiple services is like a relay race: if one runner falls, the team must know how to recover. Without a plan, your system ends up with inconsistent data and angry customers.

We, at Meteora Web, have been working with distributed architectures for years. We know that ACID transaction theory doesn't apply to microservices. Here you need a different approach: the saga pattern. In this guide, we'll see how it works, when to use it, and how to implement it without costly mistakes.

Why ACID transactions don't work in microservices?

In a monolithic database, a transaction is atomic: either everything succeeds or everything is rolled back. In microservices, each service has its own database. You can't do a ROLLBACK on a service that doesn't know about the others. The concept of distributed transactions with 2PC (two-phase commit) exists, but it's fragile: if a service fails during commit, everything blocks. In a distributed system, latency and failures are the norm, not the exception.

The saga pattern breaks a long transaction into a sequence of local transactions. Each local transaction updates the service's database and publishes an event. The next service listens for the event and continues. If a step fails, compensating transactions are executed to undo previous steps.

A concrete example: the e-commerce order

Imagine an order involving three services: Orders, Payments, Inventory.

1. Orders creates the order with status "pending".
2. Payments charges the card.
3. Inventory decrements stock.

If inventory fails because the product is unavailable, you need to cancel the payment. The compensating transaction for step 2 is a refund. For step 1, cancel the order. This is the saga pattern in action.

How does the saga pattern work in practice?

There are two ways to orchestrate a saga: choreography and orchestration. The choice depends on the complexity of the flow and the need for control.

Choreography: events chaining

Each service listens to events from others and decides whether to act. No central coordinator. It's simple to implement, but the flow is implicit: if an event doesn't arrive, it's hard to figure out where it got stuck.

// Orders service: publishes event after creation
const event = { type: 'ORDER_CREATED', orderId: 123, amount: 100 };
await eventBus.publish('orders', event);

// Payments service: listens and charges
await eventBus.subscribe('orders', async (event) => {
  if (event.type === 'ORDER_CREATED') {
    await chargeCreditCard(event.orderId, event.amount);
    await eventBus.publish('payments', { type: 'PAYMENT_SUCCESS', orderId: event.orderId });
  }
});

// Inventory service: listens and updates
await eventBus.subscribe('payments', async (event) => {
  if (event.type === 'PAYMENT_SUCCESS') {
    await updateStock(event.orderId);
  }
});

With choreography, each service is autonomous. But as the flow grows, it becomes a tangle that's hard to debug.

Orchestration: a coordinator decides

An orchestrator (a dedicated service) guides the saga. It knows all steps and their compensations. It's more centralized but easier to manage and monitor.

// Orchestrator: defines steps and compensations
const saga = {
  steps: [
    { name: 'createOrder', compensate: 'cancelOrder' },
    { name: 'chargePayment', compensate: 'refundPayment' },
    { name: 'updateStock', compensate: 'restoreStock' }
  ]
};

async function runSaga(order) {
  const executed = [];
  for (const step of saga.steps) {
    try {
      await executeStep(step.name, order);
      executed.push(step);
    } catch (err) {
      console.error(`Step ${step.name} failed:`, err);
      for (const executedStep of executed.reverse()) {
        await compensateStep(executedStep.compensate, order);
      }
      throw err;
    }
  }
}

We prefer orchestration for complex flows. It gives you visibility and control. Choreography works for simple, independent flows.

What are compensating transactions and how to design them?

A compensating transaction is an action that undoes the effects of a previous transaction. It's not a rollback: it's a new transaction that restores the state. It must be idempotent, meaning executing it multiple times produces the same result.

Common examples:

• Payment charged → refund.
• Stock decremented → increment.
• Email sent → you can't unsend it, but you can send a correction email.

Not all operations are compensable. If you've sent a push notification, you can't "undo" it. In that case, you must accept temporary inconsistency and handle it at the application level.

How to make compensations idempotent

Use a transaction ID that is unique. Each compensation must check if it has already been executed for that ID. In PostgreSQL, you can use a dedicated table to track operations.

CREATE TABLE transaction_log (
  transaction_id UUID PRIMARY KEY,
  step_name TEXT NOT NULL,
  status TEXT NOT NULL,
  created_at TIMESTAMP DEFAULT NOW()
);

-- Before executing a compensation, check if it exists
INSERT INTO transaction_log (transaction_id, step_name, status)
VALUES ('uuid-123', 'refund_payment', 'started')
ON CONFLICT (transaction_id) DO NOTHING;

-- If the row exists, the compensation has already been executed

This table protects you from double executions, especially if the service crashes and restarts.

How to handle failures and retries in a saga?

Failures are inevitable. Your system must decide whether to retry or compensate. The general rule: if the error is temporary (timeout, network), retry. If it's permanent (invalid data), compensate.

Implement a retry with exponential backoff for temporary errors. If after N attempts it still fails, start compensation.

async function executeWithRetry(fn, maxRetries = 3) {
  let attempt = 0;
  while (attempt < maxRetries) {
    try {
      return await fn();
    } catch (err) {
      if (err.retryable && attempt < maxRetries - 1) {
        const delay = Math.pow(2, attempt) * 1000;
        await sleep(delay);
        attempt++;
      } else {
        throw err;
      }
    }
  }
}

In orchestration, the orchestrator must persist the saga state in a database. This way, if the orchestrator crashes, it can resume from where it stopped. Use a pattern like outbox to publish events reliably.

The Outbox pattern for consistency

When a service updates the database and publishes an event, you must ensure both operations are atomic. The outbox pattern solves this: you save the event in an outbox table in the same transaction as the database. A separate process reads the outbox and publishes events to the message broker.

BEGIN;
-- Update order status
UPDATE orders SET status = 'confirmed' WHERE id = 123;
-- Insert event into outbox
INSERT INTO outbox (event_id, payload) VALUES ('uuid-456', '{"type":"ORDER_CONFIRMED","orderId":123}');
COMMIT;

With the outbox, you don't lose events. If the service crashes after commit, the publishing process will still send them.

When to use the saga pattern and when to avoid it?

The saga pattern is not a one-size-fits-all solution. Use it when you have a transaction spanning multiple services and you can't use ACID. Avoid it if you can redesign the system to have fewer dependencies between services.

Some scenarios where saga is necessary:

• E-commerce orders with payment and inventory.
• Travel bookings: flight, hotel, car.
• Money transfers between accounts in different services.

Scenarios where you can avoid it:

• If services share the same database, use an ACID transaction.
• If the operation is asynchronous and doesn't require immediate consistency, you can accept eventual consistency without compensation.

We, at Meteora Web, have seen projects where saga was overkill. A simple async message was enough. The key is understanding your business consistency requirements.

What tools to use to implement a saga?

You don't have to build everything from scratch. There are frameworks and libraries that make implementation easier.

Java frameworks: Axon and Eventuate

Axon Framework supports the saga pattern with event sourcing. Eventuate Tram Sagas by Chris Richardson (the pattern's author) is another solid option.

Libraries for Node.js and Python

In Node.js, you can use saga-orchestrator (npm). In Python, saga-pattern (PyPI). These are simple libraries that implement orchestration.

Message brokers as infrastructure

Kafka or RabbitMQ are the backbone. They handle events and guarantee delivery. We often use Kafka for its high reliability and scalability.

For official documentation, check the saga pattern on microservices.io and the Confluent guide on Kafka.

Common mistakes to avoid in distributed transactions

Even the best developers make mistakes. Here are the errors we see most often in projects that come to us.

Not making operations idempotent

If a compensation is executed twice, it can corrupt data. Use a transaction ID and always check the status.

Ignoring orchestrator failure

If the orchestrator crashes, the saga gets stuck. Persist the state and implement a recovery mechanism.

Mixing sync and async without criteria

Synchronous calls between services increase coupling. Prefer asynchronous events. But if you need to return a response to the client, use a pattern like saga with async response.

What to do now

Here are immediate actions to implement the saga pattern in your system:

1. Analyze your flows: identify transactions that span multiple services and where consistency is needed.
2. Choose the model: choreography for simple flows, orchestration for complex ones.
3. Design compensations: for each step, define the undo action and make it idempotent.
4. Implement retry: with exponential backoff for temporary errors.
5. Persist saga state: use a table or database to resume after a crash.

If you want to dive deeper into microservices architecture, read our pillar guide on microservices. And if your system already has consistency issues, talk to us. We know how to fix them.

Distributed transactions are not a problem to underestimate. With the saga pattern, though, you can handle them robustly. Your system will stay consistent even when things go wrong. And that, in our work, makes the difference between a client who returns and one who leaves.