Scaling Transaction Systems — Problems & Solutions

Most engineers know how to build a payment system. Very few know what breaks it at scale — and exactly when.

This document covers the engineering problems that appear as transaction volume grows from 1M to 10M per day, organised by subsystem. Every problem is stated precisely — what breaks and why — then solved with multiple options and explicit trade-offs. Because in a real design review there is rarely one right answer, only a defensible choice for a given constraint set.

What is covered across 10 sections:

• Ingestion & API layer — async ingestion, rate limiting, load shedding
• Idempotency & duplicate suppression — the TOCTOU race, state machines, distributed locks
• Database write path — single primary ceiling, hot rows, connection pool exhaustion
• Database read path — read replicas, CQRS, table partitioning, hot/cold tiering
• Async processing — dual-write problem, transactional outbox, CDC, poison messages
• Consistency — sagas, 2PC trade-offs, reserve/capture pattern, write skew
• Reconciliation & EOD batch — incremental reconciliation, file-based integration, auto-triage
• Caching — cache stampede, thundering herd, cache/DB inconsistency
• Observability & resilience — distributed tracing, circuit breakers, correctness monitoring
• Tiered decision summary — what to actually do at 1M, 5M, and 10M/day

Who this is for:

Backend and distributed systems engineers preparing for senior or staff-level interviews at fintech, payments, or e-commerce companies. Also useful as a reference for engineers currently scaling payment systems in production.

Prepared by a fintech engineer who has built payment systems at PayPal and Goldman Sachs — the trade-offs in this document come from production decisions, not textbook theory.