Build a reliable payment system handling millions of transactions daily
Master idempotency, distributed transactions, fraud detection, PCI compliance, and exactly-once payment processing
A payment processing system (like Stripe, PayPal, or Razorpay) handles the movement of money between parties - from customers to merchants. Unlike most systems where occasional errors are acceptable, payment systems have zero tolerance for financial discrepancies.
Money is involved. A bug that charges a customer twice or fails to pay a merchant can have serious legal and financial consequences.
Financial Accuracy
Every cent must be accounted for
Security & Compliance
PCI-DSS, fraud prevention required
External Dependencies
Banks, card networks, regulations
Critical Rule: In payment systems, consistency trumps availability. It's better to reject a payment than to accidentally charge twice. Unlike social media (eventual consistency OK), every transaction must be exactly-once.
Understanding scale helps design appropriate consistency and availability trade-offs.
10M / 86,400 sec ≈ ~115 TPS
115 × 10 = ~1,150 TPS
10M × 1KB × 365 days = ~3.6 TB/year
10M × 2 × 365 = ~7.3B entries/year
Design Implication: 1,150 TPS at peak isn't massive, but the consistency requirements are extreme. We need strong ACID guarantees, which limits horizontal scaling options. Focus on correctness over raw throughput.
The payment system connects merchants, customers, and financial institutions through a carefully orchestrated flow.
┌─────────────────────────────────────────────────────────────────────────────┐
│ PAYMENT SYSTEM ARCHITECTURE │
└─────────────────────────────────────────────────────────────────────────────┘
Customer Merchant Admin
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ API GATEWAY │
│ (Rate limiting, Authentication, Request validation) │
└─────────────────────────────────────────────────────────────────────────────┘
│
┌─────────────────────────┼─────────────────────────┐
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Payment API │ │ Merchant API │ │ Admin API │
│ (Checkout) │ │ (Dashboard) │ │ (Operations) │
└────────┬────────┘ └────────┬────────┘ └────────┬────────┘
│ │ │
└───────────────────────┼───────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ PAYMENT ORCHESTRATOR │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Idempotency │ │ State │ │ Fraud │ │ Retry │ │
│ │ Service │ │ Machine │ │ Checker │ │ Handler │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
│
┌───────────────────────┼───────────────────────┐
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Payment Gateway │ │ Ledger │ │ Notification │
│ Connector │ │ Service │ │ Service │
│ (Visa, Master) │ │ (Double-Entry) │ │ (Webhooks) │
└────────┬────────┘ └────────┬────────┘ └─────────────────┘
│ │
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Card Networks │ │ PostgreSQL │
│ Bank APIs │ │ (ACID DB) │
│ (External) │ │ │
└─────────────────┘ └─────────────────┘
Design Choice: We use PostgreSQL (not NoSQL) because payment systems need strong ACID guarantees. Transactions, ledger entries, and idempotency keys must be consistent. Scale vertically first, then shard by merchant_id if needed.
A payment goes through multiple states. Modeling this as a state machine ensures we never lose track of where money is.
Payment State Transitions:
┌─────────────┐
│ CREATED │
└──────┬──────┘
│ validate()
▼
┌─────────────┐
┌────────▶│ PENDING │◀────────┐
│ └──────┬──────┘ │
│ │ process() │ retry()
│ ▼ │
│ ┌─────────────┐ │
│ │ PROCESSING │─────────┘
│ └──────┬──────┘
│ │
│ ┌───────────┼───────────┐
│ ▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ SUCCEEDED │ │ FAILED │ │ CANCELLED │
└──────┬──────┘ └─────────────┘ └─────────────┘
│
│ refund()
▼
┌─────────────┐
│ REFUNDING │
└──────┬──────┘
│
┌───────────┴───────────┐
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ PARTIALLY_REFUND│ │ FULLY_REFUNDED │
└─────────────────┘ └─────────────────┘
Terminal States: SUCCEEDED, FAILED, CANCELLED, FULLY_REFUNDED
Customer clicks "Pay $50"
│
▼
┌────────────────────────────────────────────────────────────────────────────┐
│ 1. CREATE PAYMENT │
│ ───────────────── │
│ • Generate payment_id (UUID) │
│ • Validate idempotency_key (check Redis) │
│ • Store payment record (status: CREATED) │
│ • Return payment_id to client │
└────────────────────────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────────┐
│ 2. RISK ASSESSMENT │
│ ───────────────── │
│ • Run fraud ML model (< 100ms) │
│ • Check velocity limits (3 payments/min/card) │
│ • Verify billing address (AVS) │
│ • If risky: REJECT or FLAG for review │
└────────────────────────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────────┐
│ 3. AUTHORIZE WITH GATEWAY │
│ ─────────────────────────── │
│ • Update status: PROCESSING │
│ • Call card network (Visa/Mastercard) │
│ • Bank validates: card valid, funds available │
│ • Receive authorization_code │
│ • Funds are HELD (not transferred yet) │
└────────────────────────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────────┐
│ 4. CAPTURE FUNDS │
│ ───────────────── │
│ • Request actual money transfer │
│ • Update status: SUCCEEDED │
│ • Create ledger entries (double-entry) │
│ • Queue settlement batch │
└────────────────────────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────────┐
│ 5. NOTIFY & SETTLE │
│ ───────────────── │
│ • Send webhook to merchant │
│ • Email receipt to customer │
│ • Batch settlement to merchant (T+1 or T+2) │
└────────────────────────────────────────────────────────────────────────────┘
CREATE TABLE payments (
payment_id UUID PRIMARY KEY,
idempotency_key VARCHAR(64) UNIQUE,
merchant_id UUID NOT NULL,
customer_id UUID,
amount_cents BIGINT NOT NULL,
currency VARCHAR(3) NOT NULL,
status payment_status NOT NULL,
payment_method JSONB,
gateway_ref VARCHAR(128),
auth_code VARCHAR(64),
failure_reason TEXT,
metadata JSONB,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
version INT DEFAULT 1 -- Optimistic locking
);
CREATE TYPE payment_status AS ENUM (
'CREATED', 'PENDING', 'PROCESSING',
'SUCCEEDED', 'FAILED', 'CANCELLED',
'REFUNDING', 'PARTIALLY_REFUNDED',
'FULLY_REFUNDED'
);// Valid state transitions
const VALID_TRANSITIONS = {
CREATED: ['PENDING', 'CANCELLED'],
PENDING: ['PROCESSING', 'CANCELLED'],
PROCESSING: ['SUCCEEDED', 'FAILED', 'PENDING'],
SUCCEEDED: ['REFUNDING'],
REFUNDING: ['PARTIALLY_REFUNDED', 'FULLY_REFUNDED'],
// Terminal states have no transitions
FAILED: [],
CANCELLED: [],
FULLY_REFUNDED: []
};
function canTransition(from, to) {
return VALID_TRANSITIONS[from]?.includes(to);
}
// Always validate before updating!
if (!canTransition(payment.status, newStatus)) {
throw new InvalidStateTransitionError();
}Key Principle: State transitions are append-only in the audit log. We never delete or modify historical states. The payment record shows current state; the audit log shows the complete history.
Idempotency ensures that retrying a payment request doesn't charge the customer twice. This is THE most critical feature of any payment system.
Scenario: Network timeout after successful charge Client Server Bank │ │ │ │──── POST /pay $50 ─────▶│ │ │ │──── Charge card ───────▶│ │ │◀─── Success ────────────│ │ ✗ TIMEOUT │ │ │ │ (Response lost) │ │ │ │ │──── POST /pay $50 ─────▶│ ← Client retries! │ │ │ │ │ │──── Charge card ───────▶│ ← DOUBLE CHARGE! │ │◀─── Success ────────────│ │◀─── 200 OK ─────────────│ │ Without idempotency: Customer charged $100 instead of $50!
With Idempotency Key: Client Server Redis/DB │ │ │ │ POST /pay $50 │ │ │ Idempotency-Key: abc123 │ │ │────────────────────────▶│ │ │ │── Check key exists? ────────▶│ │ │◀─ NO (first request) ────────│ │ │── Store: abc123 → PROCESSING▶│ │ │ │ │ │ (Process payment...) │ │ │ │ │ │── Update: abc123 → DONE ────▶│ │ ✗ TIMEOUT │ + store response │ │ │ │ │ POST /pay $50 │ │ │ Idempotency-Key: abc123 │ ← Same key, retry │ │────────────────────────▶│ │ │ │── Check key exists? ────────▶│ │ │◀─ YES! Return cached resp ───│ │◀─── 200 OK (cached) ────│ │ │ │ │ Result: Customer charged exactly once!
CREATE TABLE idempotency_keys (
key VARCHAR(64) PRIMARY KEY,
merchant_id UUID NOT NULL,
request_hash VARCHAR(64), -- Hash of request body
status VARCHAR(20), -- PROCESSING, DONE
response_code INT,
response_body JSONB,
payment_id UUID,
created_at TIMESTAMP DEFAULT NOW(),
expires_at TIMESTAMP, -- Auto-cleanup
INDEX idx_merchant_key (merchant_id, key)
);
-- Keys expire after 24 hours
-- Cron job cleans up expired keysasync function handlePayment(req) {
const key = req.headers['Idempotency-Key'];
const requestHash = hash(req.body);
// Check if key exists
const existing = await db.query(
'SELECT * FROM idempotency_keys WHERE key = $1',
[key]
);
if (existing) {
// Verify same request (prevent abuse)
if (existing.request_hash !== requestHash) {
throw new Error('Key reused with different params');
}
if (existing.status === 'DONE') {
return existing.response_body; // Return cached
}
if (existing.status === 'PROCESSING') {
throw new Error('Payment in progress');
}
}
// First request - insert and process
await db.query(
'INSERT INTO idempotency_keys ...',
[key, 'PROCESSING', requestHash]
);
const result = await processPayment(req);
await db.query(
'UPDATE idempotency_keys SET status=$1, response=$2',
['DONE', result]
);
return result;
}Race Condition Warning: Two concurrent requests with the same key can both pass the "key exists?" check. Solution: Use INSERT ... ON CONFLICT or Redis SETNX with atomic operations. Never rely on SELECT then INSERT.
Every money movement is recorded as two entries: a debit from one account and a credit to another. This 500-year-old accounting principle ensures money is never created or destroyed - only moved.
Double-Entry Bookkeeping: For every transaction: SUM(debits) = SUM(credits) Example: Customer pays $50 to Merchant ┌────────────────────────────────────────────────────────────────────────┐ │ LEDGER ENTRIES │ ├────────────────────────────────────────────────────────────────────────┤ │ Entry 1: DEBIT Customer_Wallet -$50.00 │ │ Entry 2: CREDIT Merchant_Pending +$48.50 (after 3% fee) │ │ Entry 3: CREDIT Platform_Revenue +$1.50 (our fee) │ ├────────────────────────────────────────────────────────────────────────┤ │ TOTAL: -$50.00 + $48.50 + $1.50 = $0.00 ✓ Balanced! │ └────────────────────────────────────────────────────────────────────────┘ Why Double-Entry? 1. Self-auditing: If sum ≠ 0, something is wrong 2. Complete trail: Every cent is traceable 3. Reconciliation: Easy to match with bank statements
-- Accounts (wallets, merchant balances, etc.)
CREATE TABLE accounts (
account_id UUID PRIMARY KEY,
account_type VARCHAR(50), -- CUSTOMER, MERCHANT, PLATFORM, ESCROW
owner_id UUID, -- merchant_id or customer_id
currency VARCHAR(3),
balance_cents BIGINT DEFAULT 0,
created_at TIMESTAMP DEFAULT NOW()
);
-- Immutable ledger entries
CREATE TABLE ledger_entries (
entry_id BIGSERIAL PRIMARY KEY,
transaction_id UUID NOT NULL, -- Groups related entries
account_id UUID NOT NULL,
entry_type VARCHAR(10), -- DEBIT or CREDIT
amount_cents BIGINT NOT NULL, -- Always positive
balance_after BIGINT NOT NULL, -- Snapshot for fast queries
description TEXT,
created_at TIMESTAMP DEFAULT NOW(),
CONSTRAINT positive_amount CHECK (amount_cents > 0)
);
-- Entries are IMMUTABLE - no UPDATE or DELETE allowed
-- To reverse: create new opposite entries (not modify existing)
-- Index for fast balance queries
CREATE INDEX idx_ledger_account_time ON ledger_entries(account_id, created_at DESC);async function recordPayment(payment) {
const txnId = uuid();
const fee = payment.amount * 0.03;
const merchantAmount = payment.amount - fee;
await db.transaction(async (trx) => {
// Debit customer
await createEntry(trx, {
transaction_id: txnId,
account_id: payment.customer_account,
entry_type: 'DEBIT',
amount_cents: payment.amount,
});
// Credit merchant (pending)
await createEntry(trx, {
transaction_id: txnId,
account_id: payment.merchant_account,
entry_type: 'CREDIT',
amount_cents: merchantAmount,
});
// Credit platform fee
await createEntry(trx, {
transaction_id: txnId,
account_id: PLATFORM_REVENUE_ACCOUNT,
entry_type: 'CREDIT',
amount_cents: fee,
});
});
}-- Verify ledger is balanced (should be 0)
SELECT
SUM(CASE WHEN entry_type = 'DEBIT'
THEN -amount_cents
ELSE amount_cents END) as net
FROM ledger_entries
WHERE transaction_id = $1;
-- Get account balance (two methods)
-- Method 1: From account table (fast)
SELECT balance_cents FROM accounts
WHERE account_id = $1;
-- Method 2: Calculate from ledger (authoritative)
SELECT SUM(
CASE WHEN entry_type = 'CREDIT'
THEN amount_cents
ELSE -amount_cents END
) as balance
FROM ledger_entries
WHERE account_id = $1;
-- Run reconciliation job to verify both matchAudit Rule: Ledger entries are append-only. To reverse a payment, create new entries with opposite signs - never delete or modify existing entries. This creates a complete audit trail.
A payment involves multiple services and external systems. We can't use a single database transaction. The Saga pattern coordinates these distributed operations with compensating actions for failures.
Payment Saga (Choreography):
Step 1: Reserve Funds
↓ success
Step 2: Run Fraud Check
↓ success
Step 3: Authorize with Bank
↓ success
Step 4: Update Ledger
↓ success
Step 5: Notify Merchant
✓ COMPLETE
If any step fails → Execute compensating transactions in reverse:
Example: Step 3 fails (bank rejects)
Step 3: Bank Authorization ✗ FAILED
↓
Compensate Step 2: Mark fraud check as void
↓
Compensate Step 1: Release reserved funds
↓
Mark payment as FAILED
↓
Notify customer of failure
┌──────────────────────────────────────────────────────────────────────────┐
│ SAGA STEP │ ACTION │ COMPENSATION │
├──────────────────────────────────────────────────────────────────────────┤
│ 1. Reserve Funds │ Hold $50 │ Release $50 │
│ 2. Fraud Check │ Run ML model │ Log as cancelled │
│ 3. Bank Auth │ Call Visa API │ Void authorization │
│ 4. Update Ledger │ Create entries │ Create reverse entries │
│ 5. Notify │ Send webhook │ Send failure webhook │
└──────────────────────────────────────────────────────────────────────────┘
class PaymentSaga {
steps = [
{ name: 'reserve', action: reserveFunds,
compensate: releaseFunds },
{ name: 'fraud', action: checkFraud,
compensate: voidFraudCheck },
{ name: 'authorize', action: authWithBank,
compensate: voidAuth },
{ name: 'ledger', action: updateLedger,
compensate: reverseLedger },
{ name: 'notify', action: notifyMerchant,
compensate: notifyFailure },
];
async execute(payment) {
const completed = [];
for (const step of this.steps) {
try {
await step.action(payment);
completed.push(step);
} catch (error) {
// Compensate in reverse order
for (const done of completed.reverse()) {
await done.compensate(payment);
}
throw error;
}
}
}
}CREATE TABLE saga_state (
saga_id UUID PRIMARY KEY,
payment_id UUID NOT NULL,
current_step VARCHAR(50),
status VARCHAR(20), -- RUNNING, COMPENSATING
completed_steps JSONB, -- [{step, result}]
failed_step VARCHAR(50),
error_message TEXT,
created_at TIMESTAMP,
updated_at TIMESTAMP
);
-- Saga state is persisted after each step
-- On crash: resume from last completed step
-- Ensures saga completes even after failures
-- Recovery process:
-- 1. Find sagas with status = 'RUNNING'
-- 2. Check current_step
-- 3. Resume from that step or compensateSaga Pattern (Recommended)
Two-Phase Commit (Avoid)
Fraud detection must be fast (in the payment critical path) and accurate (minimize false positives that block legitimate customers).
Payment Request
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ FRAUD DETECTION PIPELINE │
│ │
│ ┌────────────────┐ ┌────────────────┐ ┌────────────────┐ │
│ │ Rule Engine │──▶│ ML Model │──▶│ Risk Score │ │
│ │ (Blocklists) │ │ (Real-time) │ │ Aggregator │ │
│ └────────────────┘ └────────────────┘ └────────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ • Blocked cards • Transaction • Combine scores │
│ • Blocked IPs patterns • Apply thresholds │
│ • Velocity checks • Device fingerprint • Return decision │
│ • Country blocklist • Behavioral analysis │
│ │
└─────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ APPROVE │ │ REVIEW │ │ REJECT │
│ Score < 30 │ │ Score 30-70 │ │ Score > 70 │
└─────────────┘ └─────────────┘ └─────────────┘
// Redis-based velocity checking
async function checkVelocity(cardHash, limits) {
const now = Date.now();
const key = `velocity:${cardHash}`;
// Sliding window rate limit
const pipeline = redis.pipeline();
// Remove old entries (outside 1 hour window)
pipeline.zremrangebyscore(key, 0, now - 3600000);
// Count recent transactions
pipeline.zcard(key);
// Add current transaction
pipeline.zadd(key, now, `${now}:${uuid()}`);
// Set expiry
pipeline.expire(key, 3600);
const results = await pipeline.exec();
const count = results[1][1];
// Check limits
if (count >= limits.perHour) {
return { blocked: true, reason: 'hourly_limit' };
}
// Also check per-minute (last 60 seconds)
const minuteCount = await redis.zcount(key, now - 60000, now);
if (minuteCount >= limits.perMinute) {
return { blocked: true, reason: 'minute_limit' };
}
return { blocked: false };
}Latency Budget: Fraud detection runs in the critical payment path. Total time budget: <100ms. Rules: ~10ms, ML inference: ~50ms, network overhead: ~40ms.
Integrating with card networks (Visa, Mastercard) and banks requires handling unreliable external systems gracefully.
┌─────────────────────────────────────────────────────────────────────────┐
│ GATEWAY CONNECTOR LAYER │
└─────────────────────────────────────────────────────────────────────────┘
Payment Orchestrator
│
▼
┌─────────────────┐
│ Gateway Router │ ── Route based on: card type, currency, merchant config
└────────┬────────┘
│
├──────────────┬──────────────┬──────────────┐
▼ ▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Stripe │ │ Adyen │ │ Razorpay │ │ Bank API │
│ Connector │ │ Connector │ │ Connector │ │ Connector │
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘
│ │ │ │
│ Each connector handles: │
│ • Request/response mapping │
│ • Authentication │
│ • Error translation │
│ • Retry logic │
│ • Circuit breaker │
▼ ▼ ▼ ▼
[Stripe API] [Adyen API] [Razorpay API] [Bank API]
interface PaymentGateway {
// Authorize: check funds, place hold
authorize(request: AuthRequest): AuthResponse;
// Capture: actually move the money
capture(authId: string, amount: number): CaptureResponse;
// Void: cancel an authorization
void(authId: string): VoidResponse;
// Refund: return money to customer
refund(captureId: string, amount: number): RefundResponse;
// Query: check transaction status
getTransaction(txnId: string): TransactionStatus;
}
// Each gateway implements this interface
class StripeConnector implements PaymentGateway {
async authorize(request) {
// Map to Stripe's PaymentIntent API
// Handle Stripe-specific errors
// Translate response to common format
}
}// Normalize gateway responses
const ERROR_MAPPING = {
'stripe': {
'card_declined': 'DECLINED',
'insufficient_funds': 'INSUFFICIENT_FUNDS',
'expired_card': 'CARD_EXPIRED',
'processing_error': 'GATEWAY_ERROR',
},
'adyen': {
'Refused': 'DECLINED',
'Not enough balance': 'INSUFFICIENT_FUNDS',
'Expired Card': 'CARD_EXPIRED',
}
};
function normalizeError(gateway, errorCode) {
return ERROR_MAPPING[gateway][errorCode]
|| 'UNKNOWN_ERROR';
}
// This allows consistent error handling
// regardless of which gateway processed itPrimary/Secondary
Route to primary gateway. On failure, automatically retry with secondary.
Smart Routing
Choose gateway based on success rates, fees, and card type.
Health-Based
Monitor gateway health. Route away from degraded gateways.
External gateways fail. Network timeouts happen. We need robust retry logic that doesn't make things worse during outages.
Retry Strategy:
Attempt 1: Immediate
│
✗ Failed (timeout)
│
▼ Wait 1 second
Attempt 2
│
✗ Failed (5xx error)
│
▼ Wait 2 seconds
Attempt 3
│
✗ Failed
│
▼ Wait 4 seconds
Attempt 4
│
✓ Success!
Rules:
• Only retry on transient errors (timeout, 5xx)
• Never retry on permanent errors (4xx, card declined)
• Add jitter to prevent thundering herd
• Max retries: 3-5 attempts
• Max total time: 30 seconds
async function retryWithBackoff(fn, options) {
const { maxRetries = 3, baseDelay = 1000 } = options;
for (let attempt = 0; attempt <= maxRetries; attempt++) {
try {
return await fn();
} catch (error) {
// Don't retry permanent errors
if (!isRetryable(error)) {
throw error;
}
if (attempt === maxRetries) {
throw error;
}
// Exponential backoff with jitter
const delay = baseDelay * Math.pow(2, attempt);
const jitter = Math.random() * delay * 0.1;
await sleep(delay + jitter);
}
}
}
function isRetryable(error) {
// Retry: timeouts, 5xx, network errors
// Don't retry: 4xx, business logic errors
return error.code === 'ETIMEDOUT' ||
error.status >= 500 ||
error.code === 'ECONNRESET';
}class CircuitBreaker {
state = 'CLOSED'; // CLOSED, OPEN, HALF_OPEN
failures = 0;
lastFailure = null;
async call(fn) {
if (this.state === 'OPEN') {
if (Date.now() - this.lastFailure > 30000) {
this.state = 'HALF_OPEN';
} else {
throw new Error('Circuit breaker OPEN');
}
}
try {
const result = await fn();
this.onSuccess();
return result;
} catch (error) {
this.onFailure();
throw error;
}
}
onSuccess() {
this.failures = 0;
this.state = 'CLOSED';
}
onFailure() {
this.failures++;
this.lastFailure = Date.now();
if (this.failures >= 5) {
this.state = 'OPEN'; // Stop calling!
}
}
}Danger: Never blindly retry payment authorizations! After a timeout, the payment might have succeeded. Always check transaction status before retrying, or use idempotency keys to prevent duplicate charges.
Reconciliation ensures our records match the external gateways and banks. Settlement is when merchants actually receive their money.
Daily Reconciliation (runs at 2 AM):
┌────────────────────────────────────────────────────────────────────────┐
│ 1. FETCH EXTERNAL REPORTS │
│ • Download settlement files from Visa, Mastercard │
│ • Fetch transaction reports from payment gateways │
│ • Get bank statement data │
└────────────────────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────┐
│ 2. MATCH TRANSACTIONS │
│ For each external transaction: │
│ • Find matching internal record by gateway_ref │
│ • Compare: amount, currency, status, timestamp │
│ • Mark as: MATCHED, MISSING, MISMATCH │
└────────────────────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────┐
│ 3. FLAG DISCREPANCIES │
│ • MISSING_INTERNAL: Gateway has it, we don't (charge without record)│
│ • MISSING_EXTERNAL: We have it, gateway doesn't (failed capture) │
│ • AMOUNT_MISMATCH: Amounts don't match │
│ • STATUS_MISMATCH: Our status differs from gateway │
└────────────────────────────────────────────────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────────────────────┐
│ 4. RESOLVE & REPORT │
│ • Auto-resolve simple cases (timing differences) │
│ • Queue complex cases for manual review │
│ • Generate reconciliation report │
│ • Alert on high discrepancy rate │
└────────────────────────────────────────────────────────────────────────┘
// Settlement to Merchants (T+2)
Day 0: Payment captured
└─ Funds in "pending" balance
Day 1: Reconciliation confirms payment
└─ Funds move to "available" balance
Day 2: Settlement batch runs
└─ Bank transfer to merchant
// Settlement Batch Job
async function runSettlement() {
const merchants = await getMerchantsWithBalance();
for (const merchant of merchants) {
const balance = await getAvailableBalance(merchant);
if (balance < MINIMUM_PAYOUT) continue;
// Create payout record
const payout = await createPayout({
merchant_id: merchant.id,
amount: balance,
bank_account: merchant.bank_account
});
// Initiate bank transfer
await bankTransfer(payout);
// Update ledger
await recordPayoutLedger(payout);
}
}CREATE TABLE reconciliation_records (
id BIGSERIAL PRIMARY KEY,
recon_date DATE NOT NULL,
gateway VARCHAR(50),
internal_txn_id UUID,
external_txn_id VARCHAR(128),
internal_amount BIGINT,
external_amount BIGINT,
internal_status VARCHAR(50),
external_status VARCHAR(50),
match_status VARCHAR(20), -- MATCHED, MISMATCH...
discrepancy_type VARCHAR(50),
resolution TEXT,
resolved_at TIMESTAMP,
created_at TIMESTAMP DEFAULT NOW()
);
-- Metrics to track
-- • Match rate: Should be > 99.9%
-- • Resolution time: Avg time to resolve
-- • Discrepancy rate by gatewayPCI-DSS (Payment Card Industry Data Security Standard) is mandatory for any system handling card data. Non-compliance can result in heavy fines and loss of ability to process payments.
1. Network Security
2. Data Protection
3. Access Control
4. Monitoring & Testing
Tokenization Flow (reduces PCI scope drastically):
Customer Browser Your Server Token Vault
│ │ │
│ Enter: 4242-4242-4242-4242 │ │
│ │ │
│────── Card data ──────────────────────────────────────────────▶│
│ │ │
│◀───── Token: tok_abc123 ──────────────────────────────────────│
│ │ │
│────── Token: tok_abc123 ─────▶│ │
│ │ │
│ │── tok_abc123 + charge ──────▶│
│ │ │
│ │◀── Success (card charged) ───│
Benefits:
• Your server NEVER sees raw card number
• PCI scope limited to frontend + vault
• Tokens are useless if stolen
• Easier compliance (SAQ-A vs SAQ-D)
Token Format:
• Looks like: tok_1234567890abcdef
• Not reversible without vault access
• Merchant-specific (can't use elsewhere)
Never Log Card Data: Ensure logging frameworks mask card numbers. A single log line with a PAN puts you out of compliance. Use regex filters: /\b\d{13,19}\b/ to detect and mask.
Practice with an AI interviewer and get instant feedback on your system design skills.