Ela Wieliczko

Case Study: Production SQL Tuning

Fintech Transaction Engine - High-Load Database Optimization

!

The Challenge

Business context

Every time a client opened their banking dashboard, the app loaded the 20 most recent transactions for their account. The ledger table held 45,000,000 rows (~15,000 per account). Each refresh took 3–6 seconds and pulled gigabytes from disk – under concurrent traffic these requests saturated database I/O and slowed down everything else on the server.

Reproduced for this case study on a 45M-row dataset modeled on the production table – all plans below are real EXPLAIN output.

1

The Query

Hot path: dashboard “recent transactions”

SELECT t.id, t.amount, t.created_at, a.account_number
FROM transactions t
JOIN accounts a ON t.account_id = a.id
WHERE t.account_id = 1402
ORDER BY t.created_at DESC
LIMIT 20;

Simple on paper: one account, newest first, 20 rows. The question is how much work the database does to return those 20 rows.

2

Plan Before — 5,458 ms

EXPLAIN (ANALYZE, BUFFERS)

Limit  (actual time=5457.421..5458.201 rows=20 loops=1)
  Buffers: shared hit=451 read=555108                          ← [D]
  ->  Sort  (actual time=5457.420..5458.199 rows=20 loops=1)          ← [C]
        Sort Key: t.created_at DESC
        Sort Method: top-N heapsort  Memory: 30kB
        ->  Nested Loop  (actual time=1.594..5451.965 rows=15058 loops=1)
              ->  Index Scan using accounts_pkey on accounts a
                    Index Cond: (id = 1402)
              ->  Gather  (actual time=1.573..5448.742 rows=15058 loops=1)
                    Workers Launched: 2
                    ->  Parallel Seq Scan on transactions t              ← [A]
                          (actual time=3.301..5445.450 rows=5019 loops=3)
                          Filter: (account_id = 1402)
                          Rows Removed by Filter: 14994981           ← [B]
Execution Time: 5458.229 ms

Where the problem is (read the plan bottom-up):

[A] Parallel Seq Scan – the bottleneck. It finishes at 5,445 ms out of 5,458 ms total: 99.8% of the time is spent in this single node. Everything above it (join, sort, limit) costs ~13 ms. There is no index on account_id, so the planner has no way to locate one account’s rows – its only option is to read the entire table.
[B] Rows Removed by Filter: 14,994,981 × 3 workers. The database examined all ~45 million rows to keep 15,058 (0.03%) – and the user sees only 20 of them.
[C] Sort. Rows come out of the table in physical order, not by date. All 15,058 matches must be collected and sorted before LIMIT 20 can apply – so the limit saves no work at all.
[D] Buffers read=555,108. 555k pages × 8 kB ≈ 4.3 GB read from disk for a single dashboard refresh. The table does not fit in memory, so every run goes to disk – across repeated runs this scan took 3.1–6.1 s. This is what kills the server under concurrency, not just the latency of one request.

Root cause: no access path matching both the filter (account_id) and the ordering (created_at DESC). Cost grows linearly with table size – the ledger only gets bigger.

3

The Fix

Composite covering index, built without downtime

CREATE INDEX CONCURRENTLY idx_transactions_account_history
ON transactions (account_id, created_at DESC)
INCLUDE (id, amount);

Each part solves one problem from the plan:

account_id first → fixes [A] and [B]: the B-tree jumps straight to account 1402 instead of scanning 45M rows.
created_at DESC second → fixes [C]: within one account, entries are already stored newest-first. No Sort is needed, and LIMIT 20 can stop after reading 20 entries.
INCLUDE (id, amount) → fixes [D]: all selected columns are stored in the index leaves (a covering index), so the table itself is never read.
CONCURRENTLY → the index is built on the live production table without blocking INSERTs of new transactions.

4

Plan After — 0.085 ms

EXPLAIN (ANALYZE, BUFFERS)

Limit  (actual time=0.037..0.060 rows=20 loops=1)
  Buffers: shared hit=25                                        ← [D]
  ->  Nested Loop  (actual time=0.036..0.056 rows=20 loops=1)
        ->  Index Only Scan using idx_transactions_account_history   ← [A]
              on transactions t  (actual time=0.022..0.033 rows=20 loops=1)   ← [B]
              Index Cond: (account_id = 1402)
              Heap Fetches: 0
              Buffers: shared hit=22
        ->  Materialize  (actual time=0.001..0.001 rows=1 loops=20)
              ->  Index Scan using accounts_pkey on accounts a
                    Index Cond: (id = 1402)
Execution Time: 0.085 ms

What changed in the plan:

[A] Index Only Scan replaces the Parallel Seq Scan – no Gather, no parallel workers needed.
[B] rows=20. The index delivered exactly 20 entries and stopped. Nothing was read and thrown away.
[C] The Sort node is gone – rows come out of the index already in created_at DESC order.
[D] 25 pages (~200 kB), all from memory. Heap Fetches: 0 confirms the table itself was never touched.

Query cost no longer depends on table size – only on LIMIT. At 450M rows it would still read ~20 entries.

✓

Summary & Takeaways

Outcome and lessons

Metric	Before	After
Execution time	5,458 ms (3.1–6.1 s range)	0.085 ms
Access method	Parallel Seq Scan	Index Only Scan
Rows examined	~45,000,000	20
Sort step	15,058 rows	none
Pages read	555,559 (~4.3 GB)	25 (~200 kB)
Read from disk	555,108 pages	0

Conclusionthe problem was not the hardware, the table size or the query itself – it was a missing access path. One index matching both the filter and the sort order turned a query that scales with the whole table into one that only does the work the user actually sees. No application code changed, no downtime.

Takeaways – red flags in an execution plan:

Find the node where time accumulates. Read the plan bottom-up – here one node held 99.8% of the runtime.
Rows Removed by Filter far above rows returned → the database reads data only to throw it away. Missing index on the filter column.
Sort below a LIMIT → the index does not match ORDER BY, so every match is sorted to return a few rows.
Buffers: read= in the hundreds of thousands → the real cost for the whole server, not just for one request.
Composite index order matters: equality filter first, sort column second, remaining selected columns in INCLUDE.
Next step: for “Load more”, fetch following pages with keyset pagination (WHERE created_at < :last_seen) instead of OFFSET – the same index then serves every page in the same ~0.1 ms.

Trade-offs taken into account:

Storage: index 2.1 GB vs table 4.3 GB. A covering index on 45M rows is not small – the INCLUDE columns make it noticeably larger. Dropping them would shrink the index but bring back heap lookups; for this hot path the trade is worth it. The query touches only ~25 pages of it, so the size does not affect read speed.
Every new index adds a small cost to each INSERT – justified here by a read-heavy, high-frequency query.
Heap Fetches: 0 relies on an up-to-date visibility map, so autovacuum must keep pace with write traffic on the ledger.
Build time: 72 s online for 45M rows with default settings. CONCURRENTLY is slower than a plain build (two table passes, waits for running transactions) and, if interrupted, leaves an invalid index that must be dropped and rebuilt. On production, raising maintenance_work_mem for the build session shortens it noticeably.

Impact: >35,000x Faster / ~22,000x Fewer Pages Read Measured: 45M rows / PostgreSQL

Case Study: Production Data Deduplication

Financial Transaction Ledger - Integrity Recovery on Oracle

!

The Challenge

Business context

An upstream payment provider delivered some webhooks twice. The ledger table had no PRIMARY KEY or UNIQUE constraint on the transaction id, so every second delivery was stored as an exact clone – same id, user, amount and timestamp. Account balances and reports started double-counting money.

The catch: the clones are identical in every column. A plain DELETE ... WHERE id = 36625 removes both copies – there is nothing in the data to tell the original from the duplicate.

Reproduced for this case study on Oracle Database 23ai with a 250k-row ledger and 50k injected clones – all results below are real query output.

1

The Problem in the Data

250,000 rows / 200,000 unique ids

SELECT COUNT(*) AS total_rows, COUNT(DISTINCT id) AS unique_ids FROM transactions;

TOTAL_ROWS|UNIQUE_IDS|
----------+----------+
    250000|    200000|

SELECT ROWID, t.* FROM transactions t WHERE id = 36625;

ROWID             |ID   |USER_ID|AMOUNT|TRANSACTION_DATE       |
------------------+-----+-------+------+-----------------------+
AAAR4TAAYAAAAALAAE|36625|    428|578.72|2026-08-23 06:35:30.458|
AAAR4TAAYAAAAEuACc|36625|    428|578.72|2026-08-23 06:35:30.458|

Every column is identical – except ROWID. It is Oracle’s pseudocolumn holding the row’s physical address: data object, file, block and slot, encoded in base64. Decoded, the original lives in block 11, slot 4; the clone was appended much later to block 302, slot 156. Two rows can never share an address, so it is the one handle that tells the twins apart.

2

The Solution

One statement: number the copies, delete by physical address

DELETE FROM transactions
WHERE ROWID IN (
    SELECT rid FROM (
        SELECT ROWID AS rid,
               ROW_NUMBER() OVER (PARTITION BY id
                                  ORDER BY transaction_date, ROWID) AS rn
        FROM transactions
    )
    WHERE rn > 1
);

How it works:

PARTITION BY id groups all copies of the same transaction; ROW_NUMBER() labels them 1, 2, 3…
ROWID as the tie-breaker in ORDER BY makes the choice deterministic: the copy with the lowest physical address is kept as the original.
Every rn > 1 is a clone; DELETE removes exactly those physical rows – never the original.
Dry run first: the same subquery wrapped in COUNT(*) returned 50,000 before a single row was touched.

3

Execution Plan — 0.31 s

DBMS_XPLAN.DISPLAY_CURSOR (ALLSTATS LAST)

-----------------------------------------------------------------------------------------------
| Id | Operation              | Name         | E-Rows | A-Rows |   A-Time   | Buffers | Used-Mem |
-----------------------------------------------------------------------------------------------
|  0 | DELETE STATEMENT       |              |        |      0 |00:00:00.31 |   55143 |          |
|  1 |  DELETE                | TRANSACTIONS |        |      0 |00:00:00.31 |   55143 |          |  ← [E]
|* 2 |   HASH JOIN SEMI       |              |   250K |  50000 |00:00:00.16 |    2173 |  14M (0) |  ← [D]
|  3 |    TABLE ACCESS FULL   | TRANSACTIONS |   250K |   250K |00:00:00.01 |    1087 |          |  ← [A]
|  4 |    VIEW                | VW_NSO_1     |   250K |  50000 |00:00:00.10 |    1086 |          |
|* 5 |     VIEW               |              |   250K |  50000 |00:00:00.10 |    1086 |          |  ← [C]
|  6 |      WINDOW SORT       |              |   250K |   250K |00:00:00.10 |    1086 |  12M (0) |  ← [B]
|  7 |       TABLE ACCESS FULL| TRANSACTIONS |   250K |   250K |00:00:00.01 |    1086 |          |  ← [A]
-----------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
   2 - access(ROWID="RID")
   5 - filter("RN">1)

Starts, Reads, OMem and 1Mem columns omitted for width. Reads: 1 physical block for the whole statement.

Reading the plan (bottom-up):

[A] Two full scans of the table (Id 7 and 3), ~1,086 blocks and 0.01 s each – one to number the copies, one to find them for deletion. Both served from the buffer cache.
[B] WINDOW SORT numbers the copies within each id: 250K rows in 12 MB, fully in memory – (0) means zero passes to temp. Oracle’s automatic PGA management sized the work area on its own.
[C] Filter rn > 1 keeps exactly 50,000 rows – the injected clones. The estimate (E-Rows 250K) is off because the optimizer cannot predict a window-function filter; harmless here, worth knowing inside larger queries.
[D] HASH JOIN SEMI on ROWID = RID. The optimizer unnested the IN subquery (VW_NSO_1) into a semi-join: 50,000 rowids in a 14 MB hash table, probed by the second scan. Finding all clones took 0.16 s.
[E] DELETE takes the remaining 0.15 s, and buffer gets jump from 2,173 to 55,143. The expensive part is not the search but removing rows and writing undo and redo – that is what grows at production scale.

4

Verification

Same checks after the delete

SELECT COUNT(*)                      AS total_rows,
       COUNT(DISTINCT id)            AS unique_ids,
       COUNT(*) - COUNT(DISTINCT id) AS remaining_duplicates
FROM transactions;

TOTAL_ROWS|UNIQUE_IDS|REMAINING_DUPLICATES|
----------+----------+--------------------+
    200000|    200000|                   0|

One row per transaction id, every original preserved, zero duplicates left.

✓

Summary & Takeaways

Outcome and lessons

Metric	Before	After
Total rows	250,000	200,000
Unique transaction ids	200,000	200,000
Duplicate rows	50,000	0

Cleanup run	Value
Execution time	0.31 s
Time spent finding clones	0.16 s
Sort spilled to disk	no (12 MB in memory)

Conclusionwhen duplicates are identical in every column, the data itself offers no way to pick one copy. Oracle’s physical row address does. One window function plus a ROWID-based delete cleaned the ledger in a single statement, keeping every original row intact.

Takeaways:

Exact clones need a physical handle. ROWID is the Oracle answer (PostgreSQL’s equivalent is ctid).
ROWID is stable, but not permanent. Unlike ctid it survives an UPDATE, but it changes on ALTER TABLE ... MOVE, SHRINK SPACE, partition row movement and export/import – use it within a statement, never store it.
Dry run, then verify. Count the rows to be deleted first; afterwards, rows removed must equal duplicates found and the unique-id count must not change.
Fix the cause, not just the data: add ALTER TABLE transactions ADD CONSTRAINT pk_transactions PRIMARY KEY (id) and make the webhook handler idempotent with MERGE ... WHEN NOT MATCHED THEN INSERT – Oracle has no ON CONFLICT, but the duplicates can never come back.

Trade-offs taken into account:

No table lock. DELETE locks only the rows it removes, and in Oracle readers are never blocked – the ledger stays fully available during cleanup.
Undo and redo, not the search, dominate the cost (55k vs 2k buffer gets). At production scale, delete in batches (e.g. 10k rows per commit); if a large share of the table is duplicated, CREATE TABLE ... AS SELECT of the clean rows plus a rename is often faster.
Space is reused, not returned. Freed blocks serve future inserts, but the high-water mark stays. SHRINK SPACE reclaims it – and requires ROW MOVEMENT, which changes ROWIDs.
Engine differences matter. The same job on PostgreSQL with default work_mem (4 MB) spilled its sort to disk; Oracle kept it in memory. Same SQL idea, different tuning knobs.

Impact: 50,000 Duplicates Removed in 0.31 s / 0 Originals Lost Measured: 250k rows / Oracle Database 23ai

Case Study: The N+1 Query Storm

Banking Back-Office - ORM Performance with Spring Boot & Hibernate

!

The Challenge

Business context

The back-office accounts overview lists 500 accounts with balance, number of transactions and the latest activity. It took ~1.5 seconds to load – yet the database showed no slow queries, every index was in place, and the code looked perfectly clean.

The time was not lost in any single query. It was lost in how many of them one page triggered.

Reproduced for this case study with Java 25, Spring Boot, Hibernate and SQL Server 2022 on 500 accounts / 20,000 transactions – all numbers are real output, median of 20 runs after warm-up.

1

The Code — Looks Innocent

Lazy collection touched inside a loop

@Entity
public class Account {
    @OneToMany(mappedBy = "account")          // LAZY by default
    private List<AccountTransaction> transactions;
}

@Transactional(readOnly = true)
public List<AccountSummary> accountsOverview() {
    return repo.findAll().stream()            // 1 query: all accounts
               .map(this::summarize)
               .toList();
}

private AccountSummary summarize(Account a) {
    var txs = a.getTransactions();            // +1 query per account
    // balance = sum, count = size, last = max(createdAt)
}

Nothing here looks like a database call. But every getTransactions() on a lazy collection silently fires its own SELECT. One page = 1 + 500 queries.

2

What the Database Saw

sys.dm_exec_query_stats

execution_count|avg_ms|query_text
---------------+------+-------------------------------------------------------------
          13000| 1.435|select t1_0.account_id, t1_0.id, t1_0.amount, t1_0.created_at   ← [A] [B]
               |      |from transactions t1_0 where t1_0.account_id = @P0

Where the problem is:

[A] execution_count 13,000 = 500 per request × 26 requests in the benchmark. The same statement, with a different @P0 each time.
[B] avg 1.4 ms – every single query is fast. That is why N+1 never shows up in a slow-query log or a top-10 “most expensive queries” report. Nothing is slow; there is just far too much of it.
Where the 1.5 s goes: ~0.7 s of database time (500 × 1.4 ms), the rest in 500 network round-trips, JDBC driver work and hydrating 20,500 entities. On a real network, where the database sits in another zone, every extra round-trip makes it worse.

Root cause: not the database, not a missing index – the access pattern. A lazy association read inside a loop.

3

Fix 1 — JOIN FETCH

The textbook answer: 501 → 1 statement

@Query("select distinct a from Account a " +
       "left join fetch a.transactions order by a.id")
List<Account> findAllWithTransactions();

Result: 124 ms – 12× faster, but not done:

1 SQL statement instead of 501 – the round-trips are gone.
The query takes 29.7 ms in SQL Server, yet the request takes 124 ms: the remaining ~95 ms is Java hydrating 20,500 entities from 20,000 joined rows – only to sum them up and throw them away.

4

Fix 2 — DTO Projection

Let the database do the aggregation: 0 entities

@Query("""
       select new com.ela.nplusone.AccountSummary(
              a.accountNumber, a.ownerName,
              sum(t.amount), count(t), max(t.createdAt))
       from Account a left join a.transactions t
       group by a.id, a.accountNumber, a.ownerName
       order by a.id
       """)
List<AccountSummary> findSummaries();

Result: 15.9 ms – 92× faster than the original:

1 statement, 500 rows, 0 entities. SQL Server returns exactly what the screen shows – nothing more.
9.7 ms in the database, ~6 ms in the application. The aggregation moved to where the data lives.
Same output in all three variants – e.g. account PL…001: balance -7,887.36, 40 transactions – verified by the benchmark.

✓

Summary & Takeaways

Outcome and lessons

Metric	N+1	JOIN FETCH	DTO projection
SQL statements per request	501	1	1
Entities loaded	20,500	20,500	0
Response time (median)	1,473.5 ms	124.1 ms	15.9 ms
Time in database per query	1.4 ms × 501	29.7 ms	9.7 ms

Conclusionthe database was never the bottleneck – every query took about a millisecond. The cost was 501 round-trips hidden behind one getter, and 20,500 objects built only to be summed. JOIN FETCH removes the round-trips; projecting straight into DTOs also removes the work Java should never have done. 92× faster, same result, no schema change, no new index.

Takeaways:

Look for frequency, not duration. N+1 hides from slow-query logs. Sort by execution_count in sys.dm_exec_query_stats (PostgreSQL: pg_stat_statements.calls, Oracle: V$SQL.EXECUTIONS).
Count statements in tests. Hibernate statistics (getPrepareStatementCount()) let an integration test assert “this endpoint runs 1 query” – the regression can never sneak back in.
Lazy loading is a good default. The bug is touching a lazy association inside a loop.
Read-only screens need DTOs, not entities. Load entities when you intend to change them.
Quick mitigation: @BatchSize or hibernate.default_batch_fetch_size turns 1 + N into 1 + N / batch without touching queries – useful when a rewrite is not possible yet (not measured here).

Trade-offs taken into account:

JOIN FETCH on a collection multiplies rows – 20,000 rows for 500 accounts. Combined with pagination, Hibernate paginates in memory (warning HHH90003004): dangerous on large lists.
DTO projections are read-only. Perfect for overviews and reports; for editing, load the entities you change.
Aggregation moves load to the database. Cheap here thanks to the index on (account_id, created_at); on bigger data, check the plan of the aggregate query as well.
Measured locally on SQL Server 2022 running under emulation on Apple Silicon: absolute times are higher than on production hardware, the ratios between variants hold. With real network latency, the gap grows further.

Impact: 92x Faster / 501 → 1 SQL Statements / 0 Entities Loaded Measured: 500 accounts · 20k tx / SQL Server 2022

Case Study: Containing a Failing Dependency

Payment Service - Circuit Breaker with Kotlin & Resilience4j

!

The Challenge

Business context

A payment service asks an external fraud-check provider about every payment. One afternoon the provider degrades: it still answers, but in 5 seconds instead of 50 ms. Within seconds, payments start failing – and so does the account balance endpoint, which never calls the fraud service at all.

A slow dependency turned into a full outage. Nothing crashed; everything just waited.

Reproduced with Kotlin, Spring Boot 4 and Resilience4j 2.3; fraud provider simulated in WireMock (50 ms → 5 s); Tomcat with 50 worker threads; load from hey – 100 concurrent payment clients plus 5 balance clients. All numbers are real output.

1

The Code — No Boundary

Default HTTP client, direct call

private val defaultClient = RestClient.builder()
    .baseUrl(baseUrl)
    .build()                                   // no read timeout at all

@GetMapping("/payments/unprotected")
fun unprotected(): PaymentResult {
    val verdict = fraud.checkWithoutTimeout(id)      // holds a Tomcat thread
    return PaymentResult(id, verdict.decision, "fraud-service")   // for as long as the provider wants
}

With the provider healthy this works fine: 5,411 payments in 15 s, 357 req/s, p50 156 ms, p99 203 ms, zero errors. The problem only shows when the provider slows down.

2

Provider Slows to 5 s — Unprotected

hey: 100 payment clients + 5 balance clients, 30 s

GET /payments/unprotected                    100 concurrent
  Slowest: 15.0451 secs    Average: 10.0381 secs
  50% in 10.0390 secs                                         ← [B]
  [200]  57 responses
  [162] context deadline exceeded (client timeout)          ← [D]

GET /balance  – never calls fraud-service      5 concurrent
  Fastest: 15.0246 secs   Slowest: 15.0248 secs                ← [C]
  [200]  5 responses
  [5]   context deadline exceeded
  Requests/sec: 0.2855

Where the problem is:

[A] 50 threads, each parked for 5 s. Every payment request holds a Tomcat worker while it waits for the provider. After the first wave the pool is full.
[B] Latency in waves of 5 / 10 / 15 s. The histogram shows three clusters: requests that got a thread, those that waited one turnover, those that waited two.
[C] The unrelated endpoint dies too. /balance shares the same thread pool, so it queues behind the stuck payments: every successful call took exactly 15 s, half timed out.
[D] 74% of payments failed (162 of 219) – and the survivors took 5–15 s.

Root cause: no timeout and no isolation. The provider’s latency became our latency, and its capacity became our capacity.

3

The Fix

Timeout + circuit breaker + business fallback

private val timedClient = RestClient.builder().baseUrl(baseUrl)
    .requestFactory(JdkClientHttpRequestFactory(
        HttpClient.newBuilder().connectTimeout(Duration.ofMillis(300)).build()
    ).apply { setReadTimeout(Duration.ofMillis(800)) })
    .build()

val config = CircuitBreakerConfig.custom()
    .slidingWindowType(COUNT_BASED).slidingWindowSize(20)
    .minimumNumberOfCalls(10)
    .failureRateThreshold(50f)
    .slowCallDurationThreshold(Duration.ofMillis(500))
    .slowCallRateThreshold(50f)
    .waitDurationInOpenState(Duration.ofSeconds(10))
    .permittedNumberOfCallsInHalfOpenState(5)
    .build()

@GetMapping("/payments/protected")
fun guarded(): PaymentResult = try {
    val verdict = circuit.executeSupplier { fraud.checkWithTimeout(id) }
    PaymentResult(id, verdict.decision, "fraud-service")
} catch (e: CallNotPermittedException) {
    PaymentResult(id, "PENDING_REVIEW", "fallback: circuit open")
} catch (e: Exception) {
    PaymentResult(id, "PENDING_REVIEW", "fallback: ${e.javaClass.simpleName}")
}

Each part has one job:

Read timeout 800 ms → caps the cost of a single call. A thread is never parked longer than that.
Slow-call threshold 500 ms → the breaker reacts to slowness, not only to errors. A provider that answers in 5 s is treated as broken.
Window 20 / minimum 10 / 50% → decisions are based on recent evidence, not on one unlucky call.
Open for 10 s, then 5 probe calls → the breaker stops paying the cost entirely, and checks on its own whether the provider is back.
Fallback PENDING_REVIEW → the payment is accepted and queued for a later fraud check instead of hanging or being rejected.

4

Same Outage — Protected

Identical load, identical 5 s provider

GET /payments/protected                      100 concurrent
  50% in 0.0023 secs    99% in 0.0078 secs    Slowest: 1.6674 secs
  [200]  1000000 responses                                        ← [A]
  Requests/sec: 38464.3212

GET /balance  – never calls fraud-service      5 concurrent
  50% in 0.0016 secs    99% in 0.0040 secs    Slowest: 0.0405 secs   ← [B]
  [200]  64591 responses
  Requests/sec: 3229.1207

GET /circuit
  {"state":"OPEN","failureRate":100.0,"slowCallRate":100.0,
   "bufferedCalls":5,"notPermittedCalls":346634}                ← [C]

What changed:

[A] Every request answered, in 2.3 ms. Honest reading: these are fallbacks – the provider is still slow. The service does not pretend fraud checks happened; it stays responsive and parks payments for review. The throughput number shows how cheap a fast “no” is, not payment volume.
[B] /balance does not notice the outage: 1.6 ms median, 0 errors, 3,229 req/s – instead of 15 s and timeouts.
[C] The breaker is doing its job: 100% failure and slow-call rate, 346,634 calls rejected without touching the provider in the latest open period.
The slowest 1.67 s are ~110 calls: the first wave before the breaker opened, and the probe calls during half-open checks.

5

Self-Healing

State transitions from the application log

12:29:25.236  CLOSED    → OPEN        load hits, 100% slow + failed
12:29:35.237  OPEN      → HALF_OPEN   exactly 10 s later: probe
12:29:36.045  HALF_OPEN → OPEN        probes time out after 800 ms
12:29:46.045  OPEN      → HALF_OPEN
12:29:46.850  HALF_OPEN → OPEN
              ... provider recovers, traffic resumes ...
12:30:06.750  OPEN      → HALF_OPEN
12:30:06.814  HALF_OPEN → CLOSED      5 probes succeed within 64 ms

GET /payments/protected  (100 requests after recovery)
  50% in 0.0575 secs   [200] 100 responses
GET /circuit
  {"state":"CLOSED","failureRate":0.0,"slowCallRate":0.0,"bufferedCalls":20}

No restart, no manual switch. While the provider was down, the breaker re-checked every 10 s and stayed open. Once it was back, traffic returned to real fraud checks (~57 ms) on its own. Five requests that arrived while the probes were in flight got a fast fallback – half-open only lets the probes through.

✓

Summary & Takeaways

Outcome and lessons

During the outage	Unprotected	Circuit breaker
Payment requests answered	26% (57 / 219)	100%
Payment latency (median)	10.0 s	2.3 ms (fallback)
`/balance` latency (median)	15.0 s, 50% timeouts	1.6 ms, 0 errors
`/balance` throughput	0.29 req/s	3,229 req/s
Recovery when provider is back	–	automatic, 64 ms

Conclusionthe slow provider was not the outage – the missing boundary was. A timeout caps the cost of one call, the circuit breaker stops paying it altogether, and the fallback keeps the business moving. The rest of the service never noticed: an unrelated endpoint went from 15 s and timeouts to 1.6 ms, under the same failure.

Takeaways:

Every remote call needs a timeout. Default HTTP clients often have none – the JDK client used here waits as long as the server wants.
Slow is worse than down. A dead service fails fast; a slow one eats threads. Count slow calls (slowCallRateThreshold), not only errors.
The fallback is a business decision. “Accept and review later” vs “reject” is agreed with product and risk, not invented in code.
Make the state visible. Alert on OPEN and export breaker metrics – a silently open breaker means payments are quietly piling up for review.
This is what DORA resilience testing asks for: evidence that a failing third-party provider does not take down critical functions.

Trade-offs taken into account:

Thresholds need tuning per dependency. Too sensitive and the breaker flaps on noise; too tolerant and the cascade starts before it opens.
The first wave still pays. ~110 calls waited up to 1.7 s before the breaker opened and during probes – the timeout keeps that bounded.
Isolation goes one step further with a bulkhead: a separate, limited pool per dependency, so even before the breaker opens, one provider cannot take every thread.
Virtual threads (Java 21+) would remove the thread exhaustion – but not the 5 s wait for users or the load on a struggling provider. The breaker is still needed.
Measured locally with the provider simulated on the same machine; absolute numbers differ in production, the pattern does not.

Impact: Unrelated Endpoint 15 s → 1.6 ms / Payment Errors 74% → 0% Measured: Kotlin · Resilience4j · 100 concurrent clients

Case Study: The Payday Rebalance Storm

Ledger Consumer - Kafka with Kotlin & Quarkus

!

The Challenge

Business context

On payday, core banking publishes salary transfers to Kafka and a ledger consumer books each one. The ledger database is under load, so every booking takes a little longer than usual – and in the middle of the batch one consumer pod is killed and restarted.

The result: the same salaries booked again and again, while most of the batch was never booked at all. No exception, no outage – just a consumer group quietly going in circles.

Reproduced with Kotlin, Quarkus 3.40 and Apache Kafka (KRaft) using the plain consumer API: 10,000 transfers on 6 partitions keyed by account, 3 consumers, ~10 ms per booking; consumer c1 killed after 8 s and restarted 2 s later. max.poll.interval.ms scaled from 5 min to 3 s so the experiment fits in a minute – same ratio as 500 records × slow writes in production. All numbers are real output.

1

The Setup — Defaults That Look Harmless

Consumer configuration and booking loop

max.poll.records      = 500        // default
enable.auto.commit    = true       // default
max.poll.interval.ms  = 3000       // scaled down from 300000

while (running) {
    val records = consumer.poll(Duration.ofMillis(200))
    for (r in records) {
        ledger.book(r.value())        // ~10 ms, no duplicate check
    }
}

// one batch = 500 × 10 ms = 5 s   >   3 s allowed between polls

2

What Happened — 60 Seconds

GET /run/naive

{
  "completed": false,
  "elapsedSeconds": 60.1,
  "transfersPublished": 10000,
  "uniqueTransfersBooked": 1500,                ← [C]
  "totalLedgerBookings": 9545,
  "doubleBookedTransfers": 1500,
  "extraBookings": 8045,                        ← [C]
  "groupGeneration": 13,                        ← [A]
  "notYetBooked": 8500                          ← [D]
}

Where the problem is:

[A] 13 rebalances in 60 s. Every batch takes 5 s – longer than the 3 s the group allows between polls. The broker declares each consumer dead and moves its partitions to someone else.
[B] Auto-commit happens on the next poll – which comes too late. The finished work is never committed, so the next owner of the partition starts again from the old offset.
[C] The same 1,500 transfers, booked over and over. Every transfer that got booked was booked more than once – ~6.4 times on average, 8,045 extra ledger entries. Real money posted repeatedly.
[D] 85% of salaries never booked. After 60 s, 8,500 of 10,000 were untouched. The loop is stable – it would never finish on its own.

Root cause: batch processing time longer than max.poll.interval.ms, combined with a consumer that assumes every message arrives exactly once. Kafka promises at-least-once.

3

The Fix

Batch size, explicit commits, cooperative rebalancing, idempotency

max.poll.records              = 50      // 50 × 10 ms = 0.5 s  «  3 s
enable.auto.commit            = false
partition.assignment.strategy = CooperativeStickyAssignor

override fun onPartitionsRevoked(partitions: Collection<TopicPartition>) {
    consumer.commitSync()                    // hand over only finished work
}

for (r in records) {
    if (processedKeys.add(r.value())) {      // idempotency key = transfer id
        ledger.book(r.value())
    } else {
        duplicatesSkipped++                  // redelivery: ignore
    }
}
consumer.commitSync()                        // after every batch

Each part solves one problem:

50 records per poll → fixes [A]: a batch now takes ~0.5 s, far inside the poll interval. Consumers stop being evicted.
Manual commit after each batch and on revoke → fixes [B]: finished work is committed before partitions move, so the next owner starts where we stopped.
Cooperative rebalancing → when a consumer leaves or joins, only its partitions move; the others keep working.
Idempotency key → fixes [C]: a redelivered transfer is recognised and skipped. In production this is a unique constraint on transfer_id in the same database transaction as the booking; the in-memory set here is a stand-in.

4

Same Payday, Same Crash

GET /run/fixed

{
  "completed": true,
  "elapsedSeconds": 50.9,
  "transfersPublished": 10000,
  "messagesDelivered": 10010,
  "uniqueTransfersBooked": 10000,
  "totalLedgerBookings": 10000,
  "doubleBookedTransfers": 0,
  "duplicatesSkippedByIdempotency": 10,          ← [B]
  "groupGeneration": 2,                               ← [C]
  "commitFailures": 2                                 ← [D]
}

What changed:

[A] Every salary booked exactly once – 10,000 of 10,000 in 50.9 s, despite a consumer being killed mid-batch.
[B] The crash still caused redeliveries. 10 transfers processed by c1 but not yet committed came back to another consumer – that is at-least-once working as designed. Idempotency caught all 10. Without it: 10 double salaries.
[C] 2 group generations instead of 13 – the initial join and the crash/restart. Nothing else.
[D] 2 commit failures during the rebalance – expected and harmless: work was committed in onPartitionsRevoked, and idempotency covers anything in between.

✓

Summary & Takeaways

Outcome and lessons

Payday batch, one pod killed	Defaults	Fixed
Batch completed	no (stopped at 60 s)	yes, 50.9 s
Transfers booked	1,500 of 10,000	10,000 of 10,000
Double-booked transfers	1,500 (every one)	0
Extra ledger bookings	8,045	0
Redeliveries caught	–	10
Rebalances (group generation)	13	2

ConclusionKafka did not lose or duplicate anything on its own – it did exactly what at-least-once promises. Two things broke the ledger: a batch that took longer than the consumer was allowed, and a consumer that assumed every message arrives once. Smaller batches with explicit commits end the rebalance storm; idempotency makes the inevitable redeliveries harmless. Same load, same crash: every salary booked exactly once.

Takeaways:

Rule of thumb: max.poll.records × worst-case processing time < max.poll.interval.ms, with margin. Use p99 processing time, not the average – payday is not an average day.
At-least-once means duplicates will happen. Crashes, deployments and rebalances all cause redeliveries. Make the side effect idempotent: key + unique constraint in the same transaction.
Auto-commit is not “committed when processed”. It commits on the next poll – too late when processing is slow, too early when it crashes mid-batch.
Watch the group, not only the lag. A generation number that keeps rising and a high rebalance rate are early warnings of a storm.
Framework choice changes the failure mode, not the rule. SmallRye Reactive Messaging (Quarkus) decouples polling from processing, which prevents this eviction pattern – but redeliveries still happen on crashes and deployments, so idempotency is still required.
Connects to case 02: there, duplicates were cleaned up after the fact with ROWID; here they are stopped at the source.

Trade-offs taken into account:

Smaller batches mean more commits. commitSync per batch adds latency – the right default for a ledger. For higher throughput: commitAsync per batch plus commitSync on revoke and shutdown.
Raising max.poll.interval.ms only hides the problem – a genuinely stuck consumer then takes longer to be replaced.
Kafka transactions (exactly-once) do not cover database writes. They protect Kafka-to-Kafka flows; once the side effect is a ledger entry, idempotency is still required.
Scaled timings: 3 s poll interval and 10 ms bookings instead of 5 min and real database latency. The mechanism and the ratios are the same; absolute times differ.

Impact: 8,045 Double Bookings → 0 / 13 Rebalances → 2 Measured: 10k transfers · 3 consumers · Kafka KRaft

Case Study: The RabbitMQ Poison Loop

E-Commerce Order Queue - RabbitMQ with Java & Spring AMQP

!

The Challenge

Business context

An online shop processes orders from a RabbitMQ queue. A partner integration occasionally sends a truncated JSON order. The consumer fails to parse it – and the framework, by default, puts it back on the queue. It fails again. And again.

Healthy orders keep flowing, so dashboards look green. Meanwhile CPU climbs, logs fill with stack traces, and the broken orders are never seen by anyone – they are neither processed nor rejected. The loop never ends on its own.

Reproduced with Java 25, Spring Boot 4.1, Spring AMQP and RabbitMQ 4 (Docker): one consumer with Spring defaults (prefetch 250), 500 orders/s for 10 s (5,000 orders) with 3 truncated orders at 1 s, 4 s and 7 s, then 10 s without new traffic. All numbers are real output.

1

The Code — A Normal Listener

Parse, process, let exceptions propagate

@RabbitListener(queues = "orders")
public void onOrder(String body) {
    Order order = mapper.readValue(body, Order.class);   // throws on malformed JSON
    process(order);
}

// Spring AMQP default: listener exception -> message rejected with
// requeue = true -> back at the head of the queue -> delivered again

Nothing here is wrong in isolation. Spring does treat its own conversion errors as fatal – but a parse error thrown inside your listener is wrapped and treated as retryable. For a message that can never be parsed, “retry” means “forever”.

2

What Happened

GET /run/naive — 3 bad orders among 5,000

{
  "healthyProcessed": 5000,
  "latencyP99ms": 17,
  "latencyMaxMs": 46,                         ← [D]
  "poisonDeliveriesDuringRun": 4751,          ← [A]
  "poisonDeliveriesInNext10sIdle": 42219,     ← [B]
  "messagesInDlq": 0,                          ← [E]
  "avgCpuPercentOfOneCore": 55.5              ← [C]
}

Where the problem is:

[A] 3 bad orders, 4,751 deliveries in 10 s of traffic. Every attempt: parse, exception, full stack trace in the log, reject, requeue.
[B] The loop outlives the traffic. In the 10 s after the last order, with nothing left to do, the consumer took the same 3 messages 42,219 times – ~4,200 per second, indefinitely.
[C] 55.5% of a CPU core burned on work that can never succeed – 6.7× the fixed variant.
[D] Healthy orders slow down as the bad ones keep cutting in line: p99 17 ms instead of 4 ms, worst case 46 ms instead of 7 ms.
[E] Nobody can see the broken orders. No dead-letter queue, no alert – three customers’ orders are stuck in a loop, neither processed nor rejected.

Root cause: a permanent error treated as a transient one, with no upper bound on redeliveries.

3

The Fix

Dead-letter queue + classify the error

@Bean
Queue orders() {
    return QueueBuilder.durable("orders")
            .deadLetterExchange("")
            .deadLetterRoutingKey("orders.dlq")      // rejected -> orders.dlq
            .build();
}

@RabbitListener(queues = "orders")
public void onOrder(String body) {
    Order order;
    try {
        order = mapper.readValue(body, Order.class);
    } catch (JacksonException e) {
        // can never succeed: do not requeue, park it for a human
        throw new AmqpRejectAndDontRequeueException("Malformed order -> DLQ", e);
    }
    process(order);
}

The principle behind it:

Permanent errors – malformed payload, failed validation – will fail the same way every time. Reject without requeue: one attempt, straight to the DLQ.
Transient errors – database down, timeout – deserve a retry, but a bounded one with backoff (not part of this experiment).
The DLQ is an inbox, not a bin. The message keeps its body and headers (including x-death), so someone can inspect it, fix the cause and replay it.

4

Same Traffic, Same Bad Orders

GET /run/fixed

{
  "healthyProcessed": 5000,
  "latencyP99ms": 4,
  "latencyMaxMs": 7,
  "poisonDeliveriesDuringRun": 3,
  "poisonDeliveriesInNext10sIdle": 0,
  "messagesInDlq": 3,
  "avgCpuPercentOfOneCore": 8.3
}

What changed:

Each bad order is delivered exactly once – 3 deliveries for 3 messages, then silence: 0 attempts once traffic stops.
CPU drops from 55.5% to 8.3% of a core; the consumer only works when there is work.
Healthy orders stay fast: p99 4 ms, worst case 7 ms.
All 3 broken orders are waiting in orders.dlq – visible, countable, alertable.

5

Under Peak Load

850 orders/s — consumer near capacity

                         default requeue      DLQ
latency p50              798 ms               688 ms
latency p99              1,653 ms             1,435 ms
poison deliveries idle   41,397               0
CPU (% of one core)      43.5                 10.9

An honest result: at 850 orders/s a single consumer is at its limit, and latency in both variants is dominated by capacity, not by the poison messages – the loop adds ~15%. The fix does not create capacity. What it removes is the permanent waste and the invisible orders, which show up identically at any load.

✓

Summary & Takeaways

Outcome and lessons

500 orders/s, 3 malformed	Default requeue	DLQ
Bad-order deliveries during traffic	4,751	3
Bad-order deliveries after traffic (10 s)	42,219	0
CPU (average, % of one core)	55.5%	8.3%
Healthy order latency p99 / max	17 / 46 ms	4 / 7 ms
Bad orders visible for investigation	0 (stuck in the loop)	3 in DLQ

Conclusionthree malformed messages were enough to keep a consumer busy forever – 4,200 futile deliveries per second long after the real work was done – while the customers behind those orders were silently forgotten. Classifying the error and adding a dead-letter queue turned an endless, invisible loop into three messages in an inbox, and cut CPU by 85%.

Takeaways:

Classify errors before you retry. Permanent → DLQ immediately. Transient → bounded retry with backoff.
No unbounded requeue, ever. In RabbitMQ 4, quorum queues have a default delivery limit; classic queues – used here – have none. Know which one you run.
Green dashboards can hide a loop. Healthy traffic still flows. Watch the redelivery rate and the DLQ depth, not only throughput.
A DLQ needs an owner. Alert when it is non-empty, and have a way to inspect and replay messages after the cause is fixed.

Trade-offs taken into account:

Rejecting fast moves work to people. Messages in the DLQ are no longer processed automatically – that needs an operational process.
Keep the “permanent” list narrow. Misclassify a database timeout as permanent and a short outage sends thousands of good orders to the DLQ.
Retries inside the listener block the consumer. For longer backoff, use a retry queue with a TTL instead of sleeping in the handler.
The fix does not add capacity – the peak-load run shows latency there is driven by consumer throughput; scale consumers for that.
Measured locally with broker and consumer on one machine; absolute numbers differ in production, the loop behaves the same.

Impact: 42,219 → 0 Idle Redeliveries / CPU 55% → 8% Measured: 5k orders · 3 malformed · RabbitMQ 4

Case Study: The AppSec Gate

Banking API - Object-Level Authorization & Dependency Vulnerabilities in Java & Spring Security

!

The Challenge

Business context

A bank exposes an account API to its customers: details, transactions, statements, transfers, limits. Every endpoint takes an account id. Login works, passwords are checked, every request is authenticated – and yet nothing checks whether the account in the URL belongs to the caller. This is Broken Object Level Authorization, number one on the OWASP API Security Top 10.

The second risk sits below the code: the libraries it runs on. A project generated today, on the current Spring Boot release, already ships with known critical vulnerabilities – and a quick scan of pom.xml says “No issues found”.

Reproduced with Java 25, Spring Boot 4.1.1, Spring Security, JUnit 5 + spring-security-test (MockMvc), CycloneDX Maven plugin and osv-scanner 2.6.0. Two demo customers: alice owns accounts 1–2, bob owns 3–4. Scans run on 8 Oct 2026 against the OSV database. All numbers are real output.

1

The Code — Authenticated, Not Authorized

Spring Security does its job; the gap is one level deeper

http.authorizeHttpRequests(a -> a.anyRequest().authenticated())   // who are you? - checked
    .httpBasic(Customizer.withDefaults());

@GetMapping("/{id}/statement")
public Map<String, Object> statement(@PathVariable long id, Principal me) {
    Account a = store.find(id).orElseThrow(NOT_FOUND);  // is it yours? - never asked
    return Map.of("iban", a.iban(), "closingBalance", a.balance());
}

The filter chain answers who is calling. Only the application knows which accounts that person may touch – so no framework setting and no generic scanner closes this gap. It has to be in the code, and it has to be tested.

2

The Test — An Authorization Matrix

5 endpoints × 3 callers = 15 cases

// alice owns account 1
static Stream<Arguments> endpoints() {
    return Stream.of(
        Arguments.of("GET /accounts/{id}",              get("/accounts/1")),
        Arguments.of("GET /accounts/{id}/transactions", get("/accounts/1/transactions")),
        Arguments.of("GET /accounts/{id}/statement",    get("/accounts/1/statement")),
        Arguments.of("POST /accounts/{id}/transfers",   post("/accounts/1/transfers")...),
        Arguments.of("PUT /accounts/{id}/limits",       put("/accounts/1/limits")...));
}

@ParameterizedTest @MethodSource("endpoints")
void ownerIsAllowed(...)        { /* alice -> 2xx */ }

@ParameterizedTest @MethodSource("endpoints")
void otherCustomerIsDenied(...) { mvc.perform(request.with(httpBasic("bob", "bob")))
                                     .andExpect(status().isNotFound()); }

@ParameterizedTest @MethodSource("endpoints")
void anonymousIsRejected(...)   { /* no credentials -> 401 */ }

Most test suites cover the happy path: the owner gets their data. The matrix adds the question that matters for security – what does everyone else get? – for every endpoint, on every build.

3

What the Matrix Found

./mvnw test — without the ownership check

[ERROR] Tests run: 15, Failures: 5, Errors: 0  -- in AuthorizationMatrixTest
[ERROR] otherCustomerIsDenied(...)[1]  FAILURE!   GET  /accounts/{id}
[ERROR] otherCustomerIsDenied(...)[2]  FAILURE!   GET  /accounts/{id}/transactions
[ERROR] otherCustomerIsDenied(...)[3]  FAILURE!   GET  /accounts/{id}/statement
[ERROR] otherCustomerIsDenied(...)[4]  FAILURE!   POST /accounts/{id}/transfers
[ERROR] otherCustomerIsDenied(...)[5]  FAILURE!   PUT  /accounts/{id}/limits
[ERROR] Tests run: 16, Failures: 5, Errors: 0, Skipped: 0

Where the problem is:

5 of 5 endpoints served another customer’s account to any logged-in user.
Read and write. Not only balance and IBAN on the statement – the same gap accepted a transfer from someone else’s account and a change to their limits.
All other tests were green. Owners got their data (5/5), anonymous calls got 401 (5/5). Without the cross-customer column the build would have passed.

Root cause: authentication treated as authorization – the resource is loaded by id, and ownership is never checked.

4

The Fix

One ownership check, every endpoint goes through it

public Account accountFor(long id, String user) {
    Account account = store.find(id)
            .orElseThrow(() -> new ResponseStatusException(NOT_FOUND));
    if (!account.owner().equals(user)) {
        throw new ResponseStatusException(NOT_FOUND);   // not 403
    }
    return account;
}

// every controller method: Account a = service.accountFor(id, me.getName());

The principle behind it:

One place, not five. The check lives in the service that loads the account, so a new endpoint cannot “forget” it by accident.
404, not 403. A foreign account and a non-existent one look identical, so responses do not reveal which ids exist.
The matrix is the guard rail. Remove the check in any future refactoring and five tests fail the build.

[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

5

Dependencies — Upgrading Once Is Not Enough

osv-scanner on the same 4 libraries, three versions of history

                          log4j-core  commons-text  snakeyaml  jackson-databind
legacy                    2.14.1      1.9           1.30       2.13.0
  -> 7 packages, 34 vulns (3 Critical, 15 High, 16 Medium)

earlier one-off upgrade   2.24.3      1.13.0        2.4        2.18.2
  -> 5 packages, 23 vulns (0 Critical, 8 High, 15 Medium)

latest stable, Oct 2026   2.26.1      1.15.0        2.7        2.22.3
  -> No issues found

4 declared libraries turned into 7 affected packages – the rest came in transitively. And the “fixed” set from an earlier upgrade still carried 23 known issues: vulnerabilities are published after you upgrade, so a clean result is only true on the day it was produced.

6

A Brand-New App — Scan What Actually Ships

Spring Boot 4.1.1, generated the same day

$ osv-scanner scan source .
Filtered 4 local/unscannable package/s from the scan.
No issues found                                   ← [A]

$ ./mvnw cyclonedx:makeAggregateBom && osv-scanner scan source -L target/bom.json
Scanned target/bom.json file and found 45 packages
Total 3 packages affected by 10 known vulnerabilities
      (3 Critical, 5 High, 2 Medium)                   ← [B]
0 vulnerabilities can be fixed.                   ← [C]

  tomcat-embed-core  11.0.24   9.8 / 9.1 / 9.1
  jackson-core       3.1.5     7.5 / 7.5
  jackson-databind   3.1.5     7.5 / 7.5 / 7.5 / 5.6 / 5.3

What the scans show:

[A] A false all-clear. Versions managed by the Spring Boot parent are invisible in pom.xml, so the source scan silently skipped them.
[B] The SBOM sees the truth: 45 resolved packages, 10 known vulnerabilities, 3 of them critical – in an app with no business code yet.
[C] “0 can be fixed” means no newer Spring Boot release exists: the only newer parent was a milestone (4.2.0-M2), not something to run in production.

7

Triage & Patch

Reachability first, then override the managed versions

Triage example — CVSS is not the whole story:

The top finding (9.8) is CVE-2026-65905 in Tomcat’s DIGEST authenticator. This API uses Basic authentication over Spring Security, so the vulnerable code path is not reachable here – lower urgency, still patched, because configuration changes and “not reachable today” is not a guarantee. The Jackson findings are denial-of-service issues in JSON parsing, and this API parses JSON from the network – reachable, patch now.

<properties>
    <java.version>25</java.version>
    <tomcat.version>11.0.25</tomcat.version>          <!-- managed: 11.0.24 -->
    <jackson-bom.version>3.1.7</jackson-bom.version>    <!-- managed: 3.1.5  -->
</properties>

tomcat 11.0.25, jackson 3.1.6   ->  2 packages, 4 vulns (0 Critical, 4 High)
tomcat 11.0.25, jackson 3.1.7   ->  No issues found
./mvnw test                     ->  Tests run: 16, Failures: 0   BUILD SUCCESS

What changed:

10 → 4 → 0. The first patch (Jackson 3.1.6) closed the advisories known when it was released; four newer ones were fixed only in 3.1.7. Same day, two rounds.
Two lines of configuration, no code change, no framework upgrade.
The authorization matrix ran after every bump – 16/16 green – so the patch did not reopen the access-control gap.

✓

Summary & Takeaways

Outcome and lessons

Check	Before	After
Endpoints open to another customer	5 of 5	0 of 5
Authorization matrix (owner / other / anonymous)	10 / 15 pass	15 / 15 pass
Legacy libraries — known vulns	34 (3 critical)	0
New Spring Boot app (SBOM, 45 pkgs) — known vulns	10 (3 critical)	0
Source scan of `pom.xml`	“No issues” (4 skipped)	SBOM scan in CI

Conclusionboth risks were invisible to the usual signals – a green test suite and a clean scan. A 15-case authorization matrix exposed five endpoints that let any customer read and modify someone else’s account; one ownership check closed them. Scanning the resolved SBOM instead of the build file revealed 10 known vulnerabilities in a brand-new app; triage plus two version overrides brought it to zero. Neither fix is a one-off – both are now build gates.

Takeaways:

Authenticated is not authorized. Every endpoint that takes an id needs an ownership check – and a test that calls it as somebody else.
Test the matrix, not the happy path. Owner, other customer, anonymous – for every endpoint, on every build.
Scan what ships. Generate an SBOM from the resolved dependencies; a build-file scan can miss everything the framework manages.
SCA is a gate, not a project. The result changed within one afternoon. Run it in CI and on a schedule, and fail the build on new critical findings.
Triage by reachability, patch anyway. Reachability sets the order and urgency; it is not a reason to keep a known-vulnerable version.
Regulation points the same way. The EU Cyber Resilience Act requires manufacturers to keep an SBOM, and DORA expects documented vulnerability and patch management – this pipeline produces the evidence.

Trade-offs taken into account:

Overriding managed versions moves away from the combination Spring Boot tested. Run the full suite, and drop the overrides once a Boot patch release includes them.
404 instead of 403 hides existence from callers but also from support – log the real reason server-side.
The matrix only covers endpoints listed in it. New endpoints must be added; a check that every mapped route appears in the matrix closes that gap.
Vulnerability counts are a snapshot of the OSV database on 8 Oct 2026 and will differ on any other day – which is the point of running them continuously.
Demo scope: in-memory users and accounts, one ownership rule. Real systems add shared accounts, delegated access and roles – more rows in the same matrix.

Impact: 5 → 0 Open Endpoints / 10 → 0 Known Vulns Measured: 15-case matrix · 45-package SBOM · OSV

Case Study: The GC Pause Tail

Payment Pricing Service - JVM Garbage Collection with Kotlin on Java 25

!

The Challenge

Business context

A payment service prices every transfer against a large in-memory cache of FX quotes. The median request takes 0.07 ms. Dashboards are green. Yet a steady trickle of requests takes 60–90 ms – long enough to hit an upstream timeout – and nothing in the code explains it.

Averages hide this completely. At 5,000 requests per second, “one in a thousand” is five slow payments every second, all day.

Reproduced with Kotlin 2.3 on Java 25, MacBook Pro (8 cores, 16 GB): 4 GB heap, 4 million cached quotes (~1.1 GB live), 50,000 quote refreshes/s, 5,000 requests/s. 60 s measured after 10 s warm-up, 300,000 requests per run. Latency counted from the scheduled start, so requests queued behind a pause are included. Every configuration run 3 times, median shown. All numbers are real output.

1

The Workload

A long-lived cache plus short-lived request garbage

// long-lived: 4M quotes in the old generation, 50k replaced per second
val cache = ConcurrentHashMap<Long, Quote>(entries * 2)

// per request: parse a 200-line batch (short-lived objects), price it
fun handle(cache: ConcurrentHashMap<Long, Quote>, entries: Int) {
    val lines = ArrayList<Line>(200)
    repeat(200) { lines.add(Line(r.nextLong(), r.nextDouble(1000.0), "transfer-" + r.nextInt())) }
    val payload = ByteArray(16 * 1024)
    repeat(20) { sum += cache[r.nextLong(entries.toLong())]?.ask ?: 0.0 }
}

// fixed rate; latency = finish - SCHEDULED start (no coordinated omission)
workers.execute { handle(cache, entries); lat[i] = System.nanoTime() - intended }

A very common shape: a big, slowly changing reference cache and a high rate of small, short-lived objects. Nothing in it is a bug.

2

What Happened — G1 Defaults

java -Xmx4g -XX:+UseG1GC

latency p99      0.29 ms
latency p99.9    58.08 ms                               ← [A]
latency max      80.69 ms

gc.log, measured window:
  Pause Young (Normal) (G1 Evacuation Pause)  x 7
  average ~63 ms, longest ~80 ms                      ← [B]

Where the problem is:

[A] The tail is 200× the p99. 99% of requests finish in 0.29 ms; the slowest 0.1% take 58 ms or more.
[B] The longest GC pause matches the slowest request – ~80 ms vs 80.69 ms. During a stop-the-world pause every application thread is frozen, so every request that arrives waits it out.
Few, long pauses: only 7 young collections in 60 s, but each one copies a large young generation while scanning references from a 1 GB old generation.

Root cause: stop-the-world young-generation pauses, sized for throughput, not for tail latency.

3

First Attempt — Tune G1

-XX:MaxGCPauseMillis=20 (default 200)

                    G1 default     G1 pause goal 20 ms
pauses (60 s)       7              37
average pause       ~63 ms         ~17 ms
longest pause       ~80 ms         ~24 ms
latency p99         0.29 ms        3.65 ms
latency p99.9       58.08 ms       16.85 ms
latency max         80.69 ms       24.70 ms

One flag, a real trade-off:

p99.9 improves 3.4× – G1 shrinks the young generation to meet the goal, so each pause is shorter.
But there are 5× more pauses, so more requests hit one: p99 gets 12× worse (0.29 → 3.65 ms).
Pauses are still tens of milliseconds – the goal is a target, not a guarantee (longest ~24 ms against a 20 ms goal).

4

The Fix — A Concurrent Collector

-XX:+UseZGC (generational by default on Java 25)

latency p99      0.50 ms
latency p99.9    3.51 ms
latency max      19.72 ms

gc.log, measured window:
  Pause Mark Start / Mark End / Relocate Start  x 15
  longest pause    0.018 ms          (0.074 ms worst across all 6 ZGC runs)
  allocation stalls  0
CPU                0.26 cores        (G1 default: 0.22)

What changed:

p99.9 drops from 58.08 to 3.51 ms – 16.5×, without the p99 penalty of tuned G1.
GC pauses are now microseconds. ZGC marks and moves objects while the application keeps running; it stops threads only to flip a few pointers.
No code change, one JVM flag – and no allocation stalls, so the heap had enough headroom.

5

Reading the Remaining Tail Honestly

What is left is not GC — and single runs lie

run          p99.9      p99.99     max
G1 default   58.08      76.13      80.69     (runs: 54.29 / 58.08 / 63.70)
G1 tuned     16.85      21.62      24.70     (runs: 15.89 / 16.85 / 18.71)
ZGC          3.51       15.69      19.72     (runs: 2.86 / 3.51 / 5.71)
ZGC, 8 GB    8.64       20.48      25.91     (one run: 150.31, GC pauses ≤ 0.074 ms)

What the logs rule out:

ZGC’s remaining 15–20 ms outliers are not GC. No pause above 0.074 ms and zero allocation stalls in any run – by elimination, OS scheduling on a busy laptop (an inference, not a measurement).
A bigger heap did not help. With no stalls there was nothing for extra memory to fix – and one 8 GB run hit 150 ms with microsecond GC pauses, most likely memory pressure on a 16 GB machine.
The very first G1 run showed 116 ms at p99.9, twice the later median. One run is an anecdote; three runs and a median are a measurement.

✓

Summary & Takeaways

Outcome and lessons

5,000 req/s, median of 3 runs	G1 default	G1 tuned	ZGC
Latency p99	0.29 ms	3.65 ms	0.50 ms
Latency p99.9	58.08 ms	16.85 ms	3.51 ms
Latency max	80.69 ms	24.70 ms	19.72 ms
GC pauses / longest	7 / ~80 ms	37 / ~24 ms	15 / 0.018 ms
CPU (cores used)	0.22	0.27	0.26

Conclusionthe slowest payments were not slow code – they were requests frozen behind stop-the-world GC pauses, proven by the longest pause matching the slowest request to the millisecond. Tuning G1 traded a better p99.9 for a 12× worse p99. Switching to generational ZGC removed the pauses from the picture: p99.9 from 58 to 3.5 ms, one flag, no code change, at roughly 0.04 extra CPU cores.

Takeaways:

Measure the tail, not the average. p50 was 0.07–0.08 ms in every variant – it could not tell them apart.
Count from the scheduled start. Measuring only service time hides the requests that queued behind a pause (coordinated omission).
Prove the cause before changing anything. GC logs turn “probably GC” into “the 80 ms pause is the 80 ms request”.
Check every percentile after tuning. MaxGCPauseMillis fixed p99.9 and quietly broke p99.
Repeat runs and use medians. On a shared machine a single run can be off by 2×.

Trade-offs taken into account:

ZGC costs CPU: 0.26 vs 0.22 cores here (+18%), spent on concurrent GC threads. It needs spare cores; on a CPU-saturated container the gain shrinks.
ZGC needs heap headroom. Too small a heap leads to allocation stalls – the pauses come back in another form. Watch stalls in the GC log.
Not every service needs it. Batch and throughput-oriented jobs do fine with G1 or Parallel GC; ZGC is for latency SLAs.
Measured on a laptop with background load; absolute values differ on production hardware, the relative picture and the log evidence hold.

Impact: p99.9 58.1 → 3.5 ms / Pauses ~80 ms → 0.02 ms Measured: 4 configs × 3 runs · 300k requests each · Java 25

Case Study: Connection Pool Exhaustion

Payment Service Outage - Spring Boot, HikariCP & PostgreSQL

!

The Challenge

Business context

A payment service scores every transfer with an external fraud API. For months the API answers in 50 ms. One afternoon it slows to 3 seconds. Within a second the whole service is down – not only payments, but also balance checks that never call the fraud API. Logs fill with Connection is not available, request timed out.

The database dashboard is green: CPU idle, no slow queries. The first instinct in the incident call is “make the pool bigger”.

Reproduced with Java 25, Spring Boot 4.1.1, HikariCP 7.0.2 and PostgreSQL 17 (Docker). Virtual threads, so the connection pool is the only limit. Pool of 10 (Spring Boot default); connection timeout shortened from 30 s to 5 s. Open-model load: 20 payments/s plus 20 balance reads/s for 20 s; the fraud API is simulated with a configurable delay. All numbers are real output.

1

The Code — Looks Perfectly Normal

One transactional method, one remote call inside

@Transactional                                       // ← connection taken HERE
public long payNaive(long account, BigDecimal amount) {
    jdbc.queryForObject("SELECT balance FROM accounts WHERE id = ?", ...);
    Long id = jdbc.queryForObject("INSERT INTO payments ... 'PENDING' RETURNING id", ...);
    fraud.check(account, amount);                     // remote call: connection held while we wait
    jdbc.update("UPDATE payments SET status = 'APPROVED' WHERE id = ?", id);
    return id;
}                                                    // ← connection returned HERE

With a 50 ms fraud API this is invisible: each payment holds a connection for ~60 ms. The pool size quietly assumes that a partner will always be fast.

2

Healthy Baseline

Fraud API at 50 ms

payments       400 / 400 ok    p50 62.5 ms
balance reads  400 / 400 ok    p50 7.8 ms
pool           peak 2 / 10 active, 0 waiting

3

The Incident

Fraud API at 3 s — same code, same pool

payments       90 / 400 ok     p50 5,012 ms                   ← [A]
balance reads  76 / 400 ok     p50 5,010 ms                   ← [B]

t= 1s active=10/10 waiting=16
t= 3s active=10/10 waiting=98
t= 7s active=10/10 waiting=200                                  ← [C]
t=14s active=10/10 waiting=202

Where the problem is:

[A] Simple arithmetic: 10 connections held 3 s each serve ~3.3 payments/s. 20 arrive every second – the pool is full within the first second.
[B] The collateral damage is the real outage. Balance reads need one 1-ms query, but they queue for the same pool – 81% of them fail after waiting 5 s.
[C] 200 requests waiting is the steady state: 40 requests/s arrive, each waits up to 5 s for a connection, then gives up with an error.

4

Diagnosis — Who Holds the Connections?

The database view and the pool’s own evidence

$ SELECT state, count(*) FROM pg_stat_activity WHERE datname = 'bank' GROUP BY state;
        state        | count
 idle in transaction |    10                         ← the database is waiting for US

$ SELECT state, left(query, 60) FROM pg_stat_activity WHERE state <> 'idle';
 idle in transaction | INSERT INTO payments(account_id, amount, status) VALUES ($1,
 idle in transaction | INSERT INTO payments(account_id, amount, status) VALUES ($1,
 ...  (10 rows)

spring.datasource.hikari.leak-detection-threshold=2000

WARN  ProxyLeakTask : Connection leak detection triggered ... stack trace follows
java.lang.Exception: Apparent connection leak detected
    at com.zaxxer.hikari.HikariDataSource.getConnection(...)
    at o.s.jdbc.datasource.DataSourceTransactionManager.doBegin(...)  ← taken by the transaction
    at o.s.transaction.interceptor.TransactionInterceptor.invoke(...)
    at com.ela.pool.PaymentService$$SpringCGLIB$$0.payNaive(<generated>)
    at com.ela.pool.PaymentController.naive(PaymentController.java:30)

// 40 warnings in a 5 s run = every successful payment held its connection > 2 s

Reading the evidence:

idle in transaction is the fingerprint. The transaction is open, the last statement finished, and the application is busy elsewhere – here, waiting for the fraud API.
The stack trace names the culprit without guessing: the connection is acquired by @Transactional on entry to payNaive, not by any query.
It is not a true leak – every connection does come back, just 3 s too late. Hikari reports it as an “apparent” leak for exactly this reason.

5

The Reflex — A Bigger Pool

maximum-pool-size 10 → 30, code unchanged

payments       257 / 400 ok    p50 5,021 ms
balance reads  256 / 400 ok    p50 4,952 ms
pool           peak 30 / 30 active, 200 waiting

Three times the connections, still over a third of all traffic lost. 30 connections held 3 s serve 10 payments/s against 20 arriving. To survive this slowdown the pool would need 60 connections – and 120 if the partner slows to 6 s. The required pool size is set by someone else’s latency, and every extra connection is a real backend process on the database.

6

The Fix — Move the Transaction Boundary

Remote call first, short transaction after — pool back to 10

public long payFixed(long account, BigDecimal amount) {
    fraud.check(account, amount);                     // no connection, no transaction
    return tx.execute(status -> {                         // connection held only for the SQL
        jdbc.queryForObject("SELECT balance FROM accounts WHERE id = ?", ...);
        return jdbc.queryForObject("INSERT INTO payments ... 'APPROVED' RETURNING id", ...);
    });
}

payments       400 / 400 ok    p50 3,011.5 ms   (the fraud API is still slow)
balance reads  400 / 400 ok    p50 7.9 ms
pool           peak 1 / 10 active, 0 waiting

What changed:

Every request succeeds – 800 of 800, with the partner still answering in 3 s.
Balance reads are back to 7.9 ms – the same as on a healthy day. The slowdown stays where it belongs: in the payment path.
One connection at peak instead of 10 – or 30. Connection hold time no longer depends on any external system.

✓

Summary & Takeaways

Outcome and lessons

Fraud API at 3 s, 20 + 20 req/s	Pool 10	Pool 30	Fixed, pool 10
Payments succeeded	90 / 400	257 / 400	400 / 400
Balance reads succeeded	76 / 400	256 / 400	400 / 400
Balance read p50	5,010 ms	4,952 ms	7.9 ms
Requests waiting for a connection (peak)	202	200	0
Connections in use (peak)	10 / 10	30 / 30	1 / 10

Conclusionthe outage was not caused by the database, the pool size or the load – it was caused by a transaction boundary drawn one line too wide. Every payment parked a database connection while waiting for a partner API, so a 3-second slowdown elsewhere exhausted the pool and took unrelated endpoints down with it. Tripling the pool only reduced the damage. Moving the remote call out of the transaction restored 100% of traffic on the original 10 connections, with the partner still slow.

Takeaways:

Never hold a connection across a remote call. In Spring, @Transactional scope is connection scope – keep it to the SQL.
Pool exhaustion is a symptom, not a cause. Ask what holds connections, not how many you have.
idle in transaction + an idle database = the application is waiting on something else while holding a connection.
Keep leak detection on in production, with the threshold above your longest legitimate transaction – it names the method for you.
Alert on threads waiting for a connection (HikariCP’s pending metric), not only on errors – it rises seconds before the first failure.

Trade-offs taken into account:

The payment is no longer one atomic unit. The fraud decision is made before the transaction; if the write then fails, the check is repeated or compensated. This needs an idempotency key on the payment.
Payments are still 3 s slow. The fix contains the damage; protecting the payment path itself needs a timeout and a circuit breaker on the fraud call (see case 06).
Leak detection is noisy when set below normal transaction times – tune it, do not switch it off.
Measured locally with a simulated partner and a shortened connection timeout (5 s instead of 30 s); with the default, every waiting request would hang six times longer.

Impact: 166 → 800 of 800 Requests OK / Same 10 Connections Measured: 20 + 20 req/s · Partner at 3 s · PostgreSQL 17

SQL Tuning

Data Cleansing

N+1 Queries

Kafka Rebalance Storm

RabbitMQ Poison Loop

Circuit Breaker

AppSec Gate

GC Pause Tail

Connection Pool Exhaustion