High-Performance Data Access Layer
A consensus-coordinated caching layer that takes cache-hit reads from a measured 45 ms P99 to under 1 ms (≥ 80 % of reads, modeled and benchmark-gated) — with no big-bang migration and no lost writes
At a glance
- Company: growing read-heavy B2B platform (~90 % reads on the customer-facing critical path).
- Problem: single-key reads at 45 ms P99 (8 ms median) straight against sharded PostgreSQL, a ~100K ops/sec connection-pool ceiling, and GDPR / SOC 2 Type II audits approaching with no access audit trail.
- What we built: HPDAL — a three-node Go cluster with an in-process L1 cache and a Redis L2, Raft-ordered cross-node invalidation, transparent shard routing, mTLS on every path and a transactional audit trail.
- Headline numbers: cache-hit read P99 < 1 ms for ≥ 80 % of reads (modeled, benchmark-gated) from a measured 45 ms; expected read latency ≈ 4 ms (traffic-weighted model); writes at parity (≤ 55 ms P99, measured baseline); RPO (recovery point objective) = 0 by design.
- Timeline & team: 16-week production MVP build (a feature-complete v1 under benchmark gating) · 4–6 senior engineers · 9 Architecture Decision Records · 14-diagram blueprint.
- Status: delivered and handed off; the company operates it against the runbooks. Benchmark publication is scheduled — until then every improvement figure stays labelled modeled.
Figure 1 — Management View
HPDAL sits between client applications and PostgreSQL as a transparent caching and routing layer. Cache-hit reads drop from a measured 45 ms P99 to under 1 ms, expected read latency lands at ≈ 4 ms (a traffic-weighted model, not a P99), writes hold parity, and infrastructure cost is offset by read-replica reduction (~19% net). All of it conditional on one measurable assumption, stated on the diagram itself: ≥ 80% L1 cache hit ratio on a 90%-read workload.
A Different Kind of Case Study
The Challenge
A growing read-heavy B2B platform had every application talking directly to PostgreSQL. It worked — until growth made every symptom worse at once.
Latency lived at the tail. Indexed single-key reads had a median of 8 ms, but P99 sat at 45 ms — and customer-facing endpoints are experienced at the tail, not the median. Batch reads were worse: 140 ms P99 for 50 keys.
Throughput had a hard ceiling. The connection pool topped out around 100K aggregate reads/sec across 20 pooled connections. Beyond that: pool exhaustion, queuing, and a longer tail. Every new feature added read load to the same instances that had to stay healthy for writes.
Scaling the database was the expensive answer to the wrong question. ~90% of the workload was reads over a hot working set with a Pareto distribution — a small set of keys dominating traffic. Bigger instances and more replicas meant paying database prices to serve the same few gigabytes of hot data over and over. And with GDPR and SOC 2 Type II audits approaching, direct database access had no answer to who accessed what, when, from where — "we'll add logging later" stopped being an option.
The obvious shortcuts each failed a requirement. More read replicas — linear cost growth, replication lag with no stated bound, zero help with the audit trail. A Redis cache bolted into each application — every service invents its own invalidation (badly); N applications produce N incoherent caches; and a cache read still pays a 1–5 ms network hop, so the 45 ms tail becomes a 5 ms tail at best. The hard problem isn't caching — it's invalidation with guarantees. An off-the-shelf caching proxy — no shard-aware routing, no coherent cross-node invalidation, no write-through semantics, no audit.
What the problem actually required was distributed-systems engineering: an in-process cache with cross-node coherence, coordinated by consensus, in front of sharded PostgreSQL — delivered as infrastructure, not as a per-team convention, and built to production standards from the first commit.
The Solution
We designed HPDAL — a small cluster of identical Go nodes that sits transparently between applications and their sharded PostgreSQL databases. Two-tier caching: a microsecond in-process L1 backed by a shared Redis Cluster L2, engineered so ≥ 95% of reads never touch the database. Consensus-coordinated coherence: Raft orders cache-invalidation events across nodes, so every node converges on the same view within a bounded window — no TTL-and-pray. Transparent shard routing via jump consistent hashing; clients never know sharding exists. CQRS (Command Query Responsibility Segregation) discipline: the read path (cache-first, latency-critical) and write path (database-first, durability-critical) are separate code paths with separate optimization strategies. And compliance as architecture: mTLS on every connection and a transactional audit trail, designed in from Step 1 — not bolted on.
Two more principles govern the details. Put the cache in the same process as the request: the <1 ms target rules out any network hop — even same-host Redis costs ~0.3 ms. The 10× lever is physics, not cleverness. And every guarantee is bounded and monitored: cross-node staleness <500 ms, cache-miss latency ≤ 50 ms, recovery <30 s. If a number can't be measured and alerted on, it isn't a guarantee — it's a hope.
Go 1.22+ at build time
One ~15 MB binary per node
hashicorp/raft + bbolt
Consensus for cache coherence
Redis 7 Cluster
L2 cache · 3P + 3R
PostgreSQL 15 + pgx v5
Sharded source of truth
gRPC + Protobuf
Zero-copy across every tier
Kubernetes StatefulSets
Stable identity · 3 AZs
cert-manager
mTLS · 90-day auto-rotation
Prometheus + Grafana
24 metrics · 5 dashboards
Patroni
PostgreSQL HA auto-failover
One platform, six jobs — each one an infrastructure guarantee the applications no longer have to solve for themselves.
Two-Tier Caching
In-process L1 (~0.05 ms) + Redis Cluster L2 — ≥ 95% of reads never touch the database
Consensus-Coordinated Coherence
Raft orders invalidations across nodes — bounded staleness, no TTL-and-pray
Transparent Shard Routing
Jump consistent hashing — clients never know sharding exists
CQRS Discipline
Cache-first reads, DB-first writes — separate paths, separate optimization
Compliance as Architecture
mTLS everywhere + transactional audit trail — GDPR-ready · SOC 2-ready controls, designed in from Step 1
Zero-Downtime Operations
Deploys, cert rotation, rebalancing, migration — documented mechanisms with rollback paths
Boundaries Before Architecture
The fastest way to fail at middleware is to misdefine what it owns. Before any component was designed, we drew the boundary explicitly: HPDAL owns caching, coherence, shard routing, service-to-service security, and audit. It does not own data schemas, end-user authentication, or business logic — absorbing those would have turned a focused platform into an unbounded one. Equally deliberate: we measured before we promised. The 45 ms baseline comes from production instrumentation under stated conditions — a "10×" target is meaningless without a measured denominator. The next three diagrams follow the C4 model, a four-level notation for software architecture (context, containers, components, code).
Figure 2 — System Context (C4 Level 1)
HPDAL is middleware with a disciplined boundary and six external dependencies — two critical (PostgreSQL, Redis), four operational (cert-manager, Kubernetes API, observability, log aggregation). Boundaries this explicit are what keep a platform component from metastasizing into a second application tier.
Figure 3 — Platform Overview (C4 Level 2)
Three identical Go nodes, one per availability zone, deployed as a Kubernetes StatefulSet. Every node runs the same binary — API, cache, consensus, routing — so the cluster has no specialized roles to fail. PostgreSQL is the source of truth; caches are derived state (RPO = 0). Losing an entire AZ leaves a working majority (2 of 3) and full service. Four ports expose the system: :50051 gRPC data, :8443 admin, :9090 Raft, :8081 health/metrics.
Why one embedded binary instead of microservices?
The <1 ms read target requires the L1 cache in-process — splitting cache from server adds a network hop and erases the entire advantage. Raft benefits from co-location too: invalidation events apply to local memory without serialization. And a 4–6 engineer team should operate one deployable, not five. We rejected microservices (the fatal network hop), a Redis-only design (1–5 ms floor), sidecar meshes (~0.5 ms per hop), and serverless (cold starts; no persistent in-process state) — each in writing, with the arithmetic. CQRS became a code-organization pattern inside one binary, not an excuse for more deployables.
Figure 4 — Node Internals (C4 Level 3)
A ~15 MB Go binary in three layers: Transport (gRPC :50051 with the mTLS → authorization → audit interceptor chain), Service (ReadService cache-first / WriteService DB-first), and Cache + Data (tiered cache facade, Raft node, shard router, audit logger) — wrapped by a cross-cutting resilience band of circuit breakers, timeouts, and load shedding. This is the "where do I add code" map a new engineer gets on day one. The Bypass Router is the escape hatch that makes incremental adoption possible: any request can route straight to PostgreSQL from day one of a migration.
The Read-Path Mechanism
Figure 5 — Read Path Flow
A Get(key) flows through three tiers: L1 in-process (~0.05 ms — the sub-millisecond path, where ≥ 80% of reads stop), L2 Redis (~1 ms), and on a full miss, PostgreSQL behind a singleflight guard (one in-flight database query per key) that collapses N concurrent misses for the same key into one database query. The diagram shows the cold-start and warming rows too — a system that's only fast when warm should say so.
The arithmetic is worth showing, because it's the whole business case. Expected read latency (traffic-weighted mean of per-tier P99 budgets — a conservative mean, not a P99):
The 1 / 5 / 50 ms inputs are the MVP non-functional requirement (NFR) budgets for each tier's P99. The ratios compare the modeled mean to the 45 ms P99 baseline; they are not P99-to-P99. The P99 of the blended stream is governed by miss latency until the hit ratio exceeds 99 %, which is why the benchmark gates below are stated per tier.
Why is the L1 hit ratio the whole ballgame?
The model is arithmetic, not marketing — and it rests on one load-bearing assumption: ≥ 80% of reads hit L1. That's why the assumption is justified (the workload's Pareto skew was verified against production access logs; the hot set fits the 1–4 GB per-node L1 budget), monitored (a dedicated Prometheus metric with an alert at the threshold), and benchmark-gated before anyone quotes it as an outcome. And the 4.05 ms is a traffic-weighted mean, not a P99: the blended stream's P99 is set by miss latency until the hit ratio exceeds 99 %, so the gated targets are stated per tier — hit-path P99 under 1 ms, miss P99 within 50 ms.
Figure 6 — Write Path Flow
Writes are DB-first: PostgreSQL commit with fsync — including the audit row in the same transaction — then synchronous invalidation of the local L1 and the shared L2, and only then the client ack at ≤ 55 ms P99. Same-node read-after-write is immediate. Cross-node L1 copies converge via Raft in <500 ms (typically ~10 ms). L2 is always fresh cluster-wide, because the writer cleared it before acknowledging. One Snowflake-style version ID, generated at the database upsert, threads through every tier to reject stale data at each of them.
Why doesn't the cache make writes faster?
It can't — and we say so. Write-behind caching (acknowledge fast, persist later) would have flattered our latency numbers and risked losing customer data on a node failure; we rejected it in one line: RPO = 0 is non-negotiable. The write target is parity (≤ 55 ms P99), the sub-millisecond claim applies to cache-hit reads on the path that carries 90% of traffic, and on the expected-latency model, blended across the full 90/10 workload, the improvement is ~5× (modeled mean against the P99 baseline). We'd rather you hear that arithmetic from us than discover it in production.
Why acknowledge before other nodes are invalidated?
Synchronous cross-node consensus would add 2–10 ms to every write to eliminate a staleness window that's already bounded at 500 ms and typically closes in ~10 ms. Correctness where it matters — durability, same-node read-after-write, L2 freshness, all before the ack. Pragmatism where it doesn't.
Figure 7 — Consensus & Cache Coherence
Raft does exactly two jobs: ordering L1 invalidation events and electing a leader. Not data replication (PostgreSQL does that), not L2 management (the writer does, synchronously), not request routing. The state machine is one operation — Apply() → L1.Tombstone(key, version) — which is exactly why it's trustworthy. And the stale-populate race is defeated at both tiers by version comparison: a slow read carrying v5 can never overwrite an invalidation stamped v6. Equal counts as stale.
Why Raft, and why not build it?
Pub/sub alternatives (Redis Pub/Sub, gossip) offer at-most-once delivery with no ordering — a key written twice in quick succession can leave caches serving stale data until a TTL rescues them, which is exactly the failure class this system exists to eliminate. Raft provides ordered, quorum-committed delivery with leader election built in. And we adopted hashicorp/raft — battle-tested in Consul, Nomad, and Vault — rather than building: ~2 weeks of integration versus 4–6 weeks of the project's highest-risk custom code. Boring consensus is good consensus.
Engineering the Data Layer
Figure 8 — Cache Architecture
L1 alone fails: the working set exceeds one node's memory, restarts start cold, nodes can't share. L2 alone fails: a ≥ 1 ms network hop on every read can never reach <1 ms P99. Together: a 16-segment, lock-striped in-process L1 with lock-free recency tracking via atomic timestamps, over a 6-node Redis Cluster whose hash tags align cache slots with database shards for pipelined batch reads. TTLs on both tiers (5 min / 30 min, ±10% jitter) are defense-in-depth, not the coherence mechanism. Combined hit-ratio target: ≥ 95% MVP, ≥ 99% at scale — the performance model runs a conservative 98%.
Why no Redis persistence?
The cache is derived state, rebuilt from PostgreSQL on any failure. Persisting it buys nothing but recovery-time risk and cost. Deciding what not to protect is as much a reliability decision as what to protect.
Figure 9 — Data Architecture & Sharding
One jump-consistent-hash result routes the key at every tier — PostgreSQL shard, Redis hash tag, L1 namespace — so routing mismatches are prevented by construction. Prefix-based shard keys co-locate related data (user:12345:* lands together), which is exactly what makes per-shard transactional writes possible. Cache-miss reads go to the shard primary, not a replica: read-after-write consistency is worth more than replica offload.
Why jump consistent hash?
Modulo hashing redistributes nearly every key when the shard count changes — making zero-downtime rebalancing practically impossible. Range-based sharding creates hot spots under sequential keys. Jump hash is O(1), deterministic, needs no ring metadata, and moves ~1/(N+1) of keys per added shard (about a quarter on the 3 → 4 step) — the migration procedure below is built around that.
The contract we put in bold
Cross-shard batch writes are not globally atomic — they're atomic per shard. Rather than hide that behind a two-phase-commit layer and a latency cliff, it's a first-class API contract, printed twice in the diagram, with the mitigation attached: co-locate keys that must commit together. Distributed-systems honesty up front is cheaper than incident reviews later.
Figure 10 — Migration & Rebalancing
Two procedures, one shared principle: every phase is independently reversible with zero data loss. Client onboarding runs ~5 weeks through bypass mode — deploy idle, route 10% → 50% → 100% of reads as a pure pass-through while proving <1 ms overhead against direct PostgreSQL, then enable caching and DAL-routed writes simultaneously (enabling them separately would let uncached writes poison the cache for up to 30 minutes). Shard rebalancing (3 → 4) runs five phases — dual-read + dual-write, throttled background copy, seconds-long cutover, verified cleanup — with the shard-map revert as the safety valve. Direct database access stays one header away throughout: the platform never holds the data hostage.
Built for Production
Designing the happy path is the easy 20%. The remaining 80% — security and audit, bounded failure, deployment, operations — is what separates architecture from a diagram.
Figure 11 — Security & Compliance
mTLS on every path — client-to-node, node-to-node Raft, database, Redis — with 90-day certificates auto-rotated by cert-manager with zero downtime. Authorization is default-deny with four access levels tied to certificate identity, hot-reloaded from configuration, with unknown identities denied and logged at 100% on the non-droppable path. GDPR Art. 17/30/32 and SOC 2 CC6.1 · CC6.3 · CC6.7 · CC7.2 map to specific mechanisms in the design — including the Art. 17(3)(b) audit-retention exemption auditors always ask about.
Why mTLS without a service mesh?
Istio would have given us mTLS "for free" — at ~0.5 ms of sidecar latency per hop, 10× the entire ~0.05 ms L1 lookup. Go's native TLS is negligible on established, pooled connections. When your differentiator is measured in microseconds, you do not put a proxy in the hot path.
Why a hybrid audit pipeline?
Synchronous audit for everything adds 1–5 ms to reads — destroying the latency budget. Async for everything can drop write events — a compliance failure. So compliance-critical events (100% of writes, auth decisions, admin ops) ride in the same database transaction as the data they describe — they're already paying for an fsync; the audit INSERT is nearly free and rollback-safe by construction — while high-volume operational events flow through a sampled async buffer, escalating to 100% for regulated data prefixes.
Figure 12 — Resilience Patterns
Three phases of protection. Entry limits reject overload before it cascades. Per-dependency circuit breakers isolate PostgreSQL shards, Redis, Raft, and log aggregation independently — Redis down means L1 + DB keep serving; Raft down means writes still commit DB-first. And degradation is graceful, with a three-layer staleness safety net: Raft (<500 ms) → L1 TTL (≤ 5 min) → L2 TTL (≤ 30 min). Staleness is always bounded; loss is impossible by design — no write ever depended on a cache.
Figure 13 — Infrastructure & Deployment
Three AZs, one pod per node, a PodDisruptionBudget (PDB) sized to the Raft quorum, Guaranteed QoS with GOMEMLIMIT (the Go runtime's soft memory limit) at 87.5% of pod memory so the garbage collector and noisy neighbors can't eat the latency budget. Every shard's primary and replica live in different AZs — a full zone failure preserves quorum and every shard. Releases run canary-first: 24 h staging soak → single-pod canary with automated gates → sequential rollout that never breaks quorum, ~20 minutes end to end. Eight update types: seven with zero-downtime mechanisms, and one honest "⚠ requires planning" for cluster expansion.
Figure 14 — Observability & Operations
The backup strategy is one sentence: only PostgreSQL needs backups — everything else is derived, resyncable from peers, or in Git. Twenty-four Prometheus metrics, five Grafana dashboards, and a P1–P4 alert ladder with response times from 15 minutes to next-business-day — every alert maps to a runbook. Recovery is rehearsed, not hoped for: quarterly DR drills cover eight failure classes, and the six critical scenarios carry recovery-time objectives (RTOs) from <5 s (leader election) to <15 min (total cluster and volume loss) — and in every one of them, caches and Raft state rebuild from PostgreSQL, the only durable store: HPDAL itself cannot lose a committed write. The Grafana panel drawn in the figure shows illustrative values, not measurements.
The Numbers, Honestly
Baselines below are measured. Improvement figures are modeled, with the arithmetic shown. Nothing gets promoted to a "result" until it clears the published benchmark gates. Most case studies ask you to trust a dashboard screenshot — this one invites you to check the math. Status: the production MVP is delivered and handed off, the company operates it against the runbooks, and the A/B benchmark run is not yet published. When it is, this page will show the measured figures beside the model; until then every improvement figure on this page stays labelled modeled.
Measured baseline — production instrumentation, 500K-record dataset, 50 concurrent clients:
Scroll sideways to see all columns →
| Metric — direct PostgreSQL | P50 | P95 | P99 |
|---|---|---|---|
| Single-key read | 8 ms | 25 ms | 45 ms |
| Batch read (50 keys) | 35 ms | 85 ms | 140 ms |
| Single-key write | 12 ms | 35 ms | 55 ms |
| Sustained reads | ~100K aggregate ops/sec at the 20-connection pool ceiling | ||
Modeled targets — traffic-weighted expected latency from per-tier P99 budgets, benchmark-gated:
Scroll sideways to see all columns →
| Scenario | L1 / L2 / DB hit split | Expected read latency (model) | vs 45 ms P99 baseline* |
|---|---|---|---|
| Steady state — MVP floor | 80 / 15 / 5 | 4.05 ms | 11.1× |
| Steady state — scale tier* | 90 / 8 / 2 | 1.29 ms | 34.9× |
| Cache-hit reads (≥ 80 % of reads) | L1 | P99 < 1 ms | the gated tail target |
| Cache-miss reads | DB | P99 ≤ 50 ms | sets the blended-stream P99 until hit ratio > 99 % |
| Writes | — | ≤ 55 ms P99 | parity, by design |
| Blended 90/10 workload | — | — | ~5× overall (modeled mean vs P99 baseline) |
| Read throughput (capacity) | — | 100K/sec per node · 300K–500K per cluster | 3–5× |
*Expected latency is the traffic-weighted mean of per-tier P99 budgets — a conservative mean, not a P99. The ratios compare that modeled mean to the 45 ms P99 baseline; they are not P99-to-P99. The P99 of the blended stream is governed by miss latency until the hit ratio exceeds 99 %, which is why the gates are stated per tier. The scale tier uses the tighter scale-tier NFR budgets — 0.5 ms L1 · 3 ms L2 · 30 ms miss — at a conservative 98% combined hit ratio against the ≥ 99% scale target: 0.90 × 0.5 + 0.08 × 3 + 0.02 × 30 = 1.29 ms.
The economics — a design input, not an afterthought:
Scroll sideways to see all columns →
| Metric | MVP (1M records) | 6 months (3M) | 1 year (10M) |
|---|---|---|---|
| Design read load (cluster) | 100K/sec | 200K/sec | 500K/sec |
| HPDAL nodes / Redis / PG shards | 3 / 6 / 3 | 5 / 6 / 3 | 7 / 12 / 10 |
| Infrastructure cost (HPDAL tier, excluding the database) | ~$443/mo | ~$650/mo | ~$1,500/mo |
Because the cache absorbs ≥ 95% of reads, PostgreSQL read replicas become largely redundant — retiring them (~$200/month) leaves ~$243/month net against ~$1,300/month of database spend, ~19% overhead at MVP, under the <30% NFR. The scaling path is equally explicit: one Raft group carries the cluster to ~7–9 nodes; beyond that, shard-scoped consensus groups take it to 10–20+ — a pre-planned path, not an improvisation.
How every number gets proven. Three things make the sub-millisecond claim falsifiable rather than aspirational: a measured baseline (the 45 ms denominator is real, with conditions documented); a published model (one load-bearing assumption — the 80% L1 hit ratio — justified by measured access patterns and monitored from day one); and a pass/fail harness — an A/B benchmark running identical 1M+ record workloads against direct PostgreSQL and against HPDAL, with explicit gates: hit-path read P99 < 1 ms and expected read latency ≤ 4.5 ms at target hit ratios, write parity within budget, per-node throughput matching the baseline's entire cluster-wide ceiling (100K ops/sec per node — ~3× at cluster scale), zero data loss under fault injection. Until a number clears that bar in a published run, it stays labeled a model on this page — no exceptions.
How we delivered. HPDAL follows the CoreBackend Product Development Framework — the same five-phase process we run today:
Scroll sideways to see all columns →
| Phase | Deliverable | Status |
|---|---|---|
| 1 · Discovery | Business objectives, technical scope, constraints | Complete |
| 2 · System Architecture | 9 ADRs, 14 diagrams, performance model, 18-item risk register, cost model, runbooks | Complete |
| 3 · Technical Design | API contracts, schemas, Go interfaces, sequence flows | Complete |
| 4 · Implementation | 2-week sprints, production MVP | Complete — 16 weeks |
| 5 · Operations | Canary deployment, monitoring, quarterly DR drill schedule and runbooks delivered | Handed off — company-operated |
What Phase 2 alone produces: nine Architecture Decision Records with rejected alternatives, a 14-diagram suite from executive summary to component level, a measured baseline and falsifiable performance model, a capacity and cost model through 10M records, an 18-item risk register with owners — including the risks we rated against ourselves — and runbooks written before the first incident, not after. The blueprint is a deliverable in its own right: the diagrams are the same diagrams the build team implemented against; the NFRs become the Prometheus alerts; the risk register becomes the DR drill schedule.
Why CoreBackend. Most teams facing this problem choose between two bad options: an off-the-shelf cache that can't guarantee coherence, or a managed database tier that solves throughput but not latency. The right answer — consensus-coordinated, in-process caching with bounded staleness — requires distributed-systems engineering that most agencies won't touch and most product teams can't staff. That's the work we specialize in. We build the hard parts — consensus integration, cache-coherence protocols, zero-downtime migration — and we document them so thoroughly that the team running the system could operate, extend, or even rebuild it without us.
Your database isn't slow. Your architecture is making it look slow.
If your P99 tells a different story than your average, if read replicas have become a line item you dread, or if "who accessed what" is a question you can't yet answer — the first step is the same one this project took: measure the baseline, model the fix, and let the arithmetic make the case.
Engagements can start with a free Architecture Review: A 45-minute video call on one system or problem. Within 3 business days you get a short written note: main risks, first steps and a price range. Free, and yours either way.
Other case studies: Business Integration Hub · Cloud Cost Optimization Platform · All case studies
Related reading: Go Concurrency: From Fundamentals to Production (22-chapter book)