Business Integration Hub
How a professional services firm eliminated 20 hours of weekly manual work and achieved near-real-time data sync across Salesforce, QuickBooks, NetSuite, and a legacy ERP
At a glance
- Client: 150-person B2B professional services firm, Midwest US (identity withheld under NDA).
- Problem: 20+ hours a week of manual CSV exports between Salesforce, QuickBooks, NetSuite and a legacy ERP; $50K+ a year in data errors; no audit trail with a compliance audit approaching.
- What we built: a Go integration hub on AWS (API Gateway, SQS, ECS Fargate, Aurora Serverless v2, EventBridge) with idempotent, retried, fully audited delivery and per-tenant isolation.
- Headline numbers (measured, month 6): manual hours 20 → 0 per week · integration errors 50+ → 2 per month (−96 %) · sync under 2 s · 99.9 % uptime · tooling cost $1,500+/mo → $400/mo (−73 %).
- Timeline & team: 10 weeks in three phases; architect-led CoreBackend senior engineering team, with the client's operations lead embedded for mapping and cutover.
- Status: In production; the client's team operates it with our runbooks.
The Challenge
A growing professional services firm had a data problem that was costing them time, money, and trust.
Their operations team spent 20+ hours every week manually exporting CSVs between Salesforce, QuickBooks, NetSuite, and a legacy ERP system. A single mismatched invoice triggered a $12,000 billing dispute. Cumulative data errors cost them $50,000+ annually. And with a compliance audit approaching, they had no audit trail to show regulators.
They had tried solving it themselves. A no-code automation tool capped out around 1,000 records. Custom scripts lacked retry logic — one API timeout meant lost data. Neither solution offered idempotency or observability. Both failed exactly when reliability mattered most.
They needed a purpose-built integration platform — fast, reliable, and production-grade from day one.
The Solution
We designed and built a cloud-native integration hub that connects all four systems through a single, unified platform. Real-time synchronization, intelligent data transformation, guaranteed delivery with automatic retries, and a complete audit trail for compliance — all running at $400/month of infrastructure, down from $1,500+/month of previous tooling.
We architected the platform with multi-tenant isolation from day one — not because this client needed it immediately, but because we build integration hubs to be reusable. The client is the first tenant; its business units are the tenants. Every API key maps to a tenant context, data is isolated at the database level via row-level security, and rate limits are enforced per tenant tier. This means the same platform can onboard additional business units without re-architecture, turning a project cost into a long-term asset.
The Unified Integration Platform
All four systems connect through the Integration Hub — one platform, one source of truth, no manual intervention.
API Gateway
Request routing & auth
ECS Fargate
Serverless containers
SQS
Message queuing
Aurora Serverless
Auto-scaling database
EventBridge
Event-driven routing
X-Ray
Distributed tracing
Overview
Component details (7)
- S3 Storage — Mappings, Archives, Audit Logs
Stores transformation configs, event archives (7 days hot → Glacier), and CloudTrail audit logs. Versioned for compliance and rollback. - Aurora Serverless v2 — Idempotency + Event State
Tracks every event from pending → success/failed. Rejects duplicates, provides the audit trail. Scales 0.5–4 ACU with load. - API Gateway — Auth, Rate Limits, Validation
Single entry point for all integrations. Validates API keys, enforces per-tenant rate limits (100-10K req/min), checks schemas. Private backend via VPC Link. - SQS Queue — Async Decoupling Layer
Separates fast API response (202 Accepted) from processing. Guarantees delivery with retry logic — no lost messages even during downstream failures. - ECS Worker Pool — Auto-Scaling 2-20 Tasks
Transforms data using S3 mappings, calls external APIs with circuit breaker protection. Scales on queue depth. Fargate Spot for 70% cost savings. - Tenant Applications — Business Units as Tenants
Business applications syncing data across enterprise systems. Each tenant isolated via API key → tenant_id mapping. Multi-tenant architecture supports onboarding additional business units without re-architecture. - External Systems — CRM, ERP, Accounting APIs
Target systems receiving synchronized data (Salesforce, QuickBooks, NetSuite, Legacy ERP). Workers handle auth, rate limits, and failures per connector automatically.
Hover or tap any component for technical details (keyboard: Tab to a component). The same facts are listed under the diagram. Tap any component for details, or open the list below.
Data flows from tenant applications through API Gateway, into SQS for reliable queuing, then workers process and deliver to external systems.
Why a queue instead of synchronous calls?
We chose SQS over synchronous processing inside the API service so that no message is dropped during downstream failures. The extra ~50 ms of latency was worth the reliability gain — a message that is never dropped never needs manual reconciliation.
Technical Deep Dive
For architects and engineering leaders: this section explores the system design, request flow, and resilience patterns that make this platform production-grade.
The architecture follows a fully async, event-driven pattern. Every incoming request is immediately acknowledged and queued — so no request is dropped even when downstream systems are unavailable. Workers process messages independently, with built-in retry logic and circuit breakers to handle failures gracefully.
We chose AWS serverless primitives (API Gateway, SQS, Fargate, Aurora Serverless) to minimize operational overhead while maintaining production-grade reliability. The result: a system sized for thousands of requests per second (capacity projection — the observed peak is 1,200 requests per minute), that scales down to a minimal footprint when idle, and requires no infrastructure management.
Hover or tap any component in the diagrams below to see implementation details (keyboard: Tab to a component). The same facts are listed under each diagram.
Request Flow
Component details (8)
- API Clients — HTTPS / TLS 1.3
Tenant applications syncing data between business systems. Each request authenticated via API key mapped to tenant ID and rate limit tier. - Route 53 — Health-Checked DNS
Routes traffic to API Gateway with automatic failover. Health checks every 30 seconds — unhealthy endpoints removed within 60 seconds. - AWS WAF — OWASP Top 10 Protection
Blocks SQL injection, XSS, and malicious patterns before reaching the API. Reduces attack surface and unnecessary compute costs. - API Gateway REST — 100 / 1K / 10K req/min by tier
Validates API keys, enforces per-tenant rate limits, and checks request schemas. Routes to private ECS via VPC Link — no public exposure. - Application Load Balancer — Private Subnet Only
Distributes traffic across ECS tasks in private subnets. Only accessible via VPC Link — not internet-facing. - SQS Queue — 5 min visibility / 3 retries
Decouples fast API response from async processing. Retry backoff: immediate → 60 s → 300 s (set per attempt via ChangeMessageVisibility). Failed messages route to the DLQ with CloudWatch alerting. - ECS Service — 2-4 Fargate Tasks (auto-scaling)
Injects tenant context, checks idempotency against Aurora, enqueues to SQS, returns 202 Accepted. P99 response under 200 ms. - Aurora Serverless v2 — 0.5–4 ACU Auto-Scaling
Idempotency store — checks if event_id exists (returns the cached result) or inserts a new record before async processing.
Every request passes through Route 53, WAF protection, and API Gateway for authentication, rate limiting, and schema validation — then enters our private VPC via secure VPC Link.
The Request Flow above ends with a 202 Accepted response and a message in SQS — the client gets a fast acknowledgment while the real work happens asynchronously. The diagram below traces what happens next: workers pull messages from the queue, load transformation mappings, call external APIs with circuit breaker protection, and handle every possible failure mode without losing a single record.
Async Processing
Component details (11)
- SQS Queue — Long Polling (20s wait)
Workers poll continuously, reducing empty receives. The visibility timeout (5 min) keeps a message hidden while a worker processes it; duplicates from at-least-once delivery are rejected by the idempotency store. - Dead Letter Queue — After 3 Fails
Captures messages that fail all retries (immediate → 60 s → 300 s backoff). Preserves failed payloads for debugging — no message is dropped even when external systems are down. - CloudWatch Alarms — DLQ Depth > 0 Triggers Alert
Monitors failed message accumulation. Alerts fan out via SNS to PagerDuty (on-call) and Slack immediately — enables rapid investigation before SLA breach. - ECS Worker Pool — 2-20 Fargate Spot Tasks (70% savings)
Auto-scales on queue depth (>100 msgs = scale up, <20 = scale down). Transforms data, applies circuit breaker, calls external APIs. Spot instances safe for async work. - Secrets Manager — Auto-Rotation Enabled
Stores external API credentials (Salesforce, QuickBooks, NetSuite, Legacy ERP). Workers retrieve via IAM role — cached 1 hour to minimize latency and API calls. - S3 Transformation Mappings — Cached 5 Min Per Worker
JSONPath rules defining data transformations (e.g., salesforce-to-netsuite.json). Versioned for rollback — update mappings without redeploying code. - Aurora Serverless v2 — Event Status Tracking
Updates status (pending → success/failed) after processing. Stores result payload for client polling and audit trail. Multi-tenant isolated via tenant_id. - Amazon EventBridge — integration.success Events
Routes completion events to downstream consumers (analytics, webhooks, archive). Decouples workers from consumers — enables event replay for recovery. - NAT Gateway — Multi-AZ Deployment
Enables outbound internet from private subnets. Workers call external APIs through NAT — no inbound exposure, static IP for allowlisting by the connected systems. - Internet Gateway — Egress Path for Workers
Routes NAT traffic to internet for outbound API calls. Workers reach external systems while staying fully private — no public IPs assigned. - External Systems — 30s Timeout / Circuit Breaker
Target APIs receiving transformed data (Salesforce, QuickBooks, NetSuite, Legacy ERP). Circuit breaker opens after 5 failures, half-open after 30s — protects against cascading failures.
Workers pull from SQS, load transformation mappings from S3, call external APIs with circuit breaker protection, and handle failures with automatic retries and Dead Letter Queue routing.
Why async?
External APIs are slow and unreliable — response times range from 2 to 30 seconds. By queuing requests, we return 202 Accepted in under 200 ms while guaranteeing eventual delivery. The client never waits, and no request is dropped.
Why Fargate Spot for workers?
Workers are fault-tolerant by design — if a Spot instance is reclaimed, SQS redelivers the message automatically; the visibility timeout returns interrupted work to the queue. Result: 70 % lower compute cost with no reliability impact. Capacity projection: at 5K TPS this would save ~$8,400/year; the observed peak is 1,200 requests per minute.
Built for Production
Production means handling failures gracefully, securing data end-to-end, and knowing what's happening at all times. This section covers the infrastructure patterns that make the Integration Hub enterprise-ready.
Data security is enforced at every layer. All traffic flows through private subnets with no public internet exposure. Secrets are managed in AWS Secrets Manager with automatic rotation. Multi-tenant isolation ensures one tenant's data never touches another's — enforced at the database level with row-level security.
Resilience patterns protect against cascading failures. Circuit breakers prevent overwhelming struggling external APIs. Stepped backoff between retries (immediate, then 60 s, then 300 s) spreads retry load. Dead Letter Queues capture failed messages for analysis and replay — no message is dropped.
Observability provides complete visibility. X-Ray traces every request across all services. CloudWatch dashboards surface key metrics in real-time. Automated alerts notify on-call engineers before customers notice issues.
Data & Security
Component details (8)
- Amazon EventBridge — Event Routing & Replay
Publishes integration events to downstream consumers (analytics, webhooks, archive). 7-day replay capability for disaster recovery. Decouples compute from consumers. - ECS API Service — Reads Config, Writes State
Validates requests, checks idempotency in Aurora, publishes to SQS, returns 202 Accepted. Reads feature flags from Parameter Store at startup — no direct access to secrets. - ECS Workers — Full Data Layer Access
Reads S3 mappings (cached 5 min), retrieves credentials from Secrets Manager (cached 1 hr), updates Aurora status. All access via IAM roles — no hardcoded credentials. - Aurora Serverless v2 — 0.5-4 ACU Auto-Scaling
Primary store for events table (event_id, tenant_id, status, result). Multi-tenant isolation via tenant_id on every query. Point-in-time recovery enabled (5-min RPO). - S3 Archive — 7 Days Hot → Glacier
EventBridge archives stored here for long-term compliance. Lifecycle: Standard → Intelligent Tiering → Glacier. Object Lock enabled — immutable audit trail. - S3 Mappings — Versioned Configurations
JSONPath transformation rules per connector (salesforce-to-netsuite.json). Workers cache 5 minutes. Version history enables instant rollback without code deployment. - SSM Parameter Store — Feature Flags & App Config
Stores non-sensitive config: feature flags, scaling thresholds, circuit breaker settings. Read at startup — no restart required for changes. Free tier covers most usage. - AWS Secrets Manager — Auto-Rotation (30/60/90 days)
Stores external API credentials (OAuth tokens, API keys) with automatic rotation. Workers retrieve via IAM role, cached 1 hour. Rotation happens without downtime.
Aurora stores events with multi-tenant isolation. S3 handles mappings (cached 5 min) and archives (7 days → Glacier). Secrets Manager auto-rotates credentials.
Why Aurora Serverless over RDS?
We needed SQL flexibility with serverless economics. Aurora Serverless v2 scales between 0.5 and 4 ACU with load (a ~$44/mo floor) — about 65 % below the always-on provisioned instance it replaced — while absorbing month-end spikes automatically.
Resilience Patterns
429 Too Many Requests with a
Retry-After header.
event_id lookup in Aurora. Duplicate requests
return the cached result (200 OK). New
requests store a pending status and enqueue work, so
duplicates are rejected before any work is done.
CLOSED → OPEN after
5 consecutive failures, then HALF-OPEN after
30s to test recovery. Prevents cascading failures with
sub-millisecond lookups and no database calls on the hot
path.
202 Accepted with a
tracking_id in under 200 ms. Work
continues asynchronously via SQS, decoupling request
acceptance from downstream processing even when external
APIs are slow.
Component details (6)
- Rate Limiting — Per-tier throttling
Throttles requests by tenant tier: internal (100/min), partner (1K/min), high-volume (10K/min). Burst allows 2× capacity for short spikes. When limits are exceeded, returns429 Too Many Requestswith aRetry-Afterheader. - Idempotency Control — Duplicate protection
Prevents duplicate processing using anevent_idlookup in Aurora. Duplicate requests return the cached result (200 OK). New requests store a pending status and enqueue work, so duplicates are rejected before any work is done. - Circuit Breaker — Failure isolation
In-memory state machine:CLOSED → OPENafter 5 consecutive failures, thenHALF-OPENafter 30s to test recovery. Prevents cascading failures with sub-millisecond lookups and no database calls on the hot path. - Async Handoff — Decoupled processing
Returns202 Acceptedwith atracking_idin under 200 ms. Work continues asynchronously via SQS, decoupling request acceptance from downstream processing even when external APIs are slow. - Retry Logic — Stepped backoff
Backoff policy: attempt 1 immediate, attempt 2 after 60 s, attempt 3 after 300 s. SQS handles scheduling, delivering 99.9 % eventual success before escalation. - DLQ + Alarm — Failure visibility
Messages that still fail after 3 attempts are routed to a dedicated DLQ. A CloudWatch alarm triggers when DLQ depth > 0 and notifies on-call via SNS → PagerDuty and Slack. Supports manual replay of failed events.
Three phases of protection: Rate limiting and idempotency guard entry. Circuit breakers and async handoff isolate failures. Retry logic and DLQs ensure nothing is lost.
Why in-memory circuit breakers?
Redis adds latency and operational cost. In-memory circuit breakers execute in under 1 ms. Since workers are stateless and ephemeral, we persist state to Aurora asynchronously for monitoring dashboards — not for the hot path.
Infrastructure & Observability
Component details (12)
- AWS X-Ray — Distributed Tracing
End-to-end request tracing across API Gateway → ECS → SQS → Workers → External APIs. Automatically captures latency, errors, and throttling. Service map shows dependencies. - CloudWatch — Metrics, Logs & Alarms
Unified dashboards for P99 latency, error rates, queue depth. Alarms fan out via SNS to PagerDuty and Slack when thresholds breach. Log Insights for ad-hoc debugging across all services. - Amazon SQS — Standard Queue + DLQ
5-minute visibility timeout, 3 retries before DLQ. CloudWatch alarm on DLQ depth > 0 triggers immediate notification. 14-day retention for investigation. - Dead Letter Queue — Failed Message Isolation
Messages failing 3 attempts route here. DLQ depth > 0 triggers alarm immediately. Retained 14 days for investigation and selective re-queuing. - ECS Workers — Fargate Spot (70% Savings)
Async processing in private subnets. Auto-scales 2-20 on queue depth. Spot instances safe here — SQS redelivers if task interrupted. - ECS API Service — Fargate On-Demand
Synchronous requests in private subnets. Auto-scales 2-4 based on ALB request count (target: 1K req/task). On-Demand for consistent P99 < 200 ms latency. - Amazon ECS + ECR — CI/CD Pipeline Target
GitHub Actions builds → ECR → ECS deployment. Blue/green with 10% canary traffic for 10 minutes. Auto-rollback if error rate exceeds 5%. - Amazon EventBridge — Event Bus & Archive
Routes integration.success events to analytics, webhooks, and archive. 7-day replay for recovery. Custom patterns filter by tenant_id and event type. - Application Load Balancer — Internal Only, Multi-AZ
Distributes traffic across API tasks. Health checks every 15 seconds (2 failures = unhealthy). TLS 1.3 termination. Only accessible via VPC Link. - NAT Gateway — Multi-AZ High Availability
Enables outbound internet from private subnets. Workers reach external APIs through NAT. Static Elastic IP for firewall allowlisting by the connected systems. - Internet Gateway — VPC Internet Access
Routes NAT Gateway traffic to internet for outbound API calls. No inbound access to private compute resources — egress only. - Aurora Serverless v2 — Multi-AZ, Private Subnet
Primary database with automatic failover. Scales 0.5–4 ACU based on load. RDS Proxy is available for connection pooling and was not needed at current load.
X-Ray provides distributed tracing across every request. CloudWatch dashboards surface metrics, logs, and trigger alerts. We see issues before customers notice.
Results
After six months in production syncing data across Salesforce, QuickBooks, NetSuite, and the legacy ERP, the Integration Hub has eliminated every manual workflow the operations team relied on. The metrics below are a month-6 production snapshot — measured, not projected — against the targets the client's COO defined as success criteria: manual hours to zero, uptime ≥ 99.5 %, and an audit trail the compliance team accepts.
Integration Hub
Production Metrics Dashboard
We went from 20 hours of manual CSV exports every week to zero. The billing disputes stopped, and the compliance team has a complete audit trail. CoreBackend didn't just build an integration — they built infrastructure we'll use for years.— COO, 150-person professional services firm · name withheld at client request
How We Delivered
Ten weeks from discovery to go-live, in three phases. The work was led by a CoreBackend architect with a senior engineering team; the client's operations lead was embedded for field mapping, parallel running and cutover.
Why CoreBackend
Unlike off-the-shelf iPaaS tools, we build to your exact workflow — no per-connector fees, no seat pricing, rate limits you control, and no black-box vendor. Unlike generic contractors, we specialize in exactly this: production-grade backend systems, documented so thoroughly that your team can operate and extend them without us.
Four systems still talking to each other by CSV?
Every engagement starts with a free Architecture Review — a written report on your system: bottlenecks, risks, and a recommended plan. Yours either way.
Other case studies: Cloud Cost Optimization Platform · High-Performance Data Access Layer · All case studies
Related reading: engineering writing on retries, idempotency and messaging