Skip to Results
1
2
3
4
5
6
7
Case Study · Enterprise Integration

Business Integration Hub

How a professional services firm eliminated 20 hours of weekly manual work and achieved real-time data sync across Salesforce, QuickBooks, NetSuite, and a custom ERP

Client: 150-person professional services firm · Midwest US · B2B consulting
10 Weeks
To Launch
(vs. 6-month industry avg)
73%
Cost Savings
vs. Previous Solution
<2s
Sync Speed
Real-time Data Synchronization
99.9%
Uptime
First 6 Months

The Challenge

A growing professional services firm had a data problem that was costing them time, money, and trust.

Their operations team spent 20+ hours every week manually exporting CSVs between Salesforce, QuickBooks, NetSuite, and a legacy ERP system. A single mismatched invoice triggered a $12,000 billing dispute. Cumulative data errors cost them $50,000+ annually. And with a compliance audit approaching, they had no audit trail to show regulators.

They had tried solving it themselves. Zapier hit rate limits at 1,000 records. Custom scripts lacked retry logic—one API timeout meant lost data. Neither solution offered idempotency or observability. Both failed exactly when reliability mattered most.

They needed a purpose-built integration platform—fast, reliable, and production-grade from day one.

The Solution

We designed and built a cloud-native integration hub that connects all four systems through a single, unified platform. Real-time synchronization, intelligent data transformation, guaranteed delivery with automatic retries, and a complete audit trail for compliance—all running at $400/month.

We architected the platform with multi-tenant isolation from day one—not because this client needed it immediately, but because we build integration hubs to be reusable. Every API key maps to a tenant context, data is isolated at the database level via row-level security, and rate limits are enforced per tenant tier. This means the same platform can onboard additional business units or clients without re-architecture, turning a project cost into a long-term asset.

The Unified Integration Platform

Business Integration Hub – Management View
Key Insight

All enterprise systems connect through the Integration Hub—one platform, one source of truth, zero manual intervention.

Built on Amazon Web Services

API Gateway

Request routing & auth

ECS Fargate

Serverless containers

SQS

Message queuing

Aurora Serverless

Auto-scaling database

EventBridge

Event-driven routing

X-Ray

Distributed tracing

Why CoreBackend

Unlike off-the-shelf iPaaS tools, we build solutions tailored to your exact workflow—no per-connector fees, no rate limits, no vendor lock-in. Unlike generic contractors, we specialize in exactly this: backend systems that need to be bulletproof. This isn't our first integration platform. It won't be our last.

How We Delivered

Phase 1 · Weeks 1–4
Salesforce + QuickBooks
Core CRM and accounting integration with bi-directional sync and field mapping.
Phase 2 · Weeks 5–8
NetSuite + Legacy ERP
Enterprise systems onboarded with custom adapters for legacy data formats. The legacy ERP exposed only SOAP/XML endpoints with no webhook support — we built a polling adapter with differential sync to detect changes without overwhelming the system.
Phase 3 · Weeks 9–10
Production + Handoff
Go-live with monitoring, runbooks, and full knowledge transfer to team.
Zero-downtime migration with parallel running. Full documentation and runbooks included.

Overview

Business Integration Hub Overview Diagram
S3 Storage
Mappings, Archives, Audit Logs
Stores transformation configs, event archives (7 days hot → Glacier), and CloudTrail audit logs. Versioned for compliance and rollback.
Aurora Serverless v2
Idempotency + Event State
Tracks every event from pending → success/failed. Prevents duplicate processing, provides audit trail. Scales to $0 when idle — 65% cost savings vs. provisioned RDS.
API Gateway
Auth, Rate Limits, Validation
Single entry point for all integrations. Validates API keys, enforces per-tenant rate limits (100-10K req/min), checks schemas. Private backend via VPC Link.
SQS Queue
Async Decoupling Layer
Separates fast API response (202 Accepted) from processing. Guarantees delivery with retry logic — no lost messages even during downstream failures.
ECS Worker Pool
Auto-Scaling 2-20 Tasks
Transforms data using S3 mappings, calls external APIs with circuit breaker protection. Scales on queue depth. Fargate Spot for 70% cost savings.
Tenant Applications
Multi-Tenant SaaS Clients
Business applications syncing data across enterprise systems. Each tenant isolated via API key → tenant_id mapping. Multi-tenant architecture supports onboarding additional business units without re-architecture.
External Systems
CRM, ERP, Accounting APIs
Target systems receiving synchronized data (Salesforce, QuickBooks, NetSuite, Legacy ERP). Workers handle auth, rate limits, and failures per connector automatically.

Hover any component for technical details. Tap any component for details.

Key Insight

Data flows from tenant applications through API Gateway, into SQS for reliable queuing, then workers process and deliver to external systems.

Key design decision

Why SQS over direct Lambda?

We chose SQS over direct Lambda invocation to guarantee zero message loss during downstream failures. The extra 50ms latency was worth the reliability gain — no data lost means no manual reconciliation.

Technical Deep Dive

For architects and engineering leaders: this section explores the system design, request flow, and resilience patterns that make this platform production-grade.

The architecture follows a fully async, event-driven pattern. Every incoming request is immediately acknowledged and queued—ensuring zero data loss even when downstream systems are unavailable. Workers process messages independently, with built-in retry logic and circuit breakers to handle failures gracefully.

We chose AWS serverless primitives (API Gateway, SQS, Fargate, Aurora Serverless) to minimize operational overhead while maintaining enterprise-grade reliability. The result: a system that scales from zero to thousands of requests per second, costs nothing when idle, and requires no infrastructure management.

Hover over any component in the diagrams below to see implementation details.

Request Flow

Business Integration Hub Architecture Diagram
API Clients
HTTPS / TLS 1.3
Tenant applications syncing data between business systems. Each request authenticated via API key mapped to tenant ID and rate limit tier.
Route 53
Health-Checked DNS
Routes traffic to API Gateway with automatic failover. Health checks every 30 seconds — unhealthy endpoints removed within 60 seconds.
AWS WAF
OWASP Top 10 Protection
Blocks SQL injection, XSS, and malicious patterns before reaching the API. Reduces attack surface and unnecessary compute costs.
API Gateway REST
100 / 1K / 10K req/min by tier
Validates API keys, enforces per-tenant rate limits, and checks request schemas. Routes to private ECS via VPC Link — no public exposure.
Application Load Balancer
Private Subnet Only
Distributes traffic across ECS tasks in private subnets. Only accessible via VPC Link — not internet-facing.
SQS Queue
5 min visibility / 3 retries
Decouples fast API response from async processing. Retry backoff: 0s → 60s → 300s. Failed messages route to DLQ with CloudWatch alerting.
ECS Service
2-4 Fargate Tasks (auto-scaling)
Injects tenant context, checks idempotency against Aurora, enqueues to SQS, returns 202 Accepted. P99 response under 200ms.
Aurora Serverless v2
Scales to $0 When Idle
Idempotency store — checks if event_id exists (returns cached result) or inserts new record before async processing. 65% cost savings vs. provisioned RDS.
Key Insight

Every request passes through Route 53, WAF protection, and API Gateway for authentication, rate limiting, and schema validation — then enters our private VPC via secure VPC Link.

The Request Flow above ends with a 202 Accepted response and a message in SQS — the client gets a fast acknowledgment while the real work happens asynchronously. The diagram below traces what happens next: workers pull messages from the queue, load transformation mappings, call external APIs with circuit breaker protection, and handle every possible failure mode without losing a single record.

Async Processing

Business Integration Hub Async Processing Diagram
SQS Queue
Long Polling (20s wait)
Workers poll continuously, reducing empty receives. Visibility timeout (5 min) prevents duplicate processing — message stays hidden until worker completes or timeout expires.
Dead Letter Queue
After 3 Fails
Captures messages that fail all retries (0s → 60s → 300s backoff). Preserves failed payloads for debugging — no data loss even when external systems are down.
CloudWatch Alarms
DLQ Depth > 0 Triggers Alert
Monitors failed message accumulation. Alerts ops team via SNS (email/Slack) immediately — enables rapid investigation before SLA breach.
ECS Worker Pool
2-20 Fargate Spot Tasks (70% savings)
Auto-scales on queue depth (>100 msgs = scale up, <20 = scale down). Transforms data, applies circuit breaker, calls external APIs. Spot instances safe for async work.
Secrets Manager
Auto-Rotation Enabled
Stores external API credentials (Salesforce, QuickBooks, NetSuite, Legacy ERP). Workers retrieve via IAM role — cached 1 hour to minimize latency and API calls.
S3 Transformation Mappings
Cached 5 Min Per Worker
JSONPath rules defining data transformations (e.g., salesforce-to-netsuite.json). Versioned for rollback — update mappings without redeploying code.
Aurora Serverless v2
Event Status Tracking
Updates status (pending → success/failed) after processing. Stores result payload for client polling and audit trail. Multi-tenant isolated via tenant_id.
Amazon EventBridge
integration.success Events
Routes completion events to downstream consumers (analytics, webhooks, archive). Decouples workers from consumers — enables event replay for recovery.
NAT Gateway
Multi-AZ Deployment
Enables outbound internet from private subnets. Workers call external APIs through NAT — no inbound exposure, static IP for customer allowlisting.
Internet Gateway
Egress Path for Workers
Routes NAT traffic to internet for outbound API calls. Workers reach external systems while staying fully private — no public IPs assigned.
External Systems
30s Timeout / Circuit Breaker
Target APIs receiving transformed data (Salesforce, QuickBooks, NetSuite, Legacy ERP). Circuit breaker opens after 5 failures, half-open after 30s — protects against cascading failures.
Key Insight

Workers pull from SQS, load transformation mappings from S3, call external APIs with circuit breaker protection, and handle failures with automatic retries and Dead Letter Queue routing.

Key design decision

Why async?

External APIs are slow and unreliable — response times range from 2 to 30 seconds. By queuing requests, we return 202 Accepted in under 200ms while guaranteeing eventual delivery. The client never waits; nothing gets lost.

Key design decision

Why Fargate Spot for workers?

Workers are fault-tolerant by design — if a Spot instance is reclaimed, SQS redelivers the message automatically. The visibility timeout ensures no work is lost. Result: 70% cost reduction on compute, zero reliability impact. At scale (5K TPS), this saves $8,400/year.

250,000+
Messages Processed
Peak: 1,200 req/min during month-end close

Built for Production

Production means handling failures gracefully, securing data end-to-end, and knowing what's happening at all times. This section covers the infrastructure patterns that make the Integration Hub enterprise-ready.

Data security is enforced at every layer. All traffic flows through private subnets with no public internet exposure. Secrets are managed in AWS Secrets Manager with automatic rotation. Multi-tenant isolation ensures one customer's data never touches another's—enforced at the database level with row-level security.

Resilience patterns protect against cascading failures. Circuit breakers prevent overwhelming struggling external APIs. Exponential backoff with jitter spreads retry load. Dead Letter Queues capture failed messages for analysis and replay—nothing is ever lost.

Observability provides complete visibility. X-Ray traces every request across all services. CloudWatch dashboards surface key metrics in real-time. Automated alerts notify on-call engineers before customers notice issues.

Data & Security

Data & Security Architecture Diagram
Amazon EventBridge
Event Routing & Replay
Publishes integration events to downstream consumers (analytics, webhooks, archive). 7-day replay capability for disaster recovery. Decouples compute from consumers.
ECS API Service
Reads Config, Writes State
Validates requests, checks idempotency in Aurora, publishes to SQS, returns 202 Accepted. Reads feature flags from Parameter Store at startup — no direct access to secrets.
ECS Workers
Full Data Layer Access
Reads S3 mappings (cached 5 min), retrieves credentials from Secrets Manager (cached 1 hr), updates Aurora status. All access via IAM roles — no hardcoded credentials.
Aurora Serverless v2
0.5-4 ACU Auto-Scaling
Primary store for events table (event_id, tenant_id, status, result). Multi-tenant isolation via tenant_id on every query. Point-in-time recovery enabled (5-min RPO).
S3 Archive
7 Days Hot → Glacier
EventBridge archives stored here for long-term compliance. Lifecycle: Standard → Intelligent Tiering → Glacier. Object Lock enabled — immutable audit trail.
S3 Mappings
Versioned Configurations
JSONPath transformation rules per connector (salesforce-to-netsuite.json). Workers cache 5 minutes. Version history enables instant rollback without code deployment.
SSM Parameter Store
Feature Flags & App Config
Stores non-sensitive config: feature flags, scaling thresholds, circuit breaker settings. Read at startup — no restart required for changes. Free tier covers most usage.
AWS Secrets Manager
Auto-Rotation (30/60/90 days)
Stores external API credentials (OAuth tokens, API keys) with automatic rotation. Workers retrieve via IAM role, cached 1 hour. Rotation happens without downtime.
Key Insight

Aurora stores events with multi-tenant isolation. S3 handles mappings (cached 5 min) and archives (7 days → Glacier). Secrets Manager auto-rotates credentials.

Key design decision

Why Aurora Serverless over RDS?

We needed SQL flexibility with serverless economics. Aurora Serverless v2 scales to near-zero when idle — cutting database costs by 65% compared to always-on RDS while handling traffic spikes automatically.

Resilience Patterns

Resilience Patterns Diagram
Rate Limiting
Per-tier throttling
Throttles requests by customer tier: Free (100/min), Standard (1K/min), Enterprise (10K/min). Burst allows 2× capacity for short spikes. When limits are exceeded, returns 429 Too Many Requests with a Retry-After header.
Idempotency Control
Duplicate protection
Prevents duplicate processing using an event_id lookup in Aurora. Duplicate requests return the cached result (200 OK). New requests store a pending status and enqueue work, ensuring zero double-processing risk.
Circuit Breaker
Failure isolation
In-memory state machine: CLOSED → OPEN after 5 consecutive failures, then HALF-OPEN after 30s to test recovery. Prevents cascading failures with sub-millisecond lookups and no database calls on the hot path.
Async Handoff
Decoupled processing
Returns 202 Accepted with a tracking_id in under 200 ms. Work continues asynchronously via SQS, decoupling request acceptance from downstream processing even when external APIs are slow.
Retry Logic
Exponential backoff
Exponential backoff policy: Attempt 1 (immediate), Attempt 2 (+60s), Attempt 3 (+300s). SQS handles scheduling, delivering 99.9% eventual success before escalation.
DLQ + Alarm
Failure visibility
Messages that still fail after 3 attempts are routed to a dedicated DLQ. A CloudWatch alarm triggers when DLQ depth > 0 and notifies on-call via PagerDuty. Supports manual replay of failed events.
Key Insight

Three phases of protection: Rate limiting and idempotency guard entry. Circuit breakers and async handoff isolate failures. Retry logic and DLQs ensure nothing is lost.

Key design decision

Why in-memory circuit breakers?

Redis adds latency and operational cost. In-memory circuit breakers execute in under 1ms. Since workers are stateless and ephemeral, we persist state to Aurora asynchronously for monitoring dashboards — not for the hot path.

Infrastructure & Observability

Infrastructure & Observability Diagram
AWS X-Ray
Distributed Tracing
End-to-end request tracing across API Gateway → ECS → SQS → Workers → External APIs. Automatically captures latency, errors, and throttling. Service map shows dependencies.
CloudWatch
Metrics, Logs & Alarms
Unified dashboards for P99 latency, error rates, queue depth. Alarms notify PagerDuty when thresholds breach. Log Insights for ad-hoc debugging across all services.
Amazon SQS
Standard Queue + DLQ
5-minute visibility timeout, 3 retries before DLQ. CloudWatch alarm on DLQ depth > 0 triggers immediate notification. 14-day retention for investigation.
Dead Letter Queue
Failed Message Isolation
Messages failing 3 attempts route here. DLQ depth > 0 triggers alarm immediately. Retained 14 days for investigation and selective re-queuing.
ECS Workers
Fargate Spot (70% Savings)
Async processing in private subnets. Auto-scales 2-20 on queue depth. Spot instances safe here — SQS redelivers if task interrupted.
ECS API Service
Fargate On-Demand
Synchronous requests in private subnets. Auto-scales 2-4 based on ALB request count (target: 1K req/task). On-Demand for consistent P99 < 200ms latency.
Amazon ECS + ECR
CI/CD Pipeline Target
GitHub Actions builds → ECR → ECS deployment. Blue/green with 10% canary traffic for 10 minutes. Auto-rollback if error rate exceeds 5%.
Amazon EventBridge
Event Bus & Archive
Routes integration.success events to analytics, webhooks, and archive. 7-day replay for recovery. Custom patterns filter by tenant_id and event type.
Application Load Balancer
Internal Only, Multi-AZ
Distributes traffic across API tasks. Health checks every 15 seconds (2 failures = unhealthy). TLS 1.3 termination. Only accessible via VPC Link.
NAT Gateway
Multi-AZ High Availability
Enables outbound internet from private subnets. Workers reach external APIs through NAT. Static Elastic IP for customer firewall allowlisting.
Internet Gateway
VPC Internet Access
Routes NAT Gateway traffic to internet for outbound API calls. No inbound access to private compute resources — egress only.
Aurora Serverless v2
Multi-AZ, Private Subnet
Primary database with automatic failover. Scales 0.5–4 ACU based on load. Pauses to $0 when idle. RDS Proxy available for connection pooling.
Key Insight

X-Ray provides distributed tracing across every request. CloudWatch dashboards surface metrics, logs, and trigger alerts. We see issues before customers notice.

After six months in production syncing data across Salesforce, QuickBooks, NetSuite, and the legacy ERP, the Integration Hub has eliminated every manual workflow the operations team relied on. The metrics below are from live production — not projections — and reflect the benchmarks the client's COO originally defined as success criteria.

Integration Hub

Production Metrics Dashboard

LIVE
Annual Savings +73%
$ 0
ROI achieved in 6 weeks
Records Synced Today Active
0
Avg response <2s
System Uptime Exceeded
0%
Target 99.5%
Manual Hours / Week -100%
20 0
Status Fully automated
Integration Errors -96%
0/mo
was 50+
Reduction 96%
We went from 20 hours of manual CSV exports every week to zero. The billing disputes stopped, the compliance team has a complete audit trail, and we recovered the platform cost in six weeks. CoreBackend didn't just build an integration — they built infrastructure we'll use for years.
— COO, 150-Person Professional Services Firm
New connector in 3 days
SOC 2 compliant
Real-time sync