Enterprise Data Platform · AWS
Jarvis — real-time product defect intelligence at scale
A cloud-native, event-driven pipeline that ingests, deduplicates, and refines millions of product defect signals a day — replacing brittle batch jobs with a sub-second streaming design and becoming the authoritative source of truth for the whole organization.
01The challenge
Product defect signals — missing attributes, item-level data quality issues — arrived from many upstream systems in high-throughput bursts. Brittle batch workflows meant downstream consumers acted on stale, duplicated, and conflicting data, and there was no single authoritative record per product entity.
02The strategy
Go fully event-driven and serverless. Kinesis Data Streams became the ingestion backbone to absorb bursts with sub-second latency; Lambda handled stream processing and refinement; and DynamoDB — with composite keys and conditional writes — guaranteed exactly-once, idempotent processing across retries and shard replays. Enriched records fan out over a second Kinesis stream, preserving a loosely-coupled pub-sub contract.
03The execution
Built the pipeline as discrete stages: ingest → normalize & enrich (Lambda) → dedup & state (DynamoDB) → reconcile & resolve schema drift (Lambda) → fan-out (Kinesis). Instrumented end to end with CloudWatch metrics, alarms, and structured logging, and tuned for horizontal scale, fault tolerance, and cost through auto-scaling Lambda concurrency and Kinesis shard tuning.
04The results
- 3M+
- Defect events ingested at peak / hour
- 5
- Pipeline stages, ingest to fan-out
- 100%
- Exactly-once processing guarantee
See the system in motion
An interactive walkthrough of the architecture and data flow — hover, click, and drive the simulation.
The pipeline, live
Raw defect events flow left to right — ingested, deduplicated, refined, and fanned out. Hover any node to inspect it and freeze its inbound traffic.
Upstream producer
Catalog Feeds
Bulk vendor and marketplace catalog feeds stream in item records with wildly varying completeness.
Amazon Kinesis Data Streams
Kinesis Ingest
The ingestion backbone. Absorbs high-throughput bursts with sub-second latency and buffers them across shards for parallel consumption.
AWS Lambda
Processing λ
Consumes Kinesis records, applies business rules, normalizes schemas, and enriches raw defect payloads into canonical records. Concurrency scales with shard load.
Amazon DynamoDB
DynamoDB
Composite keys + conditional writes guarantee idempotent, exactly-once processing across retries and shard replays. Duplicates are rejected here.
AWS Lambda
Refinement λ
Reconciles conflicting signals, resolves schema drift, and collapses many events into one authoritative record per product entity. Rejects malformed data.
Amazon Kinesis Data Streams
Kinesis Output
Publishes enriched, high-fidelity records to a downstream stream, preserving a loosely-coupled pub-sub contract across the enterprise.
Downstream consumer
Analytics
Data-quality dashboards read the canonical defect feed.
Drive the simulation. Push traffic, inject a bad record, or freeze the stream.
Events / sec
890
Duplicates blocked
0
P99 latency
47 ms
Downstream fan-out
3
Have a similar challenge?
Tell us where it hurts — we'll show you what's possible and what it costs.