Designing for Disorder: A Pattern for Out-of-Order, Overlapping Data Streams

TL;DR

Write now, merge later.

The Problem: Disorder and Overlap

Picture a warehouse with three smart cameras. Each one is connected to its own computer, running its own AI model, independently watching boxes move across the floor.

Camera 1 covers the left side. Camera 3 covers the right. Camera 2 covers the middle — overlapping with both.

At the end of the day, you ask a simple question: How many boxes passed through?

Camera 1 says 42. Camera 2 says 31. Camera 3 says 28.

You can't just add them up. Box #117 was in both Camera 1 and Camera 2's field of view — counting it twice inflates your numbers. But dropping one camera's report means you lose real data. Every camera is absolutely correct about what it saw. The problem is that no single camera saw everything, and their observations overlap.

Now, let's make it harder. Imagine the warehouse has been running for three months before anyone thought to turn the system on. You have terabytes of historical footage. You spin up massive cloud workers to process it all at once. Suddenly, a hundred thousand events hit your database at the exact same time, completely out of order, landing right on top of live events coming in from today's cameras.

Most pipelines assume data arrives in perfect order and never overlaps. When that assumption breaks, it usually breaks silently, producing numbers that look completely plausible but are entirely wrong.

So, how do we build a system that gets the right number every single time?

The Obvious (and Wrong) Answer

The intuitive fix is to merge the data the exact moment it arrives. When Camera 1's report comes in, the system reads whatever is already in the database, merges the new data into it, and saves the updated total. One master record, always up to date.

This breaks the moment two cameras finish processing at the same millisecond.

If Worker A and Worker B both read the database at the exact same time, they both see the same outdated number. They both do their own math, and they both try to save their answer. Whoever saves last overwrites the other. Camera 1's data quietly vanishes into thin air.

You could fix this by adding a "lock" — forcing the workers to stand in a single-file line and take turns updating the database.

But now your cameras can only report one at a time. Your massive cloud workers can't run in parallel. Your system slows to an absolute crawl as you add more cameras or locations. You've traded a math problem for a massive scaling bottleneck.

There is a better way.

The Paradigm Shift: Write Now, Merge Later

What if each worker just... didn't try to merge? What if it wrote down its own observation and simply walked away?

That is the core of this architectural pattern. Every camera produces an independent record. There is no reading, no merging, and no coordination at the front door. Workers run freely in parallel at maximum speed.

When a user finally asks, "How many boxes were there today?", the system grabs all those independent records and merges them right then and there, with full knowledge of everything that has arrived up to that exact second.

Here is what the entire system looks like from a bird's-eye view:

1. Stateless Injectors: The cameras process their footage and throw their results into a central bucket. They don't talk to each other.
2. The Event Store: This is the bucket. It stores one immutable record per camera, per time window.
3. The Two Query Paths: When we need to look at the data, we take one of two paths depending on what we are trying to do.

If you want to zoom in on a highly detailed, single-entity view — like tracing the exact path of Box #117 over the last five minutes — the system uses Runtime Aggregation. It grabs the raw records, merges them on the fly, and hands you an absolutely perfect, up-to-the-millisecond result.

But what if you open a dashboard looking at six months of daily totals? Merging thousands of records on the fly would freeze your browser. So, for the big picture, the system uses a Scheduled Cache. A background job quietly wakes up every few minutes, does the heavy lifting of merging recent records, and saves the clean answers into a fast database.

The frontend is smart enough to know the difference. It routes bulk historical queries to the lightning-fast cache, and routes surgical detail queries to the live runtime merge.

The Secret Sauce: Compression and Metadata

At this point, you might be wondering: If we defer all the merging until the user asks for it, won't that take forever?

It would, if we were trying to merge raw video footage. But we aren't. We are merging Coverage Metadata.

This is the load-bearing concept of the entire architecture. Each worker takes a massive chunk of raw data (say, 1 minute of 4K video) and compresses it into a tiny, distilled list of facts: "Between 09:14 and 09:15, I saw Box 117 and Box 203. Nothing else."

That 1-kilobyte list of events is the "metadata". It tells the merge step exactly what this camera observed, and over what time window, without ever needing to look at the original video.

Because the data is compressed so beautifully, the backend can merge thousands of these records in milliseconds. It simply unions the lists: Box #117 was seen by Camera 1 and Camera 2? Great, count it once. Box #445 was only seen by Camera 3? Count it once.

In fact, because this payload is so small, you don't even need the backend to do the merging! The server can simply hand the raw, compressed records directly to the user's browser via signed URLs. The frontend Javascript can merge the data instantly on the client side.

This allows the UI to be incredibly smart. It can show the user exactly where the data came from, displaying confidence indicators like "High Accuracy: 3 Cameras Agree" or "Warning: Partial Coverage, Camera 2 Offline."

The Overhead Tax: When NOT to use this

This pattern feels like magic, but it earns its keep exclusively through that compression step. If your workers aren't summarizing complex data into small, easily mergeable metadata, this architecture is a waste of time.

Imagine we aren't tracking boxes, but tracking a user's "Active Screen Time" across devices.

• Their phone reports: Opened app at 10:00, closed at 10:03.
• Their laptop reports: Opened app at 10:01, closed at 10:05.

There is overlap here. Simple math gives us 8 minutes of screen time, but the real answer is 5 minutes. So, we should use our "Write now, merge later" pattern, right?

No. Look at the data. The "coverage metadata" required to fix this overlap is just a [start, end] timestamp. But that is already the raw event!

You aren't compressing a 500MB video into a 1KB summary. You are just taking a tiny event and saving it twice. You're paying the architectural "tax" of a complex query layer without getting any performance benefit that a simple database couldn't handle.

The rule of thumb: If the metadata you need to perform a merge looks exactly like the raw event itself, don't use this pattern.

Handling the Messy Real World (Edge Cases)

Once you understand the core system, the traditional nightmares of data engineering suddenly become remarkably easy to solve.

What about the 3-month historical flood? Remember that backlog of old footage? In a traditional system, injecting 3 months of late data destroys your current metrics. But here, the system doesn't care. The workers just dump the historical records into the bucket. Because merging happens at query time, the late-arriving events just slide perfectly into their historical time-slots without touching today's data.

What if the network stutters and sends the same event twice? We assign a unique ID to every single record the moment it's created. If the database sees that ID again, it just bounces it at the door. (This is called Idempotency).

What if a server crashes and we lose data? We never rely on the cameras to hold our only copy of the truth. Every raw webhook or event that enters our facility is logged in a durable, un-erasable storage bucket (like AWS S3 or Kafka) before the workers even touch it. If we detect a gap, we simply pull the missing data from the backup log.

What if we invent a better AI model for the cameras? This is the ultimate superpower of this architecture. Because we keep that raw, durable log of everything that ever happened, pipeline upgrades are no longer scary database migrations. If we push a new AI model, we simply "rewind" the durable log and replay it through the new workers. The entire event store and cache are regenerated from scratch with the new, smarter logic.

Closing Thoughts

The instinct in almost every software system is to merge early and store late: Receive data, figure out the final answer, save the answer.

This pattern inverts that entirely. Store early — save everything, independently, with zero coordination. Merge late — on demand when precision matters, or on a schedule when speed matters.

The cost is a slightly more complex query layer. The benefit is an ingestion pipeline that scales infinitely, handles overlapping chaos without breaking a sweat, and never silently corrupts its own data. Accuracy under disorder isn't something you achieve by trying harder to do math at the front door. It's something you design for, by ensuring that merging correctly is always an option, no matter when the data arrives.