Can this pipeline replay without changing the count?
Eight deliveries contain six real impressions. One late retry can outlive the fast job.
The question
Lumen publishes ad-impression counts by the hour an impression occurred. An AI-generated job removes duplicate deliveries in local memory, then appends its output. A separate recovery job processes late records. All data below are authored practice data; this is not a Netflix event log.
| eventId | viewerId | Occurred UTC | Arrived UTC | |
|---|---|---|---|---|
| d1 | v1 | u1 | 09:01 | 09:02 |
| d2 | v2 | u1 | 09:04 | 09:05 |
| d3 | v1 | u1 | 09:01 | 09:06 |
| d4 | v3 | u2 | 09:08 | 09:10 |
| d5 | v4 | u3 | 09:59 | 10:00 |
| d6 | v6 | u4 | 10:02 | 10:03 |
| d7 | v5 | u5 | 09:12 | 10:45 |
| d8 | v3 | u2 | 09:08 | 10:50 |
All times are on one authored day. v1 is retried at d3; v3 is retried at d8. v1 and v2 are different valid impressions from one viewer.
Count one impression per eventId in its occurredAt hour. The fast route accepts delivery age ≤60 minutes. Older deliveries enter recovery. Fast and recovery have separate local seen-key sets; a full replay restarts both local sets but keeps destination rows.
“Choose the identity, late path and destination behavior. Give the hourly records after recovery and after replay.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Count impressions, not viewers
What would you do here? Open the move you’d make.
Deduplicate by viewerId
Tempting, but it misses something. This collapses v1 and v2 even though they are separate impressions.
Deduplicate by eventId
The strong move. The consumer contract counts impressions; viewer identity has a different grain.
Count every delivery
Tempting, but it misses something. d3 and d8 repeat existing events.
See how a strong candidate says it
“I use eventId as the impression identity. v1 and v2 share viewer u1, but both belong in the result. A viewer-level duplicate key would lose a real impression.”
Why it works: The consumer contract counts impressions; viewer identity has a different grain.
Step 2: Keep late first deliveries
What would you do here? Open the move you’d make.
Retain late deliveries for recovery
The strong move. Late arrival does not mean the business event did not happen.
Discard everything older than 60 minutes
Tempting, but it misses something. This loses the first delivery of v5.
Move the late records into their arrival hour
Tempting, but it misses something. The contract groups impressions by occurrence, not delivery.
See how a strong candidate says it
“d7 contains v5 for the first time. It arrives 93 minutes after occurrence, so it belongs in recovery, not the fast route. I also retain d8; its retry status must be resolved against the destination.”
Why it works: Late arrival does not mean the business event did not happen.
Step 3: Put idempotency at the retained output
What would you do here? Open the move you’d make.
Append because the fast job already deduplicates
Tempting, but it misses something. Recovery has independent state and emits v3 again.
Compare totals only
Tempting, but it misses something. A lost event and an extra duplicate can cancel in the total.
Merge destination rows by eventId
The strong move. Separate jobs cannot rely on one local duplicate set.
See how a strong candidate says it
“I merge by eventId in the destination. The fast job emits five IDs; recovery emits v5 and v3. An append-only destination stores seven rows because it already holds v3. A merge stores the six distinct IDs.”
Why it works: Separate jobs cannot rely on one local duplicate set.
Step 4: Replay and compare the records
What would you do here? Open the move you’d make.
Stop after the first total equals six
Tempting, but it misses something. The wrong viewer key plus append also totals six but loses v2 and doubles v3.
Compare IDs and hourly rows after a second run
The strong move. A safe replay preserves record identity and hourly allocation, not just a plausible total.
Call it end-to-end exactly-once delivery
Tempting, but it misses something. The exercise does not establish source, connector or production recovery guarantees.
See how a strong candidate says it
“The retained result is v1, v2, v3, v4 and v5 in the 09:00 hour, plus v6 at 10:00. Replaying the packet leaves those six rows unchanged under merge. Append grows from seven to fourteen. This proves behavior only for the shown packet.”
Why it works: A safe replay preserves record identity and hourly allocation, not just a plausible total.
The answer, in short
- Decision
- Merge by eventId, retain recovery and compare the replayed records.
- Because
- The fast and recovery paths can emit the same event; the destination must keep its identity across both runs.
- What would change it
- A different documented event identity, destination retention or consumer aggregation would change the design.
Common traps
- Using viewerId as an impression key.
- Assuming a local duplicate set covers a separate recovery job.
Check your answer
- Did you retain both v1 and v2?
- Did you recover v5 into 09:00?
- Did you remove the second v3 at the destination?
- Did replay leave five 09:00 IDs and one 10:00 ID?
Last reviewed Nov 10, 2026