Both pipelines agree. Can both publish?
A successful shadow comparison can still duplicate the consumer’s output.
The question
Lumen runs old and new enrichment jobs on the same four authored events. A shadow destination compares their candidate outputs; a billing destination must receive one published record per event. The stable allocation bucket is fixed per eventId across every retry.
| Stable bucket | Old candidate | New candidate | |
|---|---|---|---|
| v1 | 12 | GBP1250 | GBP1250 |
| v2 | 77 | GBP800 | GBP800 |
| v3 | 42 | GBP450 | GBP450 |
| v4 | 64 | GBP900 | GBP900 |
All candidate values agree for this packet. Bucket values are authored allocation inputs, not measured hashes.
Publish every candidate from both pipelines because the shadow rows match. Then switch new traffic on by a fresh random coin flip for each delivery.
“Keep the comparison while moving a bounded share of publishing responsibility to the new job.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Keep shadow and billing separate
What would you do here? Open the move you’d make.
Publish both because their fields agree
Tempting, but it misses something. The billing destination still receives two records per event.
Stop comparing as soon as new publishing begins
Tempting, but it misses something. That removes evidence during the cutover itself.
Keep two candidates and one publishing owner
The strong move. Comparison ownership and publication ownership are separate decisions.
See how a strong candidate says it
“Both jobs may process each event for comparison, but only one job may publish that event to billing. Matching candidate rows do not make two published copies acceptable.”
Why it works: Comparison ownership and publication ownership are separate decisions.
Step 2: Assign by a stable event key
What would you do here? Open the move you’d make.
Use the fixed event bucket for publishing allocation
The strong move. Stable allocation prevents retry deliveries from independently choosing publishers.
Flip a new coin for every delivery
Tempting, but it misses something. A retry can choose a different publishing owner.
Send 50% of all deliveries to each output independently
Tempting, but it misses something. The two selections can overlap or omit an event.
See how a strong candidate says it
“For a 50-point threshold, buckets below50 go to new; the rest go to old. v1 and v3 publish from new, v2 and v4 from old. The same eventId keeps its bucket on retries.”
Why it works: Stable allocation prevents retry deliveries from independently choosing publishers.
Step 3: Verify identities and compare the candidates
What would you do here? Open the move you’d make.
Check only that each job published two rows
Tempting, but it misses something. Two rows per job do not prove complementary event identities.
Check IDs, multiplicity and candidate equality
The strong move. The invariant is one published record for each expected event identity.
Use a matching dashboard total as the only audit
Tempting, but it misses something. Aggregates can hide compensating record-level errors.
See how a strong candidate says it
“I expect four billing event IDs: v1,v2,v3,v4, each once. I also retain candidate comparisons for all four. A count of four alone would not rule out one missing ID and one duplicate.”
Why it works: The invariant is one published record for each expected event identity.
Step 4: Make rollback preserve the same boundary
What would you do here? Open the move you’d make.
Replay both jobs into billing to be safe
Tempting, but it misses something. That republishes events unless the destination enforces identity.
Call the four matching events proof for all live traffic
Tempting, but it misses something. The authored sample is only bounded evidence for this exercise.
Rollback allocation and preserve destination idempotency
The strong move. A reversible traffic dial still needs retained destination identity.
See how a strong candidate says it
“I can return the publishing threshold toward old while keeping shadow comparisons. Already-published retries still need destination eventId protection; changing the threshold does not erase prior output or prove replay safety.”
Why it works: A reversible traffic dial still needs retained destination identity.
The answer, in short
- Decision
- Compare both outputs; allocate one publisher per event.
- Because
- The fixed threshold assigns two events to each job without double publication in this packet.
- What would change it
- A discrepancy in candidate fields, a changed stable-key contract or missing destination idempotency would stop the proposed cutover.
Common traps
- Treating a shadow candidate as another published billing event.
- Making a fresh random allocation for each retry.
Check your answer
- Did both jobs retain comparison output?
- Did v1/v3 go to new and v2/v4 to old?
- Did billing retain each of the four IDs exactly once?
- Did rollback preserve destination identity?
Last reviewed Nov 10, 2026