You have a checkpoint list from a previous run and a full list of pipeline steps. Some steps finished successfully, some did not. How do you compute the list of steps to rerun?
Build a set of completed step names from the checkpoint, then filter the full step list to only those not in the set?
Exactly. A set for O(1) membership, a comprehension for the filter. The checkpoint is [[name, ok], ...]; we extract only the names whose flag is True:
done = {name for name, ok in checkpoint if ok}
remaining = [step for step in all_steps if step not in done]
Using for name, ok in checkpoint unpacks each pair cleanly. And filtering with if ok means failed steps are NOT in done, so they get rerun?
Yes. Failed steps and never-started steps both belong in the rerun list. Here's the full function:
def resume_from_checkpoint(all_steps: list, checkpoint: list) -> list:
done = {name for name, ok in checkpoint if ok}
remaining = [step for step in all_steps if step not in done]
print(f"Resume plan: {len(done)} done, {len(remaining)} remaining")
return remaining
If a step appears twice in the checkpoint — say, retried once and failed, then retried again and succeeded — what happens?
The set comprehension handles duplicates naturally. If any recorded attempt for a step is True, the name lands in done. If every attempt failed, the name is absent and the step goes into remaining. Last-state-wins only if you write the checkpoint carefully; the set approach is more forgiving.
So a crashed pipeline reloads the checkpoint, computes remaining, and runs only what's left — no wasted work repeating successful steps?
Exactly. Combined with idempotency, this means rerunning is always cheap and always safe. No duplicate writes, no repeat reads. That's the resumable-pipeline pattern.
TL;DR: Build a set of completed step names, then filter the full step list to exclude them.
{name for name, ok in checkpoint if ok} collapses duplicates and speeds lookup[step for step in all_steps if step not in done] preserves original orderremaining| Checkpoint state | Appears in remaining? |
|---|---|
| recorded as True | no — skipped |
| recorded as False | yes — re-attempt |
| never recorded | yes — first attempt |
You have a checkpoint list from a previous run and a full list of pipeline steps. Some steps finished successfully, some did not. How do you compute the list of steps to rerun?
Build a set of completed step names from the checkpoint, then filter the full step list to only those not in the set?
Exactly. A set for O(1) membership, a comprehension for the filter. The checkpoint is [[name, ok], ...]; we extract only the names whose flag is True:
done = {name for name, ok in checkpoint if ok}
remaining = [step for step in all_steps if step not in done]Using for name, ok in checkpoint unpacks each pair cleanly. And filtering with if ok means failed steps are NOT in done, so they get rerun?
Yes. Failed steps and never-started steps both belong in the rerun list. Here's the full function:
def resume_from_checkpoint(all_steps: list, checkpoint: list) -> list:
done = {name for name, ok in checkpoint if ok}
remaining = [step for step in all_steps if step not in done]
print(f"Resume plan: {len(done)} done, {len(remaining)} remaining")
return remainingIf a step appears twice in the checkpoint — say, retried once and failed, then retried again and succeeded — what happens?
The set comprehension handles duplicates naturally. If any recorded attempt for a step is True, the name lands in done. If every attempt failed, the name is absent and the step goes into remaining. Last-state-wins only if you write the checkpoint carefully; the set approach is more forgiving.
So a crashed pipeline reloads the checkpoint, computes remaining, and runs only what's left — no wasted work repeating successful steps?
Exactly. Combined with idempotency, this means rerunning is always cheap and always safe. No duplicate writes, no repeat reads. That's the resumable-pipeline pattern.
TL;DR: Build a set of completed step names, then filter the full step list to exclude them.
{name for name, ok in checkpoint if ok} collapses duplicates and speeds lookup[step for step in all_steps if step not in done] preserves original orderremaining| Checkpoint state | Appears in remaining? |
|---|---|
| recorded as True | no — skipped |
| recorded as False | yes — re-attempt |
| never recorded | yes — first attempt |
Create a free account to get started. Paid plans unlock all tracks.