A pipeline with three side-effecting steps either completes all three or — if any step fails — undoes the ones that succeeded. "Either fully done or fully reverted" is a transaction. The pattern works without a real database transaction by tracking what's been done and inverting it on failure.
completed = [] # log of (step_name, action_to_undo)
def undo(action_log):
for step_name, undo_fn in reversed(action_log):
try:
undo_fn()
except Exception as e:
print(f"undo failed for {step_name}: {e}")
state = {"x": 0, "y": 0, "z": 0}
try:
state["x"] = 1
completed.append(("step1", lambda: state.update({"x": 0})))
state["y"] = 2
completed.append(("step2", lambda: state.update({"y": 0})))
raise RuntimeError("step 3 failed")
state["z"] = 3
completed.append(("step3", lambda: state.update({"z": 0})))
except Exception as e:
print(f"caught: {e}; rolling back")
undo(completed)
print(state)Expected: {'x': 0, 'y': 0, 'z': 0}. The two completed steps got reversed; step 3 never happened.
Why reversed?
Last-in-first-out. If step 2 depends on step 1 having succeeded, undoing step 1 before step 2 could leave the system in a worse state than before step 1. Reverse order matches the dependency reverse.
And what if an undo itself fails?
That's the gnarly bit. Three options:
For today's lesson: option 1. A failed undo is itself a kind of dead-letter; it lands in a list and you fix it later.
Real databases give you transactions: BEGIN; ...; COMMIT or ROLLBACK. If anything fails between BEGIN and COMMIT, the database undoes everything for you.
Cross-tool pipelines have no such mechanism. Sending an email, creating a Sheet row, posting to Calendar — each is committed independently. If step 3 fails, steps 1 and 2 are already done.
The rollback pattern emulates a transaction at the application layer: track what you did, undo on failure.
completed = []
def do_with_undo(name, do_fn, undo_fn):
do_fn()
completed.append((name, undo_fn))
try:
do_with_undo("step1", do_step1, undo_step1)
do_with_undo("step2", do_step2, undo_step2)
do_with_undo("step3", do_step3, undo_step3)
except Exception:
for name, undo_fn in reversed(completed):
try:
undo_fn()
except Exception:
log("warn", "undo_failed", step=name)
raiseFor each side-effecting step, define its inverse:
| Action | Undo |
|---|---|
| Append a row to a Sheet | Mark the row's status as reverted (or delete) |
| Create a Calendar event | Delete the event by event_id |
| Send an email | (no real undo — see below) |
| Charge a card | Issue a refund |
| Set state["k"] = v | Set state["k"] = previous value |
Most actions have a real inverse. Sending an email doesn't — once sent, it's sent. For irreversible actions, see below.
When a step has no real undo:
The rule: irreversible steps go after the verification gates and after the reversible steps. Minimize the window in which an irreversible action can happen alongside an unrecovered reversible step.
A rollback is itself an event you want recorded:
log("info", "pipeline_failed", step=failed_step, error=str(e))
log("info", "rollback_start", completed_steps=len(completed))
for name, undo_fn in reversed(completed):
log("info", "undo", step=name)
try:
undo_fn()
except Exception as e:
log("error", "undo_failed", step=name, error=str(e))
log("info", "rollback_done")The production version of this is the saga pattern: a long-running business transaction broken into compensable steps. Frameworks like Temporal, Camunda, and AWS Step Functions implement sagas natively.
For a script, the explicit completed log + reversed rollback is enough. The shape is the same; the framework does the persistence.
Deliberately out of scope:
Those are framework territory. The application-layer pattern — log what you did, undo in reverse — is what carries through.
A pipeline with three side-effecting steps either completes all three or — if any step fails — undoes the ones that succeeded. "Either fully done or fully reverted" is a transaction. The pattern works without a real database transaction by tracking what's been done and inverting it on failure.
completed = [] # log of (step_name, action_to_undo)
def undo(action_log):
for step_name, undo_fn in reversed(action_log):
try:
undo_fn()
except Exception as e:
print(f"undo failed for {step_name}: {e}")
state = {"x": 0, "y": 0, "z": 0}
try:
state["x"] = 1
completed.append(("step1", lambda: state.update({"x": 0})))
state["y"] = 2
completed.append(("step2", lambda: state.update({"y": 0})))
raise RuntimeError("step 3 failed")
state["z"] = 3
completed.append(("step3", lambda: state.update({"z": 0})))
except Exception as e:
print(f"caught: {e}; rolling back")
undo(completed)
print(state)Expected: {'x': 0, 'y': 0, 'z': 0}. The two completed steps got reversed; step 3 never happened.
Why reversed?
Last-in-first-out. If step 2 depends on step 1 having succeeded, undoing step 1 before step 2 could leave the system in a worse state than before step 1. Reverse order matches the dependency reverse.
And what if an undo itself fails?
That's the gnarly bit. Three options:
For today's lesson: option 1. A failed undo is itself a kind of dead-letter; it lands in a list and you fix it later.
Real databases give you transactions: BEGIN; ...; COMMIT or ROLLBACK. If anything fails between BEGIN and COMMIT, the database undoes everything for you.
Cross-tool pipelines have no such mechanism. Sending an email, creating a Sheet row, posting to Calendar — each is committed independently. If step 3 fails, steps 1 and 2 are already done.
The rollback pattern emulates a transaction at the application layer: track what you did, undo on failure.
completed = []
def do_with_undo(name, do_fn, undo_fn):
do_fn()
completed.append((name, undo_fn))
try:
do_with_undo("step1", do_step1, undo_step1)
do_with_undo("step2", do_step2, undo_step2)
do_with_undo("step3", do_step3, undo_step3)
except Exception:
for name, undo_fn in reversed(completed):
try:
undo_fn()
except Exception:
log("warn", "undo_failed", step=name)
raiseFor each side-effecting step, define its inverse:
| Action | Undo |
|---|---|
| Append a row to a Sheet | Mark the row's status as reverted (or delete) |
| Create a Calendar event | Delete the event by event_id |
| Send an email | (no real undo — see below) |
| Charge a card | Issue a refund |
| Set state["k"] = v | Set state["k"] = previous value |
Most actions have a real inverse. Sending an email doesn't — once sent, it's sent. For irreversible actions, see below.
When a step has no real undo:
The rule: irreversible steps go after the verification gates and after the reversible steps. Minimize the window in which an irreversible action can happen alongside an unrecovered reversible step.
A rollback is itself an event you want recorded:
log("info", "pipeline_failed", step=failed_step, error=str(e))
log("info", "rollback_start", completed_steps=len(completed))
for name, undo_fn in reversed(completed):
log("info", "undo", step=name)
try:
undo_fn()
except Exception as e:
log("error", "undo_failed", step=name, error=str(e))
log("info", "rollback_done")The production version of this is the saga pattern: a long-running business transaction broken into compensable steps. Frameworks like Temporal, Camunda, and AWS Step Functions implement sagas natively.
For a script, the explicit completed log + reversed rollback is enough. The shape is the same; the framework does the persistence.
Deliberately out of scope:
Those are framework territory. The application-layer pattern — log what you did, undo in reverse — is what carries through.
Create a free account to get started. Paid plans unlock all tracks.