Workflow automationyou can stop watching

Workflow automation that survives contact with reality

Workflow automation is the least glamorous and most valuable thing I do. It is the plumbing: getting a record out of one system, transforming it correctly, and putting it into another, on a schedule or a trigger, without losing anything.

It sounds simple and it is not, because every system involved will eventually misbehave. APIs time out. Rate limits appear without warning. A field that was always populated arrives empty. Someone renames a column in a spreadsheet. The difference between a workflow that runs for years and one that needs a babysitter is entirely in how it handles those moments.

Where manual process actually costs you

The visible cost is the time. Somebody spends forty minutes every Monday exporting and re-importing, which is thirty-five hours a year for one process. That is real but it is the smaller number.

The larger cost is latency and error. Data that moves weekly means decisions made on week-old information. Manual transcription introduces mistakes at a rate people consistently underestimate, and those mistakes are found downstream, expensively, by someone who has to work out which system to believe. And the process is fragile: it lives in one person’s head and stops entirely when they are on leave.

How I build workflows

Idempotency first

Every workflow I build can be safely run twice. This is the single most important property and the one most commonly missing. If a run fails halfway, you must be able to simply run it again without creating duplicates or double-charging anyone.

In practice that means natural keys rather than sequence numbers, upserts rather than inserts, and a record of what has already been processed. It costs a little more up front and it removes an entire category of three-in-the-morning problem.

Retries with backoff, and a limit

Transient failures are normal. A workflow that gives up on the first timeout will fail regularly for no good reason; one that retries forever will hammer a struggling API and turn a small outage into a large one. Exponential backoff with a ceiling and a maximum attempt count is the correct behaviour, and it takes minutes to implement.

A dead-letter queue you can actually see

Some records will fail permanently. Malformed data, a genuinely deleted reference, a business rule violation. Those must go somewhere visible, with the reason attached, where a person can inspect them, fix the source and replay them.

The alternative, which I find constantly, is a catch block that logs and continues. The run reports success. Nobody notices until a customer asks why their order never appeared.

Alerts that reach someone who can act

Alerting into a channel nobody reads is the same as no alerting, with added false confidence. I route alerts to a named person, and I tune them so they fire rarely. An alert that goes off every day teaches everyone to ignore it, which means it will also be ignored on the day it matters.

Reconciliation

For anything financial or anything that must balance, the workflow is only half the job. The other half is a scheduled check that compares both sides and reports the difference. Systems drift. A daily reconciliation catches it in a day instead of at year end.

Platform or custom code

Both, depending. For a straightforward connection between two well-supported systems with modest volume, an automation platform is genuinely the right answer and I will set it up properly rather than talking you into a build.

I move to code when volume makes per-operation pricing painful, when the logic has grown into an unreadable chain of branches, when you need real version control and testing, or when a business-critical process is sitting in a platform account tied to someone’s personal email. That last one is more common than you would like.