Every business that has ever connected two pieces of software has a story about it breaking. The sync that silently stopped in March and nobody noticed until June. The nightly job that started duplicating records. The integration that worked perfectly until the supplier changed something.
These aren't bad luck. They're the predictable consequence of a handful of design decisions taken — usually implicitly — at the start. Get those right and an integration runs for years without attention. Get them wrong and you've bought a permanent maintenance liability.
Failure one: it assumes it will always succeed
The naive integration does something once and assumes it worked. Fetch the records, write them, done. No check that the fetch returned everything, no record of what was written, no way to tell later whether a given day ran at all.
Then something transient happens — the API is briefly down, a rate limit is hit, the network drops mid-transfer — and that day's data is simply absent. Nothing errors loudly, because from the code's point of view nothing went wrong; it asked for records and got none.
What working systems do instead: every run is recorded, with what it asked for, what it got, and whether it completed. Gaps become visible because a day with no run looks different from a day with no data. This costs almost nothing to build and is the single biggest determinant of whether you find out about problems.
Failure two: it can't be run twice
Anything that has to run on a schedule will eventually need running again — because it failed, because someone changed historical data, because you're backfilling after an outage.
If running it twice creates duplicates, you now have a system that can only be repaired by hand. And the moment repair requires care, repairs get deferred, and deferred repairs become permanent gaps.
What working systems do instead: every record has a stable identifier from the source system, and writes are upserts rather than inserts. Run it five times and you get the same result as running it once. This is called idempotency and it's the difference between an integration you can operate and one you have to nurse.
"What happens if this runs twice?" and "How would we know if it hadn't run at all?"
Ask any supplier those two questions about any integration they're proposing. The quality of the answers tells you more than anything else in the conversation — including anything about which technologies they're using.
Failure three: it fetches everything, every time
The easy way to sync is to ask for all the records and replace what you had. It works beautifully in testing, where there are 200 records.
At 40,000 records it takes twenty minutes, hits rate limits, occasionally times out halfway, and puts meaningful load on the system you're reading from. Eventually somebody reduces the frequency to nightly to make it survivable, and now your data is up to a day stale by design.
What working systems do instead: ask only for what changed since last time. Most decent APIs support this. It's the question almost nobody asks during procurement and the one that determines whether hourly refreshes are cheap or impossible.
Failure four: both ends can write
Two-way sync sounds obviously better. It is dramatically harder, and it's where most integration projects go wrong.
The moment both systems can change the same field, you need conflict rules: what happens when a customer's address is edited in both places between syncs? Last-write-wins loses data silently. Timestamp comparison assumes both clocks agree and both systems record modification times accurately, which they often don't. Field-level merging is complex enough that it needs its own testing regime.
What working systems do instead: decide which system owns each piece of information, and make every flow one-directional from the owner. Almost every real business requirement is satisfied this way. Two-way sync is occasionally genuinely necessary; it should be a deliberate, costed decision rather than a default assumption.
Failure five: it trusts the shape of the data
Suppliers change their APIs. Fields get added, occasionally removed, sometimes subtly redefined. A field that was always populated starts arriving empty for a new class of record.
An integration that assumes the shape will hold breaks on the day it doesn't — and typically breaks by writing nonsense rather than by stopping, which is worse.
What working systems do instead: validate on the way in. If a required field is missing or a value is outside the expected range, the record is rejected into a queue for a human to look at, and the rest of the batch continues. You end up with a small pile of exceptions instead of a large pile of corrupted records.
Failure six: nobody is told when it fails
The most common failure of all. The integration is built, it works, and monitoring is left as a later task that never happens. When it eventually breaks, the discovery mechanism is somebody noticing that a number looks wrong.
What working systems do instead: a heartbeat. If the expected run didn't happen, or completed with errors, or returned suspiciously little, somebody is told. The bar here is low — an email is fine — but the absence of it is what turns a two-hour problem into a three-month one.
The uncomfortable part: some things can't be integrated
Occasionally you'll find a system with no API, no scheduled export, and no realistic path to reading its data. There are three honest responses, and choosing deliberately matters more than which you choose.
- Reorganise around it — make the un-connectable system the owner of whatever it holds, and derive everything else from it.
- Shrink the manual step — a person exports a file and drops it somewhere, rather than retyping. Less elegant, entirely reliable.
- Replace the system — legitimate, but a much bigger decision than it looks from inside an integration project.
What doesn't work is scraping a system that doesn't want to be read. It works in the demo, breaks whenever the supplier changes a layout, and breaks silently — usually in the week after everyone stopped keeping the manual version.
What this means when you're buying
You don't need to evaluate any of this technically. You need to ask six questions and listen to the shape of the answers:
- What happens if it runs twice?
- How would we know if it hadn't run at all?
- Does it fetch everything each time, or only what changed?
- Which system owns each piece of information?
- What happens to a record that doesn't look right?
- Who gets told when something fails?
A supplier who has thought about integrations will have crisp answers to all six and will probably be pleased to be asked. One who hasn't will talk about the technologies instead. That distinction is worth more than any reference or portfolio.