Why More Information Doesn’t Mean More Understanding

We’ve all heard the mantra: data is the new oil. It’s become the rallying cry of digital transformation programmes, investor pitches, and boardroom strategy sessions. But here’s what nobody mentions when they trot out that tired metaphor: oil stinks. It’s toxic. It’s extraordinarily difficult to extract. It requires massive infrastructure, specialised expertise, and relentless refinement before it becomes anything remotely useful. And even then, used carelessly, it poisons everything it touches.

The comparison is more apt than the evangelists realise.

1. The Great Deception

Somewhere along the way, we convinced ourselves that accumulating information was synonymous with gaining understanding. That if we could just capture enough data points, build enough dashboards, and train enough models, clarity would emerge from the chaos. This is perhaps the most dangerous illusion of the modern enterprise.

I’ve watched organisations drown in their own data lakes, though calling them lakes is generous. Most are swamps. Murky, poorly mapped, filled with debris from abandoned projects and undocumented schema changes. Petabytes of customer interactions, transaction logs, sensor readings, and behavioural metrics, all meticulously captured, haphazardly catalogued, and largely ignored. The dashboards multiply. The reports proliferate. And yet the fundamental questions remain unanswered: What should we do? Why are we doing it? What does success actually look like?

Information is not knowledge. Knowledge is not wisdom. And wisdom is not guaranteed by any quantity of the preceding.

2. The Refinement Problem

Crude oil, freshly extracted, is nearly useless. It must be transported, heated, distilled, treated, and transformed through dozens of processes before it becomes the fuel that powers anything. Each step requires expertise, infrastructure, and enormous capital investment. Skip any step, and you’re left with toxic sludge.

Data follows the same brutal economics. Raw data is not an asset. It’s a liability. It costs money to store, creates security and privacy risks, and generates precisely zero value until someone with genuine expertise transforms it into something actionable. Yet organisations hoard data like digital dragons sitting on mountains of gold, convinced that possession equals wealth.

The transformation from data to wisdom requires multiple refinement stages: Data must become information through structure and context. Information must become knowledge through analysis and interpretation. Knowledge must become wisdom through experience, judgement, and critically, self awareness. Each transition demands different skills, different tools, and different kinds of thinking. Most organisations have invested heavily in the first transition and almost nothing in the rest.

3. Tortured Data Will Confess Anything

There’s an old saying among statisticians: torture the data long enough and it will confess to anything. This isn’t a joke. It’s a warning that most organisations have failed to heed.

With enough variables, enough segmentation, and enough creative reframing, you can make data support almost any conclusion you’ve already decided upon. This is the dark side of sophisticated analytics: the tools that should illuminate truth become instruments of confirmation bias. The analyst who brings inconvenient findings gets asked to “look at it differently.” The dashboard that shows declining performance gets redesigned to highlight a more flattering metric. The model that contradicts the executive’s intuition gets retrained until it agrees.

If the data is telling you something that seems wrong, there are two possibilities. The first is that you’ve discovered a genuine insight that challenges your assumptions. This is rare and valuable. The second, far more common, is that something in your data pipeline is broken: bad joins, stale caches, misunderstood definitions, silent failures in upstream systems. Always validate. Always check your assumptions. And be deeply suspicious of any analysis that confirms exactly what you hoped it would.

4. Embedded Lies

Here’s something that keeps me up at night: data doesn’t just contain errors. It contains embedded lies. Not malicious lies, necessarily, but structural deceits built into the very fabric of what we choose to measure and how we measure it.

Consider fraud in financial services. Industry estimates suggest that only around 8% of fraud is actually reported. That means any organisation fixating on reported fraud metrics is studying the tip of an iceberg while congratulating themselves on their visibility. The dashboards look impressive. The trend lines might even be heading in the right direction. But you’re optimising for a shadow of reality.

The organisation that achieves genuine wisdom doesn’t ask “how much fraud was reported last quarter?” It asks questions like: “Who else paid money into accounts we now know were fraudulent but never reported it? What patterns preceded the fraud we caught, and where else do those patterns appear? What are we not seeing, and why?”

These questions are harder. They require linking disparate data sources, challenging comfortable assumptions, and accepting that your metrics have been lying to you. Not because anyone intended deception, but because the data only ever captured what was convenient to capture. The fraud that gets reported is the fraud that was easy to detect. The fraud that doesn’t get reported is, almost by definition, the sophisticated fraud you should actually be worried about.

5. The Illusion of Knowing Ourselves

Here’s where it gets uncomfortable. The data obsession isn’t just an organisational failure. It’s a mirror reflecting a deeper human delusion. We believe we are rational agents making deliberate, informed decisions. Neuroscience and behavioural economics have spent decades demolishing this comfortable fiction.

We are pattern matching machines running on heuristics, rationalising decisions we’ve already made unconsciously. We seek information that confirms what we already believe. We mistake correlation for causation. We see patterns in noise and miss signals in data. We are spectacularly bad at understanding our own motivations, biases, and blind spots.

This matters because organisations are collections of humans, and they inherit all our cognitive limitations while adding a few of their own. When an executive demands “more data” before making a decision, they’re often not seeking understanding. They’re seeking comfort. The data becomes a security blanket, a way to defer responsibility, a defence against future criticism. “The data told us to do it.”

But the data never tells us to do anything. We tell ourselves stories about what the data means, filtered through our assumptions, our incentives, and our fears. Without self knowledge, without understanding our own biases and limitations, more data simply gives us more raw material for self deception.

6. The Famine Amidst Plenty

We are living through a peculiar paradox: a famine of wisdom amidst a gluttony of data. We have more information than any civilisation in history and arguably less capacity to make sense of it. The problem isn’t access. It’s digestion.

Consider how we’ve changed the way we consume information. Twenty years ago, reading a book or a longform article was normal. Today, we scroll through endless feeds, consuming fragments, never staying with any idea long enough to truly understand it. We’ve optimised for breadth at the expense of depth, for novelty at the expense of comprehension, for reaction at the expense of reflection.

Organisations have mirrored this dysfunction. The average executive receives hundreds of emails daily, sits through back to back meetings, and is expected to make consequential decisions in the gaps between. They have access to realtime dashboards showing every conceivable metric, yet they lack the time and mental space to think deeply about any of them. The tyranny of the urgent crowds out the importance of the significant.

Wisdom requires time. It requires sitting with uncertainty. It requires the humility to admit what we don’t know and the patience to discover it properly. None of these things scale. None of them show up on a dashboard. None of them impress investors or boards.

7. What Organisations Should Actually Do

If data is indeed the new oil, then we need to think like refineries, not like hoarders. This means fundamental changes in how we approach information.

First, ruthlessly prioritise. Not all data deserves collection, storage, or analysis. The question isn’t “can we capture this?” but “does this help us make better decisions about things that actually matter?” Most organisations would benefit from capturing less data, not more, but capturing the right data with much greater intentionality.

Second, drain the swamp before building the lake. If you can’t trust your existing data, adding more won’t help. Invest in data quality, in clear ownership, in documentation that actually gets maintained. A small, clean, well understood dataset is infinitely more valuable than a vast murky swamp where nobody knows what’s true.

Third, invest in the refinement stages. For every pound spent on data infrastructure, organisations should be spending at least as much on the human capabilities to interpret it: skilled analysts, yes, but also domain experts who understand context, and experienced leaders who can exercise judgement. The bottleneck is rarely data. It’s the capacity to transform data into actionable understanding.

Fourth, build validation into everything. Assume your data is lying to you until proven otherwise. Cross reference. Sanity check. Ask “what would have to be true for this number to be correct?” and then verify those preconditions. Create a culture where questioning data is rewarded, not punished.

Fifth, ask the questions your data can’t answer. The most important insights often live in the gaps. What aren’t you measuring? What can’t you see? If only 8% of fraud is reported, what does the other 92% look like? These questions require imagination and domain expertise, not just better analytics.

Sixth, create space for reflection. Wisdom doesn’t emerge from realtime dashboards or daily standups. It emerges from stepping back, asking deeper questions, and allowing insights to crystallise over time. This is profoundly countercultural in most organisations, which reward visible activity over invisible thinking. But the most consequential decisions (strategy, culture, longterm investments) require exactly this kind of slow, deliberate cognition.

Seventh, institutionalise self awareness. This might sound soft, but it’s absolutely critical. Decisions made from a place of self knowledge, understanding why we want what we want, recognising our biases, acknowledging our blind spots, are categorically different from decisions made in ignorance of our own psychology. Build in mechanisms that surface assumptions, challenge groupthink, and create psychological safety for dissent.

Eighth, measure what matters. The easiest things to measure are rarely the most important. Clicks are easier to count than customer trust. Output is easier to measure than outcomes. Activity is easier to track than impact. The discipline of identifying what actually matters, and accepting that some of it may resist quantification, is essential to breaking free from data theatre.

8. Decisions From a Place of Knowing

The goal isn’t to reject data. That would be as foolish as rejecting evidence. The goal is to put data in its proper place: as one input among many, useful but not sufficient, informative but not determinative.

The best decisions I’ve witnessed, the ones that created genuine value, that navigated genuine uncertainty, that proved robust in the face of changing circumstances, didn’t come from better dashboards. They came from leaders who understood themselves well enough to know when they were rationalising versus reasoning, who had cultivated judgement through experience and reflection, and who treated data as a conversation partner rather than an oracle.

This kind of wisdom is slow to develop and impossible to automate. It requires exactly the kind of patient, deep work that our information saturated environment makes increasingly difficult. But it remains the essential ingredient that separates organisations that thrive from those that merely survive.

9. Conclusion: From Gluttony to Nourishment

Data is indeed the new oil. Which means it’s messy, it’s dangerous, and in its raw form, it’s nearly useless. It stinks. It requires enormous effort to extract. It demands sophisticated infrastructure and genuine expertise to refine. And like oil, its careless use creates pollution: in this case, pollution of our decisionmaking, our organisations, and our understanding of ourselves.

The organisations that will win the next decade aren’t the ones with the biggest data lakes, or swamps. They’re not the ones with the fanciest analytics platforms or the most impressive dashboards. They’re the ones that recognise the difference between information and understanding, between metrics and meaning, between data and wisdom.

They’ll be the organisations that ask hard questions about what their data isn’t showing them. That validate relentlessly rather than trust blindly. That understand tortured data will confess to anything and refuse to torture it. That recognise the embedded lies in their measurements and actively hunt for what they’re missing.

Most importantly, they’ll be organisations led by people who know themselves. Who understand their own biases, who can distinguish between reasoning and rationalising, who have the humility to admit uncertainty and the patience to sit with it. Because in the end, the quality of our decisions cannot exceed the quality of our self knowledge.

The famine won’t end by consuming more data. It will end when we learn to digest what we already have: slowly, carefully, wisely. When we stop mistaking the swamp for a lake, the noise for a signal, and the comfortable lie for the inconvenient truth.

The first step in that transformation is the hardest one of all: admitting that we don’t know nearly as much as we think we do. Not about our customers, not about our markets, and certainly not about ourselves.

The famine won’t end until we stop gorging and start digesting.

1. Technology Is an Infinite Game and That Is the Point

Technology has no finish line. There is no end state, no final architecture, no moment where you can stand back and declare victory and go home. It is an infinite game made up of a long sequence of hard fought battles, each one draining, each one expensive, each one slower than anyone would like. The moment you solve one problem, the context shifts and the solution becomes the next constraint.

Everything feels too expensive, too difficult and too slow. Under that pressure, a familiar thought pattern emerges. If only we could transfer the risk to someone else. If only we could write enough SLAs, sharpen enough penalties and load the contract until gravity itself guarantees success. If only we could hire lawyers, run a massive outsourcing RFP and make the uncertainty go away.

This is the first lie of the infinite game. Risk does not disappear just because you have moved it onto someone else’s balance sheet. It simply comes back later with interest, usually at the worst possible time.

2. The Euphoria Phase and the Hangover That Follows

The outsourcing cycle always begins with euphoria. There is a media statement. Words like strategic and synergies are deployed liberally. Executive decks are filled with arrows pointing up and to the right. Contracts are signed. Photos are taken. Everyone congratulates each other on having made the hard decisions. Then reality arrives quietly.

Knowledge transfer begins and immediately reveals that much of what matters was never written down. Attrition starts to bite, first at the edges, then at the core. The people who actually understood why things were built the way they were begin to leave, often because they were treated as interchangeable delivery units rather than as the source of the IP itself.

You attempted to turn IP creation into the procurement of pencils. You specified outputs, measured compliance and assumed the essence of the work could be reduced to a checklist. What you actually outsourced was your ability to adapt.

3. Finite Contracts Versus Infinite Competition

Outsourcing is fundamentally a finite game. It is about grinding every last cent out of a well defined specification. It is about predictability, cost control and contractual certainty. Those are not bad things in isolation. Competition, however, is infinite.

You are not playing to complete a statement of work. You are playing on a chessboard with an infinite number of possible moves, where the only goal is to win. To be better than the competition. To innovate faster than they do. To thrive in an environment that changes daily.

The absurdity of this mismatch is often visible in the contracts themselves. Somewhere deep in the appendices you will find a line item for innovation spend, because the board asked for it. As if innovation can be pre purchased, time boxed and invoiced monthly. Innovation is not a deliverable. It is an outcome of ownership, proximity and deep understanding.

4. When Outsourcing Actually Works

Outsourcing does work, but only under very specific conditions. It works when the thing you are outsourcing is not mission critical. When it is a side show. When failure is survivable and learning is optional.

Payroll systems, commodity infrastructure, clearly bounded operational tasks can often be externalised safely. The moment the outsourced capability becomes core to differentiation, speed or revenue generation, the model starts to collapse under its own weight.

The more central the capability is to winning the infinite game, the more dangerous it is to distance yourself from it.

5. Frameworks as a Way Down the Complexity Food Chain

There is a more subtle version of the same instinct, and it shows up in the overuse of vendor frameworks. Platforms like Salesforce allow you to express your IP in a controlled and well defined way. You are deliberately moving yourself down the complexity food chain. Things become easier to reason about, easier to hire for and easier to operate.

There is nothing inherently wrong with this. In many cases it is a rational trade off.

The danger appears when this pattern is applied indiscriminately. When every problem is forced into a vendor shaped abstraction. When flexibility is traded for integration points until you find yourself wrapped in a beautifully integrated set of shackles. Each individual decision looked sensible. The aggregate outcome is paralysis.

You did not remove complexity. You just externalised it and made it harder to escape.

6. No Strategic End State Exists

Technology strategy does not converge. There is no strategic end state. Looking for one is like looking for a strategic newspaper. It changes every day.

Mortgaging today to pay for a hypothetical tomorrow is how organisations lose their ability to move. Long term plans that require years of faith before any value is delivered are a luxury few businesses can afford. Try to operate with a tiny balance sheet. Keep commitments short. Keep feedback loops tight.

Whatever you do, do it efficiently and quickly. Get it to production. Make money from it. Learning that does not touch reality is just theory.

Anyone who walks into the room with a five year roadmap before delivering anything meaningful should be sent back out of it. Some things genuinely do take time, but sustainable businesses run on a varied delivery diet. Small wins, medium bets and the occasional long horizon investment, all running in parallel.

7. The Cost of Playing the Infinite Game

Technology is hard. It always has been. It demands stamina, humility and a tolerance for discomfort. There are no permanent victories, only temporary advantages. The frustration you feel is not a sign of failure. It is the admission price for playing an infinite game.

You do not win by outsourcing the game itself. You win by staying close to the work, owning the risk and being willing to fight the next battle with clear eyes and minimal baggage.

8. The Question Nobody Wants to Ask About Outsourcing

There is a question that almost never gets asked out loud, despite how obvious it is once you say it. How many companies that specialise in outsourcing actually lose money? The answer is vanishingly few. They are very good at what they do. Much better than you.

They have better lawyers than you. They have refined their contracts, their commercial models and their delivery mechanics over decades. They will kill you softly with their way of working. With planning artefacts that look impressive but slow everything down. With invoicing structures that extract value for every ambiguity. With change requests for things you implicitly assumed were included, but were never explicitly written down.

They do not know what your business truly wants or needs, and more importantly, they do not care. They are not misaligned out of malice, they are misaligned by design. Their incentives are not your incentives. Their goal is not to win your market, delight your customers or protect your brand. Their goal is to run a profitable outsourcing business.

They will absolutely cut your costs, at least initially. Headcount numbers go down. Unit costs look better. The spreadsheet tells a comforting story and everyone relaxes. For a while, it feels like the right decision.

Then, slowly, the cracks begin to appear.

Velocity drops. Small changes become negotiations. Workarounds replace understanding. You realise that decisions are being optimised for contractual safety rather than business outcomes. Innovation dries up, except for what can be safely branded as innovation without threatening the delivery model.

By the time this becomes visible to leadership, you are already in trouble. You have missed out on years of engagement, learning and organic innovation. The people who cared deeply about your systems and customers are gone. What remains is a brittle, over constrained estate that nobody fully understands, including the people being paid to run it.

You now have a basket case to fix, under pressure, with fewer options than before. The short term cost savings have been repaid many times over in lost opportunity, lost capability and lost time. This is not a failure of execution. It is the predictable outcome of trying to play an infinite game using finite tools.

9. Conclusion: Embrace the Reality of the Infinite Game

In the end, the frustration we feel with technology, the slowness, cost, complexity, risk, and the urge to outsource it all, is a symptom of a deeper mismatch between the type of game we are playing and the mindset we bring to it. In business and technology there is no final victory, no stable “end-state” where we can declare ourselves done and walk away. We are participating in an infinite game — one where the objective isn’t to “win once” but to remain in the game, continuously adapting, learning, and advancing.

Finite tools like contracts, SLAs, RFPs, tightly bounded outsourcing arrangements, are not evil in themselves. They work well inside a finite context where boundaries, rules and outcomes are clear. But when we try to impose them onto something that has no finish line, we almost guarantee friction, misalignment, and eventual stagnation. The irony is that in trying to control uncertainty, we inadvertently kill the very flexibility and innovation that keep us relevant.

The real lesson is not to avoid complexity, but to embrace the infinite nature of what we’re doing. Technology isn’t a project you finish, it’s a landscape you navigate. Outsourcing may shift certain operational burdens, but it doesn’t transfer the necessity to learn, to iterate, to confront ambiguity, or to sustain competitive advantage. Those remain firmly in your hands.

So if there is a single strategic takeaway, it is this:

Stop looking for an end. Start building a rhythm. Deliver value early. Learn fast. And keep forging ahead.

That is how you thrive not just in technology, but in any domain where the game has no end.

Disaster recovery is one of the most comforting practices in enterprise technology and one of the least honest. Organisations spend significant time and money designing DR strategies, running carefully choreographed exercises, producing polished post exercise reports, and reassuring themselves that they are prepared for major outages. The problem is not intent. The problem is that most DR exercises are optimised to demonstrate control and preparedness in artificial conditions, while real failures are chaotic, asymmetric and hostile to planning. When outages occur under real load, the assumptions underpinning these exercises fail almost immediately.

What most organisations call disaster recovery is closer to rehearsal than resilience. It tests whether people can follow a script, whether environments can be brought online when nothing else is going wrong, and whether senior stakeholders can be reassured. It does not test whether systems can survive reality.

1. DR Exercises Validate Planning Discipline, Not Failure Behaviour

Traditional DR exercises are run like projects. They are planned well in advance, aligned to change freezes, coordinated across teams, and executed when everyone knows exactly what is supposed to happen. This alone invalidates most of the conclusions drawn from them. Real outages are not announced, they do not arrive at convenient times, and they rarely fail cleanly. They emerge as partial failures, ambiguous symptoms and cascading side effects. Alerts contradict each other, dashboards lag reality, and engineers are forced to reason under pressure with incomplete information.

A recovery strategy that depends on precise sequencing, complete information and the availability of specific individuals is fragile by definition. The more a DR exercise depends on human coordination to succeed, the less likely it is to work when humans are stressed, unavailable or wrong. Resilience is not something that can be planned into existence through documentation. It is an emergent property of systems that behave safely when things go wrong without requiring perfect execution.

2. Recovery Is Almost Always Tested in the Absence of Load

Figure 2: Recovery Under Load With and Without Chaos Testing

The single most damaging flaw in DR testing is that it is almost always performed when systems are idle. Queues are empty, clients are disconnected, traffic is suppressed, and downstream systems are healthy. This creates a deeply misleading picture of recoverability. In real outages, load does not disappear. It concentrates. Clients retry, SDKs back off and then retry again, load balancers redistribute traffic aggressively, queues accumulate messages faster than they can be drained, and databases slow down at precisely the moment demand spikes.

Back pressure and integration dependencies are the defining characteristic of real recovery scenarios, and it is almost entirely absent from DR exercises. A system that starts cleanly with no load and all its dependencies ready for traffic, may never become healthy when forced to recover while saturated and partially integrated. Recovery logic that looks correct in isolation frequently collapses when subjected to retry storms and backlog replays. Testing recovery without load is equivalent to testing a fire escape in an empty building and declaring it safe.

3. Recovery Commonly Triggers the Second Outage

DR plans tend to assume orderly reconnection. Services are expected to come back online, accept traffic gradually, and stabilise. Reality delivers the opposite. When systems reappear, clients reconnect simultaneously, message brokers attempt to drain entire backlogs at once, caches stampede databases, authentication systems spike, and internal rate limits are exceeded by internal callers rather than external users.

This thundering herd effect means that recovery itself often becomes the second outage, frequently worse than the first. Systems may technically be up while remaining unusable because they are overwhelmed the moment they re enter service. DR exercises rarely expose this behaviour because load is deliberately suppressed, leading organisations to confuse clean startup with safe recovery.

4. Why Real World DR Testing Is So Hard

The uncomfortable truth is that most organizations avoid real world DR testing not because they are lazy or incompetent, but because the technology they run makes realistic testing commercially irrational.

In traditional enterprise estates a genuine failover is not a minor operational event. A large SQL Server estate or a mainframe environment routinely takes well over an hour to fail over cleanly, and that is assuming everything behaves exactly as designed. During that window queues back up, batch windows are missed, downstream systems time out, and customers feel the impact immediately. Pulling the pin on a system like this during peak volumes is not a test, it is a deliberate business outage. No executive will approve that, and nor should they.

This creates an inevitable compromise. DR tests are scheduled during low load periods, often weekends or nights, precisely when the system behaves best. The back pressure that exists during real trading hours is absent. Cache warm up effects are invisible. Connection storms never happen. Latent data consistency problems remain hidden. The test passes, confidence is reported upward, and nothing meaningful has actually been proven.

The core issue is not testing discipline, it is recovery time characteristics. If your recovery time objective is measured in hours, then every real test carries a material business risk. As a result, organizations rationally choose theater over truth.

Change the technology and the equation changes completely. Platforms like Aurora Serverless fundamentally alter the cost of failure. A failover becomes an operational blip measured in seconds rather than an existential event measured in hours. Endpoints are reattached, capacity is rehydrated automatically, and traffic resumes quickly enough that controlled testing becomes possible even with real workloads. Once confidence is built at lower volumes, the same mechanism can be exercised progressively closer to peak without taking the business hostage.

This is the key distinction most DR conversations miss. You cannot meaningfully test DR if the act of testing is itself catastrophic. Modern architectures that fail fast and recover fast are not just operationally elegant, they are the only ones that make honest DR validation feasible. Everything else optimizes for paperwork, not resilience.

5. Availability Is Tested While Correctness Is Ignored

Most DR exercises optimise for availability signals rather than correctness. They focus on whether systems start, endpoints respond and dashboards turn green, while ignoring whether the system is still “right”. Modern architectures are asynchronous, distributed and event driven. Outages cut through workflows mid execution. Transactions may be partially applied, events may be published but never consumed, compensating actions may not run, and side effects may occur without corresponding state changes.

DR testing almost never validates whether business invariants still hold after recovery. It rarely checks for duplicated actions, missing compensations or widened consistency windows. Availability without correctness is not resilience. It is simply data corruption delivered faster.

6. Idempotency Is Assumed Rather Than Proven

Many systems claim idempotency at an architectural level, but real implementations are usually only partially idempotent. Idempotency keys are often scoped incorrectly, deduplication windows expire too quickly, global uniqueness is not enforced, and side effects are not adequately guarded. External integrations frequently replay blindly, amplifying the problem.

Outages expose these weaknesses because retries occur across multiple layers simultaneously. Messages are delivered more than once, requests are replayed long after original context has been lost, and systems are forced to process duplicates at scale. DR exercises rarely test this behaviour under load. They validate that systems start, not that they behave safely when flooded with replays. Idempotency that only works in steady state is not idempotency. It is an assumption waiting to fail.

7. DNS and Replication Lag Are Treated as Minor Details

DNS based failover is a common component of DR strategies because it looks clean and simple on diagrams. In practice it is unreliable and unpredictable. TTLs are not respected uniformly, client side caches persist far longer than expected, mobile networks are extremely sticky, corporate resolvers behave inconsistently, and CDN propagation is neither instantaneous nor symmetrical.

During real incidents, traffic often arrives from both old and new locations for extended periods. Systems must tolerate split traffic and asymmetric routing rather than assuming clean cutover. DR exercises that expect DNS to behave deterministically are rehearsing a scenario that almost never occurs in production.

8. Hidden Coupling Between Domains Undermines Recovery

Most large scale recovery failures are not caused by the system being recovered, but by something it depends on. Shared authentication services, centralised configuration systems, common message brokers, logging pipelines and global rate limits quietly undermine isolation. During DR exercises these couplings remain invisible because everything is brought up together in a controlled order. In real outages, dependencies fail independently, partially and out of sequence.

True resilience requires domain isolation with explicitly bounded blast radius. If recovery of one system depends on the health of multiple others, none of which are isolated, then recovery is fragile regardless of how well rehearsed it is.

9. Human Factors Are Removed From the Equation

DR exercises assume ideal human conditions. The right people are available, everyone knows it is a test, stress levels are low, and communication is structured and calm. Real incidents are defined by the opposite conditions. People are tired, unavailable or already overloaded, context is missing, and decisions are made under extreme cognitive load.

Systems that require heroics to recover are not resilient. They are brittle. Good systems assume humans will be late, distracted and wrong, and still recover safely.

10. DR Is Designed for Audit Cycles, Not Continuous Failure

Most DR programs exist to satisfy auditors, regulators and risk committees rather than to survive reality. This leads to annual exercises, static runbooks, binary success metrics and a complete absence of continuous feedback. Meanwhile production systems change daily.

A DR plan that is not continuously exercised against live systems is obsolete by default. The confidence it provides is inversely proportional to its accuracy.

11. Chaos Testing Is the Only Honest Substitute

Real resilience is built by failing systems while they are doing real work. That means killing instances under load, partitioning networks unpredictably, breaking dependencies intentionally, injecting latency and observing the blast radius honestly. Chaos testing exposes retry amplification, back pressure collapse, hidden coupling and unsafe assumptions that scripted DR exercises systematically hide.

It is uncomfortable and politically difficult, but it is the only approach that resembles reality. Fortunately, some of the risk of chaos testing can be replicated in a UAT environment. But this requires investment and commitment from senior leaders that understand the value of these kinds of tests. Additionally, actual production outages can be reviewed forensically to essentially give you a “free lesson”, if you have invested in accurate monitoring and take the time to review all production failures.

12. What Systems Should Actually Be Proven To Do

A meaningful resilience strategy does not ask whether systems can be recovered quietly. It proves, continuously, that systems can recover under sustained load, tolerate duplication safely, remain isolated from unrelated domains, degrade gracefully, preserve business invariants and recover with minimal human coordination even when failure timing and scope are unpredictable.

Anything less is optimism masquerading as engineering.

13. The Symmetrical Failure

One of the most dangerous and least discussed failure modes in modern systems is what can be described as a symmetrical failure. It is dangerous precisely because it is fast, silent, and often irreversible by the time anyone realises what has happened.

Imagine a table being accidentally dropped from a production database. In an environment using synchronous replication, near synchronous replication, or block level storage replication, that change is propagated almost immediately to the disaster recovery environment. Within seconds, both production and DR contain the same fault. The table is gone everywhere. At that point DR is not degraded or partially useful. It is completely useless.

This is the defining characteristic of a symmetrical failure. The failure is faithfully replicated across every environment. Replication does not discriminate between correct state and incorrect state. It simply copies bytes. From the outside, everything still looks healthy. Replication is green. Storage is in sync. Latency is low. And yet the system has converged perfectly on a broken outcome.

This class of failure is not limited to dropped tables. Any form of logical corruption that is replicated at the physical or block layer will be propagated without validation. Index corruption, application bugs that write bad data, schema changes gone wrong, runaway batch jobs, or subtle data poisoning all behave the same way. Organisations relying heavily on block level replication often underestimate this risk because the tooling frames faster replication as increased safety. In reality, faster replication often increases the blast radius.

Some symmetrical failures can be rolled back quickly. A small table dropped and detected immediately might be recoverable from backups within an acceptable window. Others are far more intrusive. Large tables, high churn datasets, or corruption detected hours or days later can push recovery times far beyond any realistic business RTO. At that point the discussion is no longer about disaster recovery maturity, but about how long the business can survive without the system.

These failure events must be explicitly designed for, with a fixation on recovery time objectives rather than recovery point objectives alone. RPO answers how much data you might lose. RTO determines whether the business survives the incident. To achieve meaningful RTOs, organisations may need to impose constraints such as maximum database sizes, maximum table sizes, stricter data isolation, or delayed and logically validated replicas. In many cases, achieving the required RTO means changing the architecture rather than tuning the existing one.

If you want resilience, you have to accept that it will not emerge from faster replication or more frequent DR tests. It emerges from designing systems that can tolerate mistakes, corruption, and human error without collapsing instantly.

14. Conclusion

Most disaster recovery exercises do not fail because teams are incompetent. They fail because they test the wrong thing. They validate restart in calm conditions, without load, pressure or ambiguity. That proves very little about how systems and organisations behave when reality intervenes.

Traditional DR misses entire classes of failure including symmetrical failures, silent corruption, hidden coupling, governance paralysis and human breakdown under stress. A DR environment that faithfully mirrors production is only useful if production is still correct and if the organisation can act decisively when it is not.

The next question leaders inevitably ask is, “what is this going to cost?” The uncomfortable but honest answer is that if resilience is done properly, it is not a project or a line item. It is a lifestyle choice. It becomes embedded in how the organisation thinks about architecture, limits, failure and recovery. It shapes design decisions long before incidents occur.

In many cases, other than time and focus, there is little direct investment required. In fact, resilience work often reduces costs. That monolithic 100 terabyte database that runs your entire organisation and requires specialised hardware, specialised skills and multi hour recovery windows is usually a design failure, not a necessity. When it goes on a diet, recovery times collapse. Hardware requirements shrink. Operational complexity drops.

More importantly, resilient designs often introduce stand in capabilities. Caches, queues, read only replicas and degraded modes allow the business to continue processing transactions while recovery is underway. The organisation does not stop simply because the primary system is being repaired. Recovery becomes a background activity rather than a full stop event.

True resilience is not achieved through bigger DR budgets or more elaborate exercises. It is achieved by changing how systems are designed, how limits are enforced and how failure is expected and absorbed. If your recovery strategy only works when everything goes to plan, it is not resilience. It is optimism masquerading as engineering.

There’s a peculiar asymmetry in how humans handle their own incompetence. It reveals itself most starkly when you compare two scenarios: a cancer patient undergoing chemotherapy, and a project manager pushing delivery dates on a complex technology initiative.

Both involve life altering stakes. Both require deep expertise the decision maker doesn’t possess. Yet in one case, we defer completely. In the other, we somehow feel qualified to drive.

The Chemotherapy Paradox

Firstly, lets be clear: incompetence is contextual. Very few people are able to declare themselves as “universally incompetent”. What does this mean? Well just because you have low or no technology competence, it doesn’t mean you are without merit or purpose. The trick is to tie your competencies to the work you are involved in….

When someone receives a cancer diagnosis, something remarkable happens to their ego. They sit across from an oncologist who explains a treatment protocol involving cytotoxic agents that will poison their body in carefully calibrated doses. Their hair will fall out. They’ll experience chronic nausea. Their immune system will crater. The treatment itself might kill them. And they say: “Okay. When do we start?”

This isn’t weakness. It’s wisdom born of acknowledged ignorance. The patient knows they don’t understand the pharmacokinetics of cisplatin or the mechanisms of programmed cell death. They can’t evaluate whether the proposed regimen optimises for tumour response versus quality of life. They lack the fourteen years of training required to even parse the relevant literature.

So they yield. Completely. They ask questions to understand, not to challenge. They follow instructions precisely. They don’t suggest alternative dosing schedules based on something they read online.

This is how humans behave when they genuinely know they don’t know.

The Technology Incompetence Paradox

Now consider the enterprise technology project. A complex migration, perhaps, or a new trading platform. The stakes are high, reputational damage, regulatory exposure, hundreds of millions in potential losses.

The project manager or business sponsor sits across from a principal engineer who explains the technical approach. The engineer describes the challenges: distributed consensus problems, data consistency guarantees, failure mode analysis, performance characteristics under load.

The manager’s eyes glaze slightly. If pressed, they’ll readily admit: “I’m not technical.”

And then, in the very next breath, they’ll ask: “But surely it can’t be that hard? Can’t we just…?”

This is the incompetence paradox in its purest form. The same person who just acknowledged they don’t understand the domain immediately proceeds to:

• Push for aggressive delivery dates
• Propose “simple” solutions
• Question engineering estimates
• Mandate shortcuts they can’t evaluate
• Drive decisions they’re fundamentally unqualified to make
• Ship dates to senior business heads without any engineering validation

In the chemotherapy scenario, acknowledged incompetence produces deference. In the technology scenario, it somehow produces confidence.

Why the Difference?

Several factors drive this asymmetry, and none of them are flattering.

Visibility of consequences. The cancer patient sees the stakes viscerally. The tumour is in their body. The chemotherapy will make them physically ill. The consequences of getting it wrong are personal and immediate. Technology failures, by contrast, are abstract until they’re not. The distributed system that can’t maintain consistency under partition? That’s someone else’s problem until it becomes a P1 incident at 3am.

Illegibility of expertise. Medicine has successfully constructed barriers to amateur interference. White coats. Incomprehensible terminology. Decades of credentialing. Technology, despite being equally complex, has failed to establish similar deference boundaries. Everyone has an iPhone. Everyone has opinions about software.

The Dunning Kruger acceleration. A little knowledge is dangerous, and technology provides just enough surface familiarity to be catastrophically misleading. The manager has used Jira. They’ve seen a Gantt chart. They once wrote an Excel macro. This creates an illusion of adjacent competence that simply doesn’t exist when facing a PET scan.

Accountability diffusion. When chemotherapy fails, the consequences land on a single body. When a technology project fails, it becomes a distributed systems problem of its own, blame fragments across teams, timelines, “changing requirements,” and “unforeseen complexity.” The manager who pushed impossible dates never personally experiences the 4am production incident.

The Absence of Technical Empathy

What’s really missing in failing technology organisations is technical empathy, the capacity to understand, at a meaningful level, what trade offs are being made and why they matter.

When a doctor says “this treatment has a 30% chance of significant side effects,” the patient grasps that this is a trade off. They may not understand the mechanism, but they understand the structure of the decision: accepting known harm for potential benefit.

When an engineer says “if we skip the integration testing phase, we increase the probability of data corruption in production,” the non technical manager hears noise. They don’t have the context to evaluate severity. They don’t understand what “data corruption” actually means for the business. They certainly can’t weigh it against the abstract pressure of “the date.”

So they default to the only metric they can measure: the schedule.

The Project Management Dysfunction

Consider the role of the typical project manager in a failing technology initiative. Their tools are timelines, status reports and burn down charts. Their currency is dates.

When has a project manager ever walked into a steering committee and said: “We need to slow down. There’s too much risk accumulating in this product. The pace is creating technical debt that will compound into failure.”?

They don’t. They can’t. They lack the technical depth to identify the risk, and their incentive structure punishes such honesty even if they could.

Instead, when the date slips, they “rebaseline.” They “replan.” They produce a new Gantt chart that looks exactly like the old one, shifted right by six weeks.

This is treated as project management. It’s actually just administrative recording of failure in progress.

The phrase “we missed the date and are rebaselining” is presented as neutral status reporting. But it obscures a critical question: why did we miss the date? Was it:

• Scope creep from stakeholders who don’t understand impact?
• Technical debt from previous shortcuts coming due?
• Unrealistic estimates imposed by people unqualified to evaluate them?
• Architectural decisions that traded speed for fragility?

The rebaseline answers none of these questions. It simply moves the failure point further into the future, where it will be larger and more expensive.

The Trade Off Vacuum

Here’s a question that exposes the dysfunction: when did a generic manager last table a meaningful technical trade off?

Not “can we do X faster?” That’s not a trade off. That’s just pressure wearing a question mark.

A real trade off sounds like: “If we reduce the scope of automated testing from 80% coverage to 60% coverage, we save three weeks but increase production defect probability by roughly 40%. Given our risk tolerance and the cost of production incidents, is that a trade we want to make?”

This requires understanding what automated testing actually does. What coverage means. How defect probability correlates with test coverage. What production incidents cost.

Generic managers don’t table these trade offs because they can’t. They lack the technical vocabulary, the domain knowledge, and often the intellectual honesty to engage at this level. Instead, they ask: “Why does testing take so long? Can’t we just test the important bits?”

And engineers, exhausted by years of this, learn to either capitulate, obfuscate or buffer their estimates so grossly that the organisation ends up outsourcing the work to another company that is more than happy to create favourable lies about timelines. None of this serves the organisation.

Solving Organisational Problems with Technology (And Making Everything Worse)

There’s a particularly insidious failure mode that emerges from this partial knowledge problem: the instinct to solve organisational dysfunction with technology.

The logic seems sound on the surface. The current system is slow. Teams are frustrated. Data is inconsistent. Processes are manual. The obvious answer? A rewrite. A new platform. A transformation programme.

What follows is depressingly predictable.

The rewrite begins with enthusiasm. A new technology stack is selected, often chosen for its novelty rather than its fit. Kubernetes, because containers are modern. A graph database, because someone read an article. Event sourcing, because it sounds sophisticated. Microservices, because monoliths are unfashionable.

Each decision is wrapped in enough management noise to sound credible. Slide decks proliferate. Vendor presentations are scheduled. Architecture review boards convene. The language is confident: “cloud native,” “future proof,” “scalable by design.”

But anyone with genuine technical depth would immediately challenge the rationality of these decisions. Why do we need a graph database for what is fundamentally a relational problem? What operational capability do we have to run a Kubernetes cluster? Who will maintain this event sourcing infrastructure in three years when the contractors have left?

These questions don’t get asked, because the people making the decisions lack the technical vocabulary to even understand them. And the engineers who could ask them have learned that such questions are career limiting.

So the rewrite proceeds. And the organisation gets worse.

I’ve seen this pattern repeatedly. A legacy system, ugly, creaking, but fundamentally functional, is replaced by a modern platform that is architecturally elegant and operationally catastrophic. The new system requires specialists that don’t exist in the organisation. It has failure modes that nobody anticipated. It solves problems that weren’t actually problems while failing to address the issues that drove the rewrite in the first place.

The teams initially call out a “bedding in period.” The new platform just needs time. People need to adjust. There are teething problems. This is normal.

Months pass. The bedding in period extends. Workarounds accumulate. Shadow spreadsheets emerge. Users quietly route around the new system wherever possible.

Eventually, the inevitable emperor’s robes moment arrives. External specialists are called in, expensive consultants with genuine technical depth, and they deliver the verdict everyone already knew: the new platform is not fit for purpose. The technology choices were inappropriate. The architecture doesn’t match the organisation’s capabilities. The complexity is unjustified.

But by now, tens of millions have been spent. Careers have been built on the transformation. Admitting failure is organisationally impossible. So the platform staggers on, a monument to what happens when partial knowledge drives technology decisions.

The tragedy is that the original problems were often organisational, not technological. The legacy system was slow because processes were broken. Data was inconsistent because ownership was unclear. Teams were frustrated because communication was poor.

No amount of Kubernetes will fix a lack of clear data ownership. No event sourcing architecture will resolve dysfunctional team dynamics. No graph database will compensate for the absence of defined business processes.

But technology feels like action. It appears on roadmaps. It has milestones and deliverables. It can be purchased, installed, and demonstrated. Organisational change is messy, slow, and hard to measure. So we default to technology, and we make everything worse.

The vendors, of course, are delighted to help. They arrive with glossy presentations and reference architectures. They speak with confidence about “digital transformation” and “platform modernisation.” They don’t mention that their incentives are misaligned with yours—they profit from complexity, from licensing, from the ongoing support contracts that complex systems require.

Each unnecessary vendor, each cool but inappropriate technology, each unjustified architectural decision adds another layer of complexity. And complexity is not neutral. It requires expertise to manage. It creates failure modes. It slows everything down. It is, in essence, a tax on every future change.

The partially knowledgeable manager sees a vendor presentation and thinks “this could solve our problems.” The technically competent engineer sees the same presentation and thinks “this would create twelve new problems while solving none of the ones we actually have.”

But the engineer’s voice doesn’t carry. They’re “just technical.” They don’t understand “the business context.” They’re “resistant to change.”

And so the organisation lurches from one technology driven transformation to the next, never addressing the underlying dysfunction, always adding complexity, always wondering why things keep getting worse.

The “Tried and Tested” Fallacy

Here’s where it gets even more frustrating. The non technical leader doesn’t always swing toward shiny new technology. Sometimes they swing to the opposite extreme: “Let’s just use something tried and tested.”

This sounds like wisdom. It sounds like hard won experience tempering youthful enthusiasm. It sounds like the voice of reason.

It’s not. It’s the same dysfunction wearing different clothes.

“Tried and tested” is a lobotomised decision bootstrapped with a meaningless phrase. What does it actually mean? Tried by whom? Tested in what context? Tested against what requirements? Proven suitable for what scale, what failure modes, what operational constraints?

The phrase “tried and tested” is a conversation stopper disguised as a conversation ender. It signals: “We have no appetite for a discussion about technology choices in this technology project.”

Let that sink in. A technology project where the leadership has explicitly opted out of meaningful dialogue about technology choices. This is not conservatism. This is abdication.

The “cool new technology” failure and the “tried and tested” failure are mirror images of the same underlying problem: decisions made without genuine engagement with the technical trade offs.

When someone says “let’s use Kubernetes because it’s modern,” they’re not engaging with whether container orchestration solves any problem you actually have.

When someone says “let’s stick with Oracle because it’s tried and tested,” they’re not engaging with whether a proprietary database at £50,000 per CPU core is justified by your actual consistency and scaling requirements.

Both statements translate to the same thing: “I cannot evaluate this decision on its merits, so I’m using a heuristic that sounds defensible.”

The difference is that “cool technology” gets blamed when projects fail. “Tried and tested” rarely does. If you fail with a boring technology stack, it’s attributed to execution. If you fail with a modern stack, the technology choice itself becomes the scapegoat.

This asymmetry in blame creates a perverse incentive. Non technical leaders learn that “tried and tested” is the career safe choice, regardless of whether it’s the right choice. They’re not optimising for project success. They’re optimising for blame avoidance.

A genuine technology decision process looks nothing like either extreme. It starts with a clear articulation of requirements. It evaluates options against those requirements. It surfaces trade offs explicitly. It makes a choice that the team understands and owns.

“We chose PostgreSQL because our consistency requirements are strict, our scale is moderate, our team has deep expertise, and the operational model fits our on call capacity.”

That’s a decision. “Tried and tested” is not a decision. It’s a refusal to make one while pretending you have.

The Path to Success

The organisations that consistently deliver complex technology successfully share a common characteristic: deep, meaningful dialogue between business stakeholders and engineering teams.

This doesn’t mean business people becoming engineers. It means:

Genuine deference on technical matters. When the engineering team says something is hard, the first response is “help me understand why” rather than “surely it can’t be that hard.”

Trade offs surfaced and owned. When shortcuts are taken, everyone understands what’s being traded. The business explicitly accepts the risk rather than pretending it doesn’t exist.

Subject matter experts in the room. Decisions about architecture, timelines, and scope are made with engineers who understand the implications, not by managers shuffling dates on a chart.

Outcome accountability that includes quality. Project managers measured solely on date adherence will optimise for date adherence, quality be damned. Organisations that include defect rates, production stability, and technical debt in their success metrics get different behaviour.

Permission to slow down. Someone with standing and authority needs the ability to say “stop: we’re accumulating too much risk” and have that statement carry weight.

The Humility Test

There’s a simple test for whether an organisation has healthy technical business relationships. Ask a senior business stakeholder to explain, in their own words, the three most significant technical risks in the current programme.

Not “the timeline is aggressive.” That’s not a technical risk; that’s a schedule statement.

Actual technical risks: “We’re using eventual consistency which means during a failure scenario, customers might see stale data for up to thirty seconds. We’ve accepted this trade off because strong consistency would add four months to the timeline.”

If they can’t articulate anything at this level of specificity, they’re driving a car they don’t understand. And unlike a rental car, when this one crashes, it takes the whole organisation with it.

Conclusion

The cancer patient accepts chemotherapy because they know they don’t know. They yield to expertise. They follow guidance. They ask questions to understand rather than to challenge.

The technology manager pushes dates because they don’t know they don’t know. Their partial knowledge, enough to be dangerous, not enough to be useful, creates false confidence. They challenge without understanding. They drive without seeing the road.

The solution isn’t to make every business stakeholder into an engineer. It’s to cultivate the same humility that the cancer patient naturally possesses: a genuine acceptance that some domains require deference, that expertise matters, and that acknowledging your own incompetence is the first step toward not letting it kill the patient.

In this case, the patient is your programme. And the chemotherapy, the painful, slow, disciplined process of building quality software, is the only treatment that works.

Rebaselining isn’t treatment. It’s just rescheduling the funeral. There is no substitute for meaningful discussions. Planning is just regenerating a stuck thought, over a different timeline.

Darwinian Architecture Philosophy

How Domain Isolation Creates Evolutionary Pressure for Better Software

After two decades building trading platforms and banking systems, I’ve watched the same pattern repeat itself countless times. A production incident occurs. The war room fills. And then the finger pointing begins.

“It’s the database team’s problem.” “No, it’s that batch job from payments.” “Actually, I think it’s the new release from the cards team.” Three weeks later, you might have an answer. Or you might just have a temporary workaround and a room full of people who’ve learned to blame each other more effectively.

This is the tragedy of the commons playing out in enterprise technology, and it’s killing your ability to evolve.

1. The Shared Infrastructure Trap

Traditional enterprise architecture loves shared infrastructure. It makes intuitive sense: why would you run fifteen database clusters when one big one will do? Why have each team manage their own message broker when a central platform team can run one for everybody? Economies of scale. Centralised expertise. Lower costs.

Except that’s not what actually happens.

What happens is that your shared Oracle RAC cluster becomes a battleground. The trading desk needs low latency queries. The batch processing team needs to run massive overnight jobs. The reporting team needs to scan entire tables. Everyone has legitimate needs, and everyone’s needs conflict with everyone else’s. The DBA team becomes a bottleneck, fielding requests from twelve different product owners, all of whom believe their work is the priority.

When the CPU spikes to 100% at 2pm on a Tuesday, the incident call has fifteen people on it, and nobody knows whose query caused it. The monitoring shows increased load, but the load comes from everywhere. Everyone claims their release was tested. Everyone points at someone else.

This isn’t a technical problem. It’s an accountability problem. And you cannot solve accountability problems with better monitoring dashboards.

2. Darwinian Pressure in Software Systems

Nature solved this problem billions of years ago. Organisms that make poor decisions suffer the consequences directly. There’s no committee meeting to discuss why the antelope got eaten. The feedback loop is immediate and unambiguous. Whilst nobody wants to watch it, teams secretly take comfort in not being the limping buffalo at the back of the herd. Teams get fit, they resist decisions that will put them in an unsafe place as they know they will receive an uncomfortable amount of focus from senior management.

Modern software architecture can learn from this. When you isolate domains, truly isolate them, with their own data stores, their own compute, their own failure boundaries, you create Darwinian pressure. Teams that write inefficient code see their own costs rise. Teams that deploy buggy releases see their own services degrade. Teams that don’t invest in resilience suffer their own outages.

There’s no hiding. There’s no ambiguity. There’s no three week investigation to determine fault. There is no watered down document that hints at the issue, but doesn’t really call it out, as all the teams couldn’t agree on something more pointed. The feedback loop tightens from weeks to hours, sometimes minutes.

This isn’t about blame. It’s about learning. When the consequences of your decisions land squarely on your own service, you learn faster. You care more. You invest in the right things because you directly experience the cost of not investing.

3. The Architecture of Isolation

Achieving genuine domain isolation requires more than just drawing boxes on a whiteboard and calling them “microservices.” It requires rethinking how domains interact with each other and with their data.

Data Localisation Through Replication

The hardest shift for most organisations is accepting that data duplication isn’t a sin. In a shared database world, we’re taught that the single source of truth is sacred. Duplicate data creates consistency problems. Normalisation is good.

But in a distributed world, the shared database is the coupling that prevents isolation. If three domains query the same customer table, they’re coupled. An index change that helps one domain might destroy another’s performance. A schema migration requires coordinating across teams. The tragedy of the commons returns.

Instead, each domain should own its data. If another domain needs that data, replicate it. Event driven patterns work well here: when a customer’s address changes, publish an event. Subscribing domains update their local copies. Yes, there’s eventual consistency. Yes, the data might be milliseconds or seconds stale. But in exchange, each domain can optimise its own data structures for its own access patterns, make schema changes without coordinating with half the organisation, and scale its data tier independently.

Queues as Circuit Breakers

Synchronous service to service calls are the other hidden coupling that defeats isolation. When the channel service calls the fraud service, and the fraud service calls the customer service, you’ve created a distributed monolith. A failure anywhere propagates everywhere. An outage in customer data brings down payments.

Asynchronous messaging changes this dynamic entirely. When a payment needs fraud checking, it drops a message on a queue. If the fraud service is slow or down, the queue absorbs the backlog. The payment service doesn’t fail, it just sees increased latency on fraud decisions. Customers might wait a few extra seconds for approval rather than seeing an error page.

This doesn’t make the fraud service’s problems disappear. The fraud team still needs to fix their outage, but you can make business choices about how to deal with the outage. For example, you can choose to bypass the checks for payments to “known” beneficiaries or below certain threshold values, so the blast radius is contained and can be managed. The payments team’s SLAs aren’t destroyed by someone else’s incident. The Darwinian pressure lands where it belongs: on the team whose service is struggling.

Proxy Layers for Graceful Degradation

Not everything can be asynchronous. Sometimes you need a real time answer. But even synchronous dependencies can be isolated through intelligent proxy layers.

A well designed proxy can cache responses, serve stale data during outages, fall back to default behaviours, and implement circuit breakers that fail fast rather than hanging. When the downstream service returns, the proxy heals automatically.

The key insight is that the proxy belongs to the calling domain, not the called domain. The payments team decides how to handle fraud service failures. Maybe they approve transactions under a certain threshold automatically. Maybe they queue high value transactions for manual review. The fraud team doesn’t need to know or care, they just need to get their service healthy again.

4. Escaping the Monolith: Strategies for Service Eviction

Understanding the destination is one thing. Knowing how to get there from where you are is another entirely. Most enterprises aren’t starting with a blank slate. They’re staring at a decade old shared Oracle database with three hundred stored procedures, an enterprise service bus that routes traffic for forty applications, and a monolithic core banking system that everyone is terrified to touch.

The good news is that you don’t need to rebuild everything from scratch. The better news is that you can create structural incentives that make migration inevitable rather than optional.

Service Eviction: Making the Old World Uncomfortable

Service eviction is the deliberate practice of making shared infrastructure progressively less attractive to use while making domain-isolated alternatives progressively more attractive. This isn’t about being obstructive. It’s about aligning incentives with architecture.

Start with change management. On shared infrastructure, every change requires coordination. You need a CAB ticket. You need sign-off from every consuming team. You need a four week lead time and a rollback plan approved by someone three levels up. The change window is 2am Sunday, and if anything goes wrong, you’re in a war room with fifteen other teams.

On domain isolated services, changes are the team’s own business. They deploy when they’re ready. They roll back if they need to. Nobody else is affected because nobody else shares their infrastructure. The contrast becomes visceral: painful, bureaucratic change processes on shared services versus autonomous, rapid iteration on isolated ones.

This isn’t artificial friction. It’s honest friction. Shared infrastructure genuinely does require more coordination because changes genuinely do affect more people. You’re just making the hidden costs visible and letting teams experience them directly.

Data Localisation Through Kafka: Breaking the Database Coupling

The shared database is usually the hardest dependency to break. Everyone queries it. Everyone depends on its schema. Moving data feels impossibly risky.

Kafka changes the game by enabling data localisation without requiring big bang migrations. The pattern works like this: identify a domain that wants autonomy. Have the source system publish events to Kafka whenever relevant data changes. Have the target domain consume those events and maintain its own local copy of the data it needs.

Initially, this looks like unnecessary duplication. The data exists in Oracle and in the domain’s local store. But that duplication is exactly what enables isolation. The domain can now evolve its schema independently. It can optimise its indexes for its access patterns. It can scale its data tier without affecting anyone else. And critically, it can be tested and deployed without coordinating database changes with twelve other teams.

Kafka’s log based architecture makes this particularly powerful. New consumers can replay history to bootstrap their local state. The event stream becomes the source of truth for what changed and when. Individual domains derive their local views from that stream, each optimised for their specific needs.

The key insight is that you’re not migrating data. You’re replicating it through events until the domain no longer needs to query the shared database directly. Once every query can be served from local data, the coupling is broken. The shared database becomes a publisher of events rather than a shared resource everyone depends on.

The Strangler Fig: Gradual Replacement Without Risk

The strangler fig pattern, named after the tropical tree that gradually envelops and replaces its host, is the safest approach to extracting functionality from monoliths. Rather than replacing large systems wholesale, you intercept specific functions at the boundary and gradually route traffic to new implementations.

Put a proxy in front of the monolith. Initially, it routes everything through unchanged. Then, one function at a time, build the replacement in the target domain. Route traffic for that function to the new service while everything else continues to hit the monolith. When the new service is proven, remove the old code from the monolith.

The beauty of this approach is that failure is localised and reversible. If the new service has issues, flip the routing back. The monolith is still there, still working. You haven’t burned any bridges. You can take the time to get it right because you’re not under pressure from a hard cutover deadline.

Combined with Kafka-based data localisation, the strangler pattern becomes even more powerful. The new domain service consumes events to build its local state, the proxy routes relevant traffic to it, and the old monolith gradually loses responsibilities until what remains is small enough to either rewrite completely or simply turn off.

Asymmetric Change Management: The Hidden Accelerator

This is the strategy that sounds controversial but works remarkably well: make change management deliberately asymmetric between shared services and domain isolated services.

On the shared database or monolith, changes require extensive governance. Four week CAB cycles. Impact assessments signed off by every consuming team. Mandatory production support during changes. Post-implementation reviews. Change freezes around month-end, quarter-end, and peak trading periods.

On domain-isolated services, teams own their deployment pipeline end to end. They can deploy multiple times per day if their automation supports it. No CAB tickets. No external sign offs. If they break their own service, they fix their own service.

This asymmetry isn’t punitive. It reflects genuine risk. Changes to shared infrastructure genuinely do have broader blast radius. They genuinely do require more coordination. You’re simply making the cost of that coordination visible rather than hiding it in endless meetings and implicit dependencies.

The effect is predictable. Teams that want to move fast migrate to domain isolation. Teams that are comfortable with quarterly releases can stay on shared infrastructure. Over time, the ambitious teams have extracted their most critical functionality into isolated domains. What remains on shared infrastructure is genuinely stable, rarely changing functionality that doesn’t need rapid iteration.

The natural equilibrium is that shared infrastructure becomes genuinely shared: common utilities, reference data, things that change slowly and benefit from centralisation. Everything else migrates to where it can evolve independently.

The Migration Playbook

Put it together and the playbook looks like this:

First, establish Kafka as your enterprise event backbone. Every system of record publishes events when data changes. This is table stakes for everything else.

Second, identify a domain with high change velocity that’s suffering under shared infrastructure governance. They’re your early adopter. Help them establish their own data store, consuming events from Kafka to maintain local state.

Third, put a strangler proxy in front of relevant monolith functions. Route traffic to the new domain service. Prove it works. Remove the old implementation.

Fourth, give the domain team autonomous deployment capability. Let them experience the difference between deploying through a four-week CAB cycle versus deploying whenever they’re ready.

Fifth, publicise the success. Other teams will notice. They’ll start asking for the same thing. Now you have demand driven migration rather than architecture-mandated migration.

The key is that you’re not forcing anyone to migrate. You’re creating conditions where migration is obviously attractive. The teams that care about velocity self select. The shared infrastructure naturally shrinks to genuinely shared concerns.

5. The Cultural Shift

Architecture is easy compared to culture. You can draw domain boundaries in a week. Convincing people to live within them takes years.

The shared infrastructure model creates a particular kind of learned helplessness. When everything is everyone’s problem, nothing is anyone’s problem. Teams optimise for deflecting blame rather than improving reliability. Political skills matter more than engineering skills. The best career move is often to avoid owning anything that might fail.

Domain isolation flips this dynamic. Teams own their outcomes completely. There’s nowhere to hide, but there’s also genuine autonomy. You can choose your own technology stack. You can release when you’re ready without coordinating with twelve other teams. You can invest in reliability knowing that you’ll reap the benefits directly.

This autonomy attracts a different kind of engineer. People who want to own things. People who take pride in uptime and performance. People who’d rather fix problems than explain why problems aren’t their fault.

The teams that thrive under this model are the ones that learn fastest. They build observability into everything because they need to understand their own systems. They invest in automated testing because they can’t blame someone else when their deploys go wrong. They design for failure because they know they’ll be the ones getting paged.

The teams that don’t adapt… well, that’s the Darwinian part. Their services become known as unreliable. Other teams design around them. Eventually, the organisation notices that some teams consistently deliver and others consistently struggle. The feedback becomes impossible to ignore.

6. Conway’s Law: Accepting the Inevitable, Rejecting the Unnecessary

Melvin Conway observed in 1967 that organisations design systems that mirror their communication structures. Fifty years of software engineering has done nothing to disprove him. Your architecture will reflect your org chart whether you plan for it or not.

This isn’t a problem to be solved. It’s a reality to be acknowledged. Your domain boundaries will follow team boundaries. Your service interfaces will reflect the negotiations between teams. The political realities of your organisation will manifest in your technical architecture. Fighting this is futile.

But here’s what Conway’s Law doesn’t require: shared suffering.

Traditional enterprise architecture interprets Conway’s Law as an argument for centralisation. If teams need to communicate, give them shared infrastructure to communicate through. If domains overlap, put the overlapping data in a shared database. The result is that Conway’s Law manifests not just in system boundaries but in shared pain. When one team struggles, everyone struggles. When one domain has an incident, twelve teams join the war room.

Domain isolation accepts Conway’s Law while rejecting this unnecessary coupling. Yes, your domains will align with your teams. Yes, your service boundaries will reflect organisational reality. But each team’s infrastructure can be genuinely isolated. Public cloud makes this trivially achievable through account-level separation.

Give each domain its own AWS account or Azure subscription. Their blast radius is contained by cloud provider boundaries, not just by architectural diagrams. Their cost allocation is automatic. Their security boundaries are enforced by IAM, not by policy documents. Their quotas and limits are independent. When the fraud team accidentally spins up a thousand Lambda functions, the payments team doesn’t notice because they’re in a completely separate account with separate limits.

Conway’s Law still shapes your domain design. The payments team builds payment services. The fraud team builds fraud services. The boundaries reflect the org chart. But the implementation of those boundaries can be absolute rather than aspirational. Account level isolation means that even if your domain design isn’t perfect, the consequences of imperfection are contained.

This is the insight that transforms Conway’s Law from a constraint into an enabler. You’re not fighting organisational reality. You’re aligning infrastructure isolation with organisational boundaries so that each team genuinely owns their outcomes. The communication overhead that Conway identified still exists, but it happens through well-defined APIs and event contracts rather than through shared database contention and incident calls.

7. The Transition Path

You can’t flip a switch and move from shared infrastructure to domain isolation overnight. The dependencies are too deep. The skills don’t exist. The organisational structures don’t support it.

But you can start. Pick a domain that’s struggling with the current model, probably one that’s constantly blamed for incidents they didn’t cause. Give them their own database, their own compute, their own deployment pipeline. Build the event publishing infrastructure so they can share data with other domains through replication rather than direct queries.

Watch what happens. The team will stumble initially. They’ve never had to think about database sizing or query optimisation because that was always someone else’s job. But within a few months, they’ll own it. They’ll understand their system in a way they never did before. Their incident response will get faster because there’s no ambiguity about whose system is broken.

More importantly, other teams will notice. They’ll see a team that deploys whenever they want, that doesn’t get dragged into incident calls for problems they didn’t cause, that actually controls their own destiny. They’ll start asking for the same thing.

This is how architectural change actually happens, not through mandates from enterprise architecture, but through demonstrated success that creates demand.

8. The Economics Question

I can already hear the objections. “This is more expensive. We’ll have fifteen databases instead of one. Fifteen engineering teams managing infrastructure instead of one platform team.”

To which I’d say: you’re already paying these costs, you’re just hiding them.

Every hour spent in an incident call where twelve teams try to figure out whose code caused the database to spike is a cost. Every delayed release because you’re waiting for a shared schema migration is a cost. Every workaround another team implements because your shared service doesn’t quite meet their needs is a cost. Every engineer who leaves because they’re tired of fighting political battles instead of building software is a cost.

Domain isolation makes these costs visible and allocates them to the teams that incur them. That visibility is uncomfortable, but it’s also the prerequisite for improvement.

And yes, you’ll run more database clusters. But they’ll be right sized for their workloads. You won’t be paying for headroom that exists only because you can’t predict which team will spike load next. You won’t be over provisioning because the shared platform has to handle everyone’s worst case simultaneously.

9. But surely AWS is shared infrastructure?

A common pushback when discussing domain isolation and ownership is: “But surely AWS is shared infrastructure?”
The answer is yes , but that observation misses the point of what ownership actually means in a Darwinian architectural model.

Ownership here is not about blame or liability when something goes wrong. It is about control and autonomy. The critical question is not who gets blamed, but who has the ability to act, change, and learn.

AWS operates under a clearly defined Shared Responsibility Model. AWS is responsible for the security of the cloud, the physical data centres, hardware, networking, and the underlying virtualization layers. Customers are responsible for security in the cloud, everything they configure, deploy, and operate on top of that platform.

Crucially, AWS gives you complete control over the things you are responsible for. You are not handed vague obligations without tools. You are given APIs, policy engines, telemetry, and automation primitives to fully own your outcomes. Identity and access management, network boundaries, encryption, scaling policies, deployment strategies, data durability, and recovery are all explicitly within your control.

This is why AWS being “shared infrastructure” does not undermine architectural ownership. Ownership is not defined by exclusive physical hardware; it is defined by decision-making authority and freedom to evolve. A team that owns its AWS account, VPC, services, and data can change direction without negotiating with a central platform team, can experiment safely within its own blast radius, and can immediately feel the consequences of poor design decisions.

That feedback loop is the point.

From a Darwinian perspective, AWS actually amplifies evolutionary pressure. Teams that design resilient, observable, well isolated systems thrive. Teams that cut corners experience outages, cost overruns, and operational pain, quickly and unambiguously. There is no shared infrastructure committee to absorb the consequences or hide failure behind abstraction layers.

So yes, AWS is shared infrastructure — but it is shared in a way that preserves local control, clear responsibility boundaries, and fast feedback. And those are the exact conditions required for domain isolation to work, and for better software to evolve over time.

10. Evolution, Not Design

The deepest insight from evolutionary biology is that complex, well adapted systems don’t emerge from top down design. They emerge from the accumulation of countless small improvements, each one tested against reality, with failures eliminated and successes preserved.

Enterprise architecture traditionally works the opposite way. Architects design systems from above. Teams implement those designs. Feedback loops are slow and filtered through layers of abstraction. By the time the architecture proves unsuitable, it’s too deeply embedded to change.

Domain isolation enables architectural evolution. Each team can experiment within their boundary. Good patterns spread as other teams observe and adopt them. Bad patterns get contained and eventually eliminated. The overall system improves through distributed learning rather than centralised planning.

This doesn’t mean architects become irrelevant. Someone needs to define the contracts between domains, design the event schemas, establish the standards for how services discover and communicate with each other. But the architect’s role shifts from designing systems to designing the conditions under which good systems can emerge.

10. The End State

I’ve seen organisations make this transition. It takes years, not months. It requires sustained leadership commitment. It forces difficult conversations about team structure and accountability.

But the end state is remarkable. Incident calls have three people on them instead of thirty. Root cause is established in minutes instead of weeks. Teams ship daily instead of quarterly. Engineers actually enjoy their work because they’re building things instead of attending meetings about who broke what.

Pain at the Source

The core idea is deceptively simple: put the pain of an issue right next to its source. When your database is slow, you feel it. When your deployment breaks, you fix it. The feedback loop is immediate and unambiguous.

But here’s what surprises people: this doesn’t make teams selfish. Far from it.

In the shared infrastructure world, teams spend enormous energy on defence. Every incident requires proving innocence. Every performance problem demands demonstrating that your code isn’t the cause. Every outage triggers a political battle over whose budget absorbs the remediation. Teams are exhausted not from building software but from fighting for survival in an environment of ambiguous, omnipresent enterprise guilt.

Domain isolation eliminates this overhead entirely. When your service has a problem, it’s your problem. There’s no ambiguity. There’s no blame game. There’s no three week investigation. You fix it and move on.

Cooperation, Not Competition

And suddenly, teams have energy to spare.

When the fraud team struggles with a complex caching problem, the payments team can offer to help. Not because they’re implicated, not because they’re defending themselves, but because they have genuine expertise and genuine capacity. They arrive as subject matter experts, and the fraud team receives them gratefully as such. There’s no suspicion that help comes with strings attached or that collaboration is really just blame shifting in disguise.

Teams become more cooperative in this world, not less. They show off where they’ve been successful. They write internal blog posts about their observability stack. They present at tech talks about how they achieved sub second deployments. Other teams gladly copy them because there’s no competitive zero sum dynamic. Your success doesn’t threaten my budget. Your innovation doesn’t make my team look bad. We’re all trying to build great software, and we can finally focus on that instead of on survival.

Breaking Hostage Dynamics

And you’re no longer hostage to hostage hiring.

In the shared infrastructure world, a single team can hold the entire organisation ransom. They build a group wide service. It becomes critical. It becomes a disaster. Suddenly they need twenty emergency engineers or the company is at risk. The service shouldn’t exist in the first place, but now it’s too important to fail and too broken to survive without massive investment. The team that created the problem gets rewarded with headcount. The teams that built sustainable, well-designed services get nothing because they’re not on fire.

Domain isolation breaks this perverse incentive. If a team builds a disaster, it’s their disaster. They can’t hold the organisation hostage because their blast radius is contained. Other domains have already designed around them with circuit breakers and fallbacks. The failing service can be deprecated, strangled out, or left to die without taking the company with it. Emergency hiring goes to teams that are succeeding and need to scale, not to teams that are failing and need to be rescued.

The Over Partitioning Trap

I should add a warning: I’ve also seen teams inflict shared pain on themselves, even without shared infrastructure.

They do this by hiring swathes of middle managers and over partitioning into tiny subdomains. Each team becomes responsible for a minuscule pool of resources. Nobody owns anything meaningful. To compensate, they hire armies of planners to try and align these micro teams. The teams fire emails and Jira tickets at each other to inch their ten year roadmap forward. Meetings multiply. Coordination overhead explodes. The organisation has recreated shared infrastructure pain through organisational structure rather than technology.

When something fails in this model, it quickly becomes clear that only a very few people actually understand anything. These elite few become the shared gatekeepers. Without them, no team can do anything. They’re the only ones who know how the pieces fit together, the only ones who can debug cross team issues, the only ones who can approve changes that touch multiple micro domains. You’ve replaced shared database contention with shared human contention. The bottleneck has moved from Oracle to a handful of exhausted architects.

It’s critical not to over partition into tiny subdomains. A domain should be large enough that a team can own something meaningful end to end. They should be able to deliver customer value without coordinating with five other teams. They should understand their entire service, not just their fragment of a service.

These nonsensical subdomains generally only occur when non technical staff have a disproportionately loud voice in team structure. When project managers dominate the discussions and own the narrative for the services. When the org chart is designed around reporting lines and budget centres rather than around software that needs to work together. When the people deciding team boundaries have never debugged a production incident or traced a request across service boundaries.

Domain isolation only works when domains are sized correctly. Too large and you’re back to the tragedy of the commons within the domain. Too small and you’ve created a distributed tragedy of the commons where the shared resource is human coordination rather than technical infrastructure. The sweet spot is teams large enough to own meaningful outcomes and small enough to maintain genuine accountability.

The Commons Solved

The shared infrastructure isn’t completely gone. Some things genuinely benefit from centralisation. But it’s the exception rather than the rule. And crucially, the teams that use shared infrastructure do so by choice, understanding the trade offs, rather than by mandate.

The tragedy of the commons is solved not by better governance of the commons, but by eliminating the commons. Give teams genuine ownership. Let them succeed or fail on their own merits. Trust that the Darwinian pressure will drive improvement faster than any amount of central planning ever could.

Nature figured this out a long time ago. It’s time enterprise architecture caught up.