A Spy Spent 3 Years Planting a Backdoor to Bring the Internet Down. One Person Noticed

On a quiet Friday evening in late March 2024, a Microsoft engineer named Andres Freund was running some routine benchmarks on his Debian development box when he noticed something strange. SSH logins were taking about 500 milliseconds longer than they should have. Failed login attempts from automated bots were chewing through an unusual amount of CPU. Most engineers would have shrugged it off. Freund did not. He pulled on the thread, and what he found on the other end was a meticulously planned, state sponsored backdoor that had been three years in the making, hidden inside a tiny compression library that almost nobody had ever heard of, but that sat underneath virtually everything on the internet.

If he had not noticed that half second delay, you might be reading about the worst cybersecurity breach in human history instead of this article.

This is the story of XZ Utils, CVE-2024-3094, and the terrifying fragility hiding in plain sight beneath the digital world.

1. Everything You Do Online Runs on Linux. Everything.

Before we get to the attack, you need to understand something that most people never think about. Almost the entire internet runs on Linux. Not Windows. Not macOS. Linux.

Over 96% of the top one million web servers on Earth run Linux. 92% of all virtual machines across AWS, Google Cloud, and Microsoft Azure run Linux. 100% of the world’s 500 most powerful supercomputers run Linux, and that has been the case since 2017. Android, which powers 85% of the world’s smartphones, is built on the Linux kernel. Every time you send a WhatsApp message, stream Netflix, make a bank transfer, check your email, order food, hail a ride, or scroll through social media, your request is almost certainly being processed by a Linux machine sitting in a data centre somewhere.

Linux is not a product. It is not a company. It started in 1991 when a Finnish university student named Linus Torvalds decided to write his own operating system kernel because he could not afford a UNIX license. The entire philosophy traces back even further, to the 1980s, when Richard Stallman got so frustrated that he could not modify proprietary printer software at MIT to fix a paper jam notification that he launched the Free Software movement and the GNU project. Torvalds wrote the kernel. The GNU project supplied the tools. Together they created a free, open operating system that anyone could inspect, modify, and redistribute.

That openness is why Linux won. It is also why what happened with XZ was possible.

2. The Most Important Software You Have Never Heard Of

XZ Utils is a compression library. It squeezes data to make files smaller. It has no website worth visiting, no marketing team, no venture capital, no logo designed by an agency. It does one thing, quietly and reliably, inside Linux systems across the planet.

You have almost certainly never typed “xz” into anything. But xz has been working for you every single day. It compresses software packages before they are downloaded to your devices. It compresses kernel images. It compresses the backups that keep your data safe. It sits in the dependency chains of tools that handle everything from web traffic to secure shell (SSH) connections, the protocol that system administrators use to remotely manage servers. If SSH is the front door to every Linux server on the internet, xz was sitting in the lock mechanism.

For years, XZ Utils was maintained by essentially one person: a Finnish developer named Lasse Collin. He worked on it in his spare time. There was no salary, no team, no corporate sponsor, no security audit budget. Just one person and an issue queue. This arrangement is completely normal in open source. It is also completely terrifying.

3. The Long Con: A Three Year Espionage Operation

In October 2021, a new GitHub account appeared under the name “Jia Tan.” The account began submitting patches to XZ Utils. Small things. Helpful things. An editor configuration file here, a minor code improvement there. The contributions were competent, consistent, and completely legitimate. Over the next year, Jia Tan built a genuine track record of useful work.

Then, starting in April 2022, something else began happening. A handful of previously unknown accounts began appearing on the XZ Utils mailing list, publicly pressuring Lasse Collin. An account calling itself “Jigar Kumar” complained that patches were languishing and declared that progress would not happen until there was a new maintainer. Another account, “Dennis Ens,” piled on. The messages were not overtly hostile but carried a steady drumbeat of criticism and implied urgency. When Collin responded by mentioning his own long term mental health struggles and limited bandwidth, one of the accounts replied with something that reads, in hindsight, like a calculated psychological strike: expressing sympathy about the mental health issues while suggesting that Collin should recognise his own limits and pass maintainership to someone else.

These were almost certainly sock puppet accounts. Security journalist Brian Krebs later noted that none of the email addresses associated with these accounts appeared anywhere else on the internet, including in massive data breach dumps. That kind of total digital absence across multiple identities is extraordinarily rare and strongly suggests a coordinated operation that was planned from the beginning.

Waiting in the wings with a spotless contribution history was Jia Tan.

Collin eventually relented. By 2023, Jia Tan had been elevated to co-maintainer with direct commit access and release signing privileges. In an email to the mailing list, Collin even noted that Jia Tan had been helping a lot off-list and was “practically a co-maintainer already.”

The trap had been set with extraordinary patience.

4. The Backdoor: Hiding a Master Key in Plain Sight

In February and March 2024, Jia Tan made the move. Two carefully crafted binary test files were committed to the XZ Utils repository. Binary files are not human readable, and storing them in a tests directory is completely normal for a compression library. Nobody would look twice at them. But those files were not test data. They were encrypted payloads.

Jia Tan then modified the build process, specifically a script called build-to-host.m4, so that during compilation the malicious payloads would be silently extracted and injected into the liblzma library. The modified build script was only present in the distributed tar archive, not in the Git source code that most reviewers would examine. This was deliberate misdirection, exploiting the gap between what developers see when they browse a repository and what actually gets compiled on a target system.

The injected code targeted OpenSSH’s authentication mechanism. Through a chain of library dependencies involving systemd and glibc, the backdoor hijacked a cryptographic function called RSA_public_decrypt, replacing it with malicious code. The effect was devastating in its elegance: anyone possessing a specific Ed448 private key could bypass SSH authentication entirely and execute arbitrary code on any affected machine.

In other words, the attacker would have had a master key to every compromised Linux server on Earth.

The vulnerability was assigned CVE-2024-3094 with a CVSS score of 10.0, the maximum possible severity rating. Computer scientist Alex Stamos called it what it was: potentially the most widespread and effective backdoor ever planted in any software product. Akamai’s security researchers noted it would have dwarfed the SolarWinds compromise. The attackers were within weeks of gaining immediate, silent access to hundreds of millions of machines running Fedora, Debian, Ubuntu, and other major distributions.

5. Saved by Half a Second

On 28 March 2024, Andres Freund, a Microsoft principal engineer who also happens to be a PostgreSQL developer and committer, was doing performance testing on a Debian Sid (unstable) installation. He noticed that SSH logins were consuming far more CPU than they should, and that even failing logins from automated bots were taking half a second longer than expected. Half a second – that is the margin by which the internet was saved from what would have been the most catastrophic supply chain attack in computing history.

Freund did not dismiss the anomaly. He investigated. He traced the CPU spike and the latency increase to the updated xz library. He dug into the build artefacts. He found the obfuscated injection code. And on 29 March 2024, he published his findings to the oss-security mailing list.

The response was immediate and global. Red Hat issued an urgent security alert. CISA published an advisory. GitHub suspended Jia Tan’s account and disabled the XZ Utils repository. Every major Linux distribution began emergency rollbacks. Canonical delayed the Ubuntu 24.04 LTS beta release by a full week and performed a complete binary rebuild of every package in the distribution as a precaution.

The tower shook, but it did not fall. And it did not fall because one engineer thought half a second of unexplained latency was worth investigating on a Friday evening.

6. The Uncomfortable Architecture of the Internet

There is a famous XKCD comic, number 2347, that shows the entire modern digital infrastructure as a towering stack of blocks, with one tiny block near the bottom labelled “a project some random person in Nebraska has been thanklessly maintaining since 2003.” It was a joke. Then XZ happened and it stopped being funny.

Here is what the actual dependency stack looks like in simplified form:

            +----------------------------------+
            |  Banking, Healthcare, Government |
            +----------------------------------+
            |  Cloud Platforms (AWS/GCP/Azure) |
            +----------------------------------+
            |  Web Servers and Applications    |
            +----------------------------------+
            |  SSH / OpenSSL / TLS             |
            +----------------------------------+
            |  systemd / glibc / XZ Utils      |
            +----------------------------------+
            |  Linux Kernel                    |
            +----------------------------------+
            |  Hardware                        |
            +----------------------------------+

Each layer assumes the one below it is solid. The higher you build, the less anyone thinks about the foundations. Trillion dollar companies, national defence systems, hospital networks, stock exchanges, telecommunications grids, and critical infrastructure all sit on top of libraries maintained by volunteers who do the work because they care, not because anyone is paying them.

The XZ incident made this fragility impossible to ignore. A compression utility that most people have never heard of turned out to be sitting in the authentication pathway for remote access to Linux systems deployed globally. A single exhausted maintainer was socially engineered into handing the keys to an adversary. And the whole thing nearly went undetected.

7. The Ghost in the Machine

We still do not know who Jia Tan actually is. Analysis of commit timestamps suggests the attacker worked office hours in a UTC+2 or UTC+3 timezone. They worked through Lunar New Year but took off Eastern European holidays including Christmas and New Year. The name “Jia Tan” suggests East Asian origin, possibly Chinese or Hokkien, but the work pattern does not align with that geography. The operational security was exceptional. Every associated email address was created specifically for this campaign and has never appeared in any data breach. Every IP address was routed through proxies.

The consensus among security researchers, including teams at Kaspersky, SentinelOne, Akamai, and CrowdStrike, is that this was almost certainly a state sponsored operation. The patience (three years), the sophistication (the build system injection, the encrypted payloads hidden in test binaries, the deliberate gap between the Git source and the release tarball), and the multi-identity social engineering campaign all point to a resourced intelligence operation, not a lone actor.

SentinelOne’s analysis found evidence that further backdoors were being prepared. Jia Tan had also submitted a commit that quietly disabled Landlock, a Linux kernel sandboxing feature that restricts process privileges. That change was committed under Lasse Collin’s name, suggesting the commit metadata may have been forged. The XZ backdoor, in other words, was likely just the first move in a longer campaign.

8. The Billion Dollar Assumption

Here is the maths that should keep every CIO awake at night. Linux powers an estimated 90% of cloud infrastructure. The global cloud market generates hundreds of billions of dollars in annual revenue. Financial services, healthcare, telecommunications, logistics, defence, and government services all depend on it. SAP reports that 78.5% of its enterprise clients deploy on Linux. The Linux kernel itself contains over 34 million lines of code contributed by more than 11,000 developers across 1,780 organisations.

And yet, deep in the foundations of this ecosystem, critical libraries are maintained by individuals working in their spare time, with no security budget, no formal audit process, no staffing, and no funding proportional to the economic value being extracted from their work.

The companies building on top of this stack generate trillions in aggregate revenue. The people maintaining the foundations often receive nothing. The gap between the value extracted and the investment returned is not a rounding error. It is a structural vulnerability, and the XZ incident proved that adversaries know exactly how to exploit it.

9. Why This Will Happen Again

The uncomfortable truth is that the open source model that made the modern internet possible also created a systemic single point of failure that cannot be patched with a software update.

Social engineering attacks are getting more sophisticated. Large language models can now generate convincing commit histories, craft personalised pressure campaigns adapted to a maintainer’s psychological profile, and manage multiple fake identities simultaneously at a scale that would have been impossible even two years ago. What took the XZ attackers three years of patient reputation building could potentially be compressed into months using AI driven automation.

Meanwhile, the number of single maintainer critical projects has not decreased. The funding landscape has improved marginally through initiatives like the Open Source Security Foundation and GitHub Sponsors, but the investment remains a fraction of what the problem demands. The fundamental dynamic, companies worth billions depending on code maintained by individuals worth nothing to those companies, has not changed.

The XZ backdoor was caught because one curious engineer refused to ignore half a second of unexplained latency. That is not a security strategy. That is luck.

10. What Needs to Change

The Jenga tower still stands, but the XZ incident demonstrated exactly how fragile it is. The blocks at the bottom, the invisible libraries, the thankless utilities, the compression tools nobody has heard of, are the ones holding everything up. And they are precisely the ones receiving the least attention.

The solution is not to abandon open source. The solution is to treat it like the critical infrastructure it actually is. That means sustained corporate investment in the projects companies depend on, not charitable donations but genuine funded maintenance and security audit commitments. It means governance models that can detect and resist social engineering campaigns targeting burnt out solo maintainers. It means recognising that the person maintaining a compression library in their spare time is not a hobbyist. They are, whether they intended it or not, a load bearing wall in the architecture of the global economy.

Richard Stallman started this whole thing because he could not fix a printer. Half a century later, the philosophy of openness he championed underpins nearly every digital interaction on Earth. That is an extraordinary achievement. But the scale has outgrown the model, and the adversaries have noticed.

The next Andres Freund might not be running benchmarks on a Friday evening. The next half second might not get noticed.

11. References