The Proxy Metric Fallacy in Cyber Risk Management

Managing proxy metrics fails because the stand in measurement, like a package version's age, only correlates with risk for one narrow attack pattern. It blocks safe updates purely for being new while waving through patient attacks like dormant backdoors. Real risk assessment must weigh the diff, maintainer behavior, exploit status, and how central the package is to your attack surface.

1. Two different questions, and only one of them is being answered

Before assessing whether a three day cooldown is a good idea, it is worth being precise about what question a dependency update policy is supposed to answer, because there are two available and they are not the same question at all.

The first is a net risk question. Every update carries risk in both directions. Taking it exposes you to whatever is wrong with the new release, including the possibility that it is deliberately malicious. Not taking it leaves you exposed to whatever is wrong with the release you are already running, including bugs, instability, and vulnerabilities that have not yet been formally advised. Managing net risk means weighing those two exposures against each other for this particular update, in this particular codebase, and deciding which exposure you would rather hold. It is a genuine judgment, and it can come out either way depending on what the update actually contains and how deeply the package is wired into what you run.

The second is a much narrower question: how old is this release. Managing age means setting a threshold, waiting until it is met, and treating the wait itself as the risk control. It answers quickly, it answers consistently, and it answers without knowing anything about either side of the net risk calculation.

Dependabot’s new cooldown answers the second question. The reasoning behind it is sound as far as it goes: wait three days and most of the malicious releases that get caught fast will have been caught and pulled before you ever see them. That is a real defence against one specific attack pattern, the loud compromise that a community spots within hours. But the mechanism does not weigh the risk of upgrading against the risk of not upgrading. It substitutes elapsed time for that entire calculation, on the assumption that older correlates closely enough with safer to stand in for it.

This is worth naming because it is such a common move, and not just in software supply chain policy. A note on terms first: I am using proxy metric here in the broader sense, meaning a measurable variable used as a surrogate for the risk we actually care about, whether it appears on a dashboard or, as with the cooldown, directly drives a control. Whenever the thing you actually care about is expensive or difficult to measure, there is a strong pull toward measuring something else that merely correlates with it, then managing that surrogate as if it were the thing itself. Age correlates with detection probability for one narrow category of attack. It says close to nothing about the update’s actual content, its provenance, or how much of your system would execute it if you took it.

2. Where the proxy breaks

The clearest way to see the gap between the proxy metric and the real target is to ask what a fixed delay does with information that has nothing to do with elapsed time.

Consider two ordinary version updates, both published 24 hours ago. Release A is a small, well understood change from a mature maintainer, with a transparent diff and a long publishing history behind it. Release B is a sprawling change from a maintainer account that has just started publishing, with unexpected new install scripts and a much larger executable surface than the package has ever shipped before. A blanket cooldown treats these two releases identically. It does not ask what actually changed, who published it, or whether anything about the publishing pattern looks unusual. It asks only how many hours have passed since publication. Release A and Release B get exactly the same treatment, because the rule was never built to distinguish them. That is the tell that age is standing in for risk rather than measuring it.

Now run the scenario the other direction. An attacker compromises a maintainer account and plants a backdoor that stays dormant for weeks before activating. A three day cooldown does nothing here. The article GitHub published about this feature says as much: a cooldown does little against attacks that play a longer game, including backdoors planted in releases and left dormant, maintainer sabotage, or a compromised build system. The proxy fails silently in exactly the case where failure is most expensive, because the thing that made age a useful signal, fast public detection, is precisely what a patient attacker avoids triggering.

There is a third failure mode worth setting out carefully, because it lives in the exemption rather than the rule. GitHub is explicit that the cooldown applies only to version updates, and that security updates still open immediately so that critical fixes are never delayed. That carve out is sensible on its face, since holding back a fix for an already public flaw would be perverse. It is also narrower than it first appears, and understanding why matters before deciding how much weight it can bear.

A Dependabot security update is not simply any release a maintainer describes as a security fix. It is triggered by an advisory in the GitHub Advisory Database, and Dependabot then raises a pull request to move the dependency to the minimum version that includes the patch. Two things follow from that. An attacker who merely compromises a maintainer account and publishes a poisoned release gains nothing from the exemption, because that release is an ordinary version update and waits in the cooldown like everything else. And the version Dependabot moves you to is determined by the advisory’s fixed version metadata rather than by whatever happens to be newest on the registry, which removes the obvious path of racing a malicious release to the top of the version list.

So GitHub is not assuming that attackers will never put malicious code into a security fix. It is making a bet that requiring an advisory raises the cost enough to be worth the exposure, combined with the judgment that delaying a disclosed flaw is the worse of the two risks. That is a defensible trade, but it remains a trade, and two shapes survive it. The first is a package that already carries an open advisory. An attacker who compromises that maintainer’s account and publishes what looks like the fix is publishing the minimum patched version, which then flows immediately to every affected repository, reaching precisely the population that has just been told it is exposed and is therefore most motivated to merge quickly. The second is manufacturing the condition itself, since maintainers can report vulnerabilities in their own packages, which gives a compromised or malicious maintainer some ability to create the advisory that unlocks the fast path.

This is the adversarial failure in miniature, and it is why the exemption deserves more attention than the rule. The moment you publish a control with a documented carve out, the carve out becomes the most valuable thing in the system to attack, because it is the one channel guaranteed to bypass the delay and arrive in front of a team already primed to act on it. A random measurement error would not find that channel. An adversary looks for nothing else.

It is also worth knowing that the carve out has not been entirely reliable in practice. There have been reports of cooldown settings being applied to security updates, with Dependabot declining to install a fix because the patched version shipped inside the cooldown window, so the latest version it considered installable sat below the fix. Whatever you think of the underlying design, a control whose exemption does not consistently fire is worse than either option chosen cleanly, because the organization believes it has an escape hatch that is not always open.

So the rule has three failure modes, and they point in different directions. It can block something that was never actually dangerous, purely because it is new. It can wave through something genuinely dangerous, purely because it is old enough to have cleared the timer without ever having been scrutinized. And it can be bypassed entirely through the one route it was designed to leave open.

3. The detection the cooldown depends on is not free

There is a deeper problem with the cooldown, and it only becomes visible when you stop thinking about a single organization and start thinking about the ecosystem the control is embedded in.

The cooldown works by waiting for detection to happen. That is the whole mechanism: three days pass, the community finds the poisoned release, the registry pulls it, and you never see it. But detection is not a natural process that occurs on its own schedule. In the fast catch cases GitHub cites, discovery happened because real people installed the release, ran it, noticed something wrong, and raised the alarm. The chalk and debug compromise was caught in roughly two hours because the packages are downloaded more than two billion times a week, which means an enormous population was executing the malicious code almost immediately. Detection was purchased with their exposure.

That creates an awkward dependency. Waiting three days only protects you if somebody else is not waiting. The cooldown is a strategy that works precisely because most of the ecosystem does not follow it, and its value degrades as adoption rises. GitHub has now made it the default across every supported ecosystem, which is a substantial fraction of the population that consumes new releases through automated tooling. If enough of that population sits out the first three days, the early adopter pool that generates the detection signal gets thinner, discovery takes longer, and the three day window that was calibrated against historical detection times stops matching the detection times it will now produce. The control erodes the conditions that made it work.

I want to be careful not to overstate this, because detection does not come only from adoption. Registry side malware scanning, static analysis by security vendors and automated tooling all inspect new releases without anyone installing them, and that portion of detection capability is unaffected by how many consumers are waiting. There is also a large population of developers who do not use Dependabot at all, who install manually, or whose builds resolve floating version ranges to the newest release regardless of anyone’s cooldown policy, and they continue to provide exposure based detection whatever GitHub’s default says. So the effect is a degradation rather than a collapse. But it is a real degradation, and the direction of travel is clear: the more the ecosystem adopts waiting as a control, the less there is to wait for.

This is a collective action problem in a fairly pure form. For any individual organization, enabling the cooldown is rational, cheap and very likely beneficial. For the ecosystem, universal adoption weakens the detection mechanism that makes the individual choice rational in the first place. Everyone hangs back waiting for somebody else to walk into the minefield, and the fewer volunteers there are, the longer the mines stay undiscovered. Notice also who bears the cost under the current arrangement: detection is generated by the developers who are least cautious, or least well tooled, or simply unaware of the option, and consumed by the organizations sophisticated enough to configure a delay. A control that quietly redistributes risk from the well resourced to the poorly resourced deserves to be named as such, even when each individual decision to adopt it is entirely defensible.

None of this is an argument that you personally should turn the cooldown off, which would be a strange conclusion to draw. It is an argument that a control whose effectiveness depends on other people not using it is a fragile foundation to build a supply chain risk posture on, and that its apparent cheapness is partly an accounting artifact of costs being carried elsewhere. Measuring the release directly has no such property. Reading the diff, checking provenance and establishing reachability produce the same answer regardless of how many other organizations are doing the same thing, and the answer does not get worse as more people adopt the practice. That is a meaningful difference in kind between managing risk and managing a proxy for it.

4. The dial that only turns one way

Suppose you accept the cooldown and run it for a while, and then somebody senior asks the obvious question: is three days the right number. Try to answer it honestly and you discover something uncomfortable about this class of control. There is no way to find out.

To know whether three days was right, you would need to compare two populations: the compromised releases that reached you, and the ones the delay kept out. The second population is invisible by construction. A cooldown that works produces no evidence of working, only an absence of incidents, which is indistinguishable from the absence you would have had if nothing had been aimed at you that month. Nothing comes back from the control to tell you it earned its place. Nothing ever will.

So the dial has a direction that feels safer, no natural stopping point, and no signal capable of telling you that you have gone far enough or too far. Under those conditions the only available response to residual anxiety is to turn it up. Three becomes seven, seven becomes fourteen, and every increment is exactly as unfalsifiable as the one before it. I have seen a seven day gate in the wild, dutifully emailing a digest of the packages it had held back, and the instinct it provokes in anyone worried about supply chain risk is not to ask whether seven is defensible. It is to wonder whether fourteen would be better. If seven does not work, fourteen will not work either, but it will feel twice as safe, and feeling twice as safe is the actual product being sold.

The trap closes with the next move. Fourteen days passes without incident, and that gets read as confirmation that fourteen days was necessary, when it is equally consistent with three days having been sufficient, or with nobody having attacked you at all in that window. The proxy manufactures its own supporting evidence indefinitely, regardless of whether it is doing anything, which means the number can ratchet upward forever and never once be tested. Meanwhile the cost of each turn is real and lands immediately: slower fixes, staler dependencies, more drift to absorb when you finally do upgrade.

Contrast that with measuring the release itself. Reading the diff, checking provenance and establishing reachability produce an answer that can come back release this one today, the change is trivial and nothing reachable executes it. That answer can be wrong, and when it is wrong you tend to find out, which means the method can be corrected and improved over time. It has a failure mode that teaches you something. A timer has no failure mode at all, only a bigger number, and the number never learns.

That property is worth generalizing, because it is how you can spot a proxy metric before it has done any damage. Ask what evidence would tell you the setting is wrong. If the honest answer is that nothing you could observe would distinguish a well calibrated value from a badly calibrated one, you are not managing risk. You are turning a dial and watching your own anxiety for feedback.

5. Why teams reach for the proxy anyway

The problem is not that age contains no information about risk. GitHub’s own review of supply chain incidents found that many malicious releases were detected and pulled within hours, so a short delay genuinely does filter some of them out. The problem is that age contains remarkably little information about this specific update, the one sitting in front of you right now. Teams reach for it anyway because actually assessing risk per update is hard, and hard problems get avoided in favor of tractable ones whenever an organization is under pressure to ship a policy rather than a judgment.

It is worth being concrete about what the fuller assessment would actually contain, because the alternative to a proxy is not vagueness. A dependency update decision draws on roughly five signals. Release maturity asks how long the release has existed and whether problems have surfaced in that time. Change risk asks how large the diff is and whether it introduces new binaries, install scripts, network calls or permission changes. Provenance risk asks whether this came from the same maintainer, through the same build process and signing key, with no unexpected ownership or publishing changes. Threat urgency asks whether there is a known vulnerability behind this release, whether it appears on an exploited vulnerabilities catalogue, and whether anyone is attacking it yet. Exposure asks whether you actually execute the code, how privileged and reachable that code is, and how large the blast radius would be if it turned out to be hostile.

Release age belongs squarely in the first of those five. The mistake is not using it. The mistake is pretending the first one answers the other four.

A cooldown timer, by contrast, is trivial to implement, trivial to explain, and trivial to defend to an auditor. It is on by default, so you don’t have to change anything to activate it. That is exactly the appeal, and exactly the danger. Simplicity is being sold as safety, when it is really just legibility. The proxy metric is easy to state and easy to check, which is not the same thing as being right.

It is worth saying that this is not a failure of security teams specifically, and there is a body of research suggesting the substitution is a general feature of how people handle judgments they find intractable. Kahneman and Frederick’s account of attribute substitution, set out in their 2002 revision of the heuristics and biases framework, describes judgment as mediated by a heuristic when a person assesses a target attribute by substituting another property that comes more readily to mind. Their framework sets three conditions for this to happen: the target attribute has to be relatively inaccessible, the substitute has to be readily accessible, and the substitution has to go unrejected by the more deliberate, reflective side of thinking. A dependency update decision satisfies all three almost perfectly. The real target, how dangerous is this release to us, is genuinely hard to access. The substitute, how old is it, could not be more accessible. And because the substitution is dressed up as a security control rather than a mental shortcut, nothing in the process prompts anyone to reject it.

The management literature has studied what happens when this moves from individual judgment into organizational machinery, under the label surrogation. Choi, Hecht and Tayler documented in a 2012 paper in The Accounting Review that managers tend to lose sight of the strategic construct a measure was built to represent and begin acting as though the measure is the construct, and their experiments found this tendency was more pronounced when compensation was tied to a single measure of that construct than when it was spread across several. A follow up study in the Journal of Accounting Research in 2013 found that involving managers in the actual selection of the strategy reduced surrogation, while involving them only in deliberation about it did not appear to have the same effect. Two caveats are worth stating plainly: this work sits in accounting and performance measurement rather than security, and it rests on controlled experiments rather than field observation of security functions, so the transfer is by analogy rather than by evidence. But the structural resemblance is hard to ignore, and it suggests something uncomfortable about single metric security reporting. If surrogation intensifies when a construct is represented by one measure, then a supply chain risk posture represented by a single cooldown setting, or a patching programme represented by a single mean time to remediate figure, is close to the worst case configuration.

The practical implication of the second finding is the one I would hold onto. If genuine involvement in choosing the approach is what reduces the tendency to mistake the measure for the thing, then handing engineering teams a default they never chose and cannot see the reasoning behind is likely to deepen the substitution rather than relieve it. A control that arrives switched on, with no argument attached, is precisely the kind of control people stop interrogating.

6. What happens when the proxy is put under real pressure

The honest test of any proxy metric is what happens when the underlying variable it was supposed to stand in for moves sharply. GitHub’s own security team appears to know this, because they built an escape hatch into the feature from the start, a configuration option to shorten, lengthen, or override the delay on a per project basis. That escape hatch is itself an admission that the fixed default cannot hold up once real risk actually spikes.

So picture the moment a live compromise is reported somewhere in your ecosystem. Nobody yet knows how far it spread, which maintainer accounts were involved, or whether the packages you actually depend on are implicated. No advisory has been published for your dependencies, so nothing here qualifies as a security update and the cooldown governs everything. You have three options, and each one exposes something about what the proxy was really doing for you.

You could lengthen the cooldown sharply across the board, pushing every version update out to a week or two while the picture clarifies. That buys you distance from the incident, but it applies equally to the hundreds of updates that have nothing whatsoever to do with it, and it tells you nothing about whether the specific packages you care about are affected. You have turned the dial on the proxy, not answered the question.

You could shorten or disable the cooldown for a particular dependency, because a version update has just landed that reverts the compromised release and you want it now. That is the right instinct, but notice what has happened: you decided that this specific update, at this specific moment, was worth treating differently from every other update of the same age. The rule did not make that decision. You did, using information the rule never had.

Or you could do what the proxy was standing in for, actually look at the releases in question, check who published them, read what changed, and act accordingly. The point here is not that you should have been doing this for every update all along. At the scale most organizations operate, with tens of thousands of dependency updates flowing through, individual human investigation of every release is economically impossible, and a cheap probabilistic filter is exactly the right tool for the overwhelming majority of them. The question is not why we use the proxy, because at scale we have to. The question is why we let the proxy become the decision, rather than one input to a decision that can still be made properly when something warrants it.

Notice that in all three cases, the actual decision making has to reappear the instant real risk appears. The proxy only worked as a labor saving device precisely because, most of the time, nothing forced anyone to test whether it was tracking the thing it claimed to track.

7. Proxy metrics are not worthless, they are just not the goal

None of this means age is a useless signal. It is a genuinely useful, cheap filter for one specific attack shape, the loud compromise that a community catches within hours. Used that way, as one layer among several, it earns its place. GitHub is explicit that this is the intended scope: a cooldown should be one layer among several, alongside lockfiles, disabled install scripts, scoped tokens, and review before merge. That framing is honest about what the proxy is for.

The fallacy of managing proxy metrics is not in using age as a signal. It is in letting a cheap, legible proxy quietly become the default definition of risk, so that the harder work of actually assessing an update, on its content, its provenance, its severity, its blast radius, stops happening except in the moments when the proxy visibly fails and someone has to step in and do the real evaluation anyway. At that point the organization has not simplified risk management. It has just deferred it to the worst possible moment, the middle of an incident, when there is the least time to do it well.

8. Cybersecurity is full of proxy metrics

Dependabot’s cooldown is a small, contained example, and I have spent most of this article on it because the substitution is unusually visible there. But the same move runs through nearly every part of enterprise security reporting, and once you start looking for it the pattern becomes difficult to unsee.

Every item in the left column is cheap to compute, easy to put on a dashboard, and straightforward to report to a board or an auditor. Every item in the right column is what the organization actually needs to know, and every one of them is expensive, contested, and resistant to being reduced to a single number. That asymmetry is exactly why the left column exists, and it is why the left column tends to win.

The CVSS row deserves a word, because the criticism is not of CVSS itself. CVSS was designed to describe standardized technical severity characteristics and it does that job honestly. The failure happens downstream, when a score gets read as a statement about organizational priority. A remotely exploitable 8.1 sitting on internet facing payment infrastructure can matter enormously more than a 9.8 buried in build tooling that nothing reachable ever executes, and no amount of staring at the two numbers will tell you that. The metric is behaving correctly. The organization is asking it a question it was never built to answer.

None of these metrics is useless, and the point is not that measurement is a mistake. The mistake is allowing the measurable thing to quietly replace the thing we actually wanted to know, so that the left column stops being evidence about the right column and starts being the target in its own right. There is a name for what happens next, Goodhart’s law, and it holds in security as reliably as anywhere else: when a measure becomes a target, it ceases to be a good measure. Coverage percentages get optimized by counting differently rather than by protecting more. Mean time to remediate improves by closing easy findings first. Vulnerability counts fall because the scanner scope narrowed. In each case the number moves in the right direction while the underlying exposure does not move at all.

9. So what: the cost of measuring the real thing is falling

All of this diagnosis is worth very little unless it changes what an organization actually does, so it is worth stating the practical conclusion plainly. Proxies took hold because measuring the real thing was prohibitively expensive, and that constraint was genuine. No organization could read every diff, verify every publishing account, or trace which of its dependencies were actually reachable in production. Age was cheap and everything else was not, so age won, and for years that was a defensible trade rather than an intellectual failure.

What has changed is that most of the five factors are now mechanically computable at a cost that no longer justifies the substitution. Change risk can be evaluated automatically: a diff that introduces install scripts, new network calls, new binaries or new permission requests is machine detectable, and it is a far stronger signal than elapsed time. Provenance is now verifiable rather than assumed, through signed builds and attestation that tie a release to the build process and account that produced it, which means an unexpected publisher or a changed key can be flagged without anyone reading code. Threat urgency can be resolved against exploitation catalogues rather than severity scores alone. Exposure is the hardest of the four, but reachability analysis has moved far enough that an organization can increasingly distinguish a dependency it merely declares from one it actually executes in a privileged, internet facing path.

The reason this shifted is worth naming directly, because it is largely about AI. Reading a diff and forming a view about whether it looks like a normal maintenance change or something structurally unusual, in the specific context of what that package has historically shipped, is exactly the kind of judgment that used to require a scarce human reviewer and now does not. A language model can read every diff in a dependency tree, summarize what changed, flag the introduction of new execution surfaces, notice that a package which has never made a network call suddenly does, and do it across thousands of updates at a cost that bears no resemblance to a human review queue. The same applies to correlating a publishing pattern against a maintainer’s history, or reasoning about whether a given code path is actually reachable from your entry points. None of this is perfect, and none of it should be treated as an oracle, but it does not have to be perfect. It only has to be better than a timer, and the bar a timer sets is genuinely very low.

That is what makes the economic argument for the proxy weaker than it was. The defence of age as a control has always rested on the claim that the real assessment does not scale. That claim was true, and it is becoming less true quite quickly. An organization that is comfortable using AI to write its code and review its pull requests, but still governs dependency risk with a three day timer, has an inconsistency worth confronting.

So the honest challenge to a three day cooldown is not that waiting is wrong. It is that waiting is a placeholder for an assessment we are now capable of making, and every additional day of delay is time bought with lost currency rather than information gained. A policy that says hold every release for three days is doing something strictly worse than a policy that says release this update immediately because the diff is trivial, the provenance is intact and nothing reachable executes it, and hold this other one indefinitely because the publishing account changed last week and it runs in our payment path.

The adaptation this asks of a security function is uncomfortable but specific. It means investing in the ability to measure code risk directly, through diff analysis, provenance verification, exploitation intelligence and reachability, and then demoting age from the decision to what it always should have been, one input among several. It also means accepting that the resulting picture will be messier than a timer and harder to put on a slide, because a real risk position does not reduce to a green tick. That is the price of measuring the thing rather than its shadow, and it is a price worth paying, because the alternative is an organization that knows precisely how old its dependencies are and almost nothing about whether they are dangerous.

There is one more reason to do the hard work rather than settle for the adjacent measure, and it is the reason this matters more in cyber defence than in most other domains where proxies are used. In most fields the proxy fails randomly, because reality is noisy and the correlation is imperfect. In security the proxy fails adversarially, because there is someone on the other side whose job is to find the gap between what you are measuring and what you actually care about, and to operate inside it. An attacker who knows about a three day cooldown simply waits four days. An attacker who knows the scanner’s scope operates outside it. Every proxy metric you rely on is a published description of where you are not looking, and unlike a noisy measurement, an adversary will deliberately seek out and exploit precisely that gap. That is why the usual defence of proxies, that they are right often enough on average, carries much less weight here than it does elsewhere.

10. What the discipline actually requires

The discipline worth keeping is to hold three things apart that organizations routinely collapse into one. Patch age is a signal. A cooldown is a control. Neither of them is the risk. A signal is evidence about the risk, a control is an action taken in response to that evidence, and the risk itself is the thing that will actually hurt you, which remains whatever it was regardless of what your signals report or which controls you have switched on. The moment those three blur together, the dashboards can get greener while the organization gets more dangerous, and nothing in the reporting will tell you it is happening.

In practice this means treating any proxy metric, age, popularity, download count, time since last commit, as a shortcut you are allowed to take when the stakes are low enough that being wrong occasionally is cheap. The moment the stakes rise, the proxy has to be set aside and the actual question asked directly: what does this specific change do, and what is it actually going to cost me if it is wrong. A policy that cannot answer that question when it matters most was never really managing risk. It was managing the appearance of managing risk, and hoping the gap between the two never got tested.