The Proxy Metric Fallacy in Cyber Risk Management

Managing proxy metrics fails because the stand in measurement, like a package version's age, only correlates with risk for one narrow attack pattern. It blocks safe updates purely for being new while waving through patient attacks like dormant backdoors. Real risk assessment must weigh the diff, maintainer behavior, exploit status, and how central the package is to your attack surface.

1. Two different questions, and only one of them is being answered

Before assessing whether a three day cooldown is a good idea, it is worth being precise about what question a dependency update policy is supposed to answer, because there are two available and they are not the same question at all.

The first is a net risk question. Every update carries risk in both directions. Taking it exposes you to whatever is wrong with the new release, including the possibility that it is deliberately malicious. Not taking it leaves you exposed to whatever is wrong with the release you are already running, including bugs, instability, and vulnerabilities that have not yet been formally advised. Managing net risk means weighing those two exposures against each other for this particular update, in this particular codebase, and deciding which exposure you would rather hold. It is a genuine judgment, and it can come out either way depending on what the update actually contains and how deeply the package is wired into what you run.

The second is a much narrower question: how old is this release. Managing age means setting a threshold, waiting until it is met, and treating the wait itself as the risk control. It answers quickly, it answers consistently, and it answers without knowing anything about either side of the net risk calculation.

Dependabot’s new cooldown answers the second question. The reasoning behind it is sound as far as it goes: wait three days and most of the malicious releases that get caught fast will have been caught and pulled before you ever see them. That is a real defence against one specific attack pattern, the loud compromise that a community spots within hours. But the mechanism does not weigh the risk of upgrading against the risk of not upgrading. It substitutes elapsed time for that entire calculation, on the assumption that older correlates closely enough with safer to stand in for it.

This is worth naming because it is such a common move, and not just in software supply chain policy. A note on terms first: I am using proxy metric here in the broader sense, meaning a measurable variable used as a surrogate for the risk we actually care about, whether it appears on a dashboard or, as with the cooldown, directly drives a control. Whenever the thing you actually care about is expensive or difficult to measure, there is a strong pull toward measuring something else that merely correlates with it, then managing that surrogate as if it were the thing itself. Age correlates with detection probability for one narrow category of attack. It says close to nothing about the update’s actual content, its provenance, or how much of your system would execute it if you took it.

2. Where the proxy breaks

The clearest way to see the gap between the proxy metric and the real target is to ask what a fixed delay does with information that has nothing to do with elapsed time.

Consider two ordinary version updates, both published 24 hours ago. Release A is a small, well understood change from a mature maintainer, with a transparent diff and a long publishing history behind it. Release B is a sprawling change from a maintainer account that has just started publishing, with unexpected new install scripts and a much larger executable surface than the package has ever shipped before. A blanket cooldown treats these two releases identically. It does not ask what actually changed, who published it, or whether anything about the publishing pattern looks unusual. It asks only how many hours have passed since publication. Release A and Release B get exactly the same treatment, because the rule was never built to distinguish them. That is the tell that age is standing in for risk rather than measuring it.

Now run the scenario the other direction. An attacker compromises a maintainer account and plants a backdoor that stays dormant for weeks before activating. A three day cooldown does nothing here. The article GitHub published about this feature says as much: a cooldown does little against attacks that play a longer game, including backdoors planted in releases and left dormant, maintainer sabotage, or a compromised build system. The proxy fails silently in exactly the case where failure is most expensive, because the thing that made age a useful signal, fast public detection, is precisely what a patient attacker avoids triggering.

So the rule has two failure modes, and they point in opposite directions. It can block something that was never actually dangerous, purely because it is new. And it can wave through something genuinely dangerous, purely because it is old enough to have cleared the timer without ever having been scrutinized.

3. Why teams reach for the proxy anyway

The problem is not that age contains no information about risk. GitHub’s own review of supply chain incidents found that many malicious releases were detected and pulled within hours, so a short delay genuinely does filter some of them out. The problem is that age contains remarkably little information about this specific update, the one sitting in front of you right now. Teams reach for it anyway because actually assessing risk per update is hard, and hard problems get avoided in favor of tractable ones whenever an organization is under pressure to ship a policy rather than a judgment.

It is worth being concrete about what the fuller assessment would actually contain, because the alternative to a proxy is not vagueness. A dependency update decision draws on roughly five signals. Release maturity asks how long the release has existed and whether problems have surfaced in that time. Change risk asks how large the diff is and whether it introduces new binaries, install scripts, network calls or permission changes. Provenance risk asks whether this came from the same maintainer, through the same build process and signing key, with no unexpected ownership or publishing changes. Threat urgency asks whether there is a known vulnerability behind this release, whether it appears on an exploited vulnerabilities catalogue, and whether anyone is attacking it yet. Exposure asks whether you actually execute the code, how privileged and reachable that code is, and how large the blast radius would be if it turned out to be hostile.

Release age belongs squarely in the first of those five. The mistake is not using it. The mistake is pretending the first one answers the other four.

A cooldown timer, by contrast, is trivial to implement, trivial to explain, and trivial to defend to an auditor. It is on by default, so you don’t have to change anything to activate it. That is exactly the appeal, and exactly the danger. Simplicity is being sold as safety, when it is really just legibility. The proxy metric is easy to state and easy to check, which is not the same thing as being right.

It is worth saying that this is not a failure of security teams specifically, and there is a body of research suggesting the substitution is a general feature of how people handle judgments they find intractable. Kahneman and Frederick’s account of attribute substitution, set out in their 2002 revision of the heuristics and biases framework, describes judgment as mediated by a heuristic when a person assesses a target attribute by substituting another property that comes more readily to mind. Their framework sets three conditions for this to happen: the target attribute has to be relatively inaccessible, the substitute has to be readily accessible, and the substitution has to go unrejected by the more deliberate, reflective side of thinking. A dependency update decision satisfies all three almost perfectly. The real target, how dangerous is this release to us, is genuinely hard to access. The substitute, how old is it, could not be more accessible. And because the substitution is dressed up as a security control rather than a mental shortcut, nothing in the process prompts anyone to reject it.

The management literature has studied what happens when this moves from individual judgment into organizational machinery, under the label surrogation. Choi, Hecht and Tayler documented in a 2012 paper in The Accounting Review that managers tend to lose sight of the strategic construct a measure was built to represent and begin acting as though the measure is the construct, and their experiments found this tendency was more pronounced when compensation was tied to a single measure of that construct than when it was spread across several. A follow up study in the Journal of Accounting Research in 2013 found that involving managers in the actual selection of the strategy reduced surrogation, while involving them only in deliberation about it did not appear to have the same effect. Two caveats are worth stating plainly: this work sits in accounting and performance measurement rather than security, and it rests on controlled experiments rather than field observation of security functions, so the transfer is by analogy rather than by evidence. But the structural resemblance is hard to ignore, and it suggests something uncomfortable about single metric security reporting. If surrogation intensifies when a construct is represented by one measure, then a supply chain risk posture represented by a single cooldown setting, or a patching programme represented by a single mean time to remediate figure, is close to the worst case configuration.

The practical implication of the second finding is the one I would hold onto. If genuine involvement in choosing the approach is what reduces the tendency to mistake the measure for the thing, then handing engineering teams a default they never chose and cannot see the reasoning behind is likely to deepen the substitution rather than relieve it. A control that arrives switched on, with no argument attached, is precisely the kind of control people stop interrogating.

4. What happens when the proxy is put under real pressure

The honest test of any proxy metric is what happens when the underlying variable it was supposed to stand in for moves sharply. GitHub’s own security team appears to know this, because they built an escape hatch into the feature from the start, a configuration option to shorten, lengthen, or override the delay on a per project basis. That escape hatch is itself an admission that the fixed default cannot hold up once real risk actually spikes.

So picture the moment a live compromise is reported somewhere in your ecosystem. Nobody yet knows how far it spread, which maintainer accounts were involved, or whether the packages you actually depend on are implicated. No advisory has been published for your dependencies, so nothing here qualifies as a security update and the cooldown governs everything. You have three options, and each one exposes something about what the proxy was really doing for you.

You could lengthen the cooldown sharply across the board, pushing every version update out to a week or two while the picture clarifies. That buys you distance from the incident, but it applies equally to the hundreds of updates that have nothing whatsoever to do with it, and it tells you nothing about whether the specific packages you care about are affected. You have turned the dial on the proxy, not answered the question.

You could shorten or disable the cooldown for a particular dependency, because a version update has just landed that reverts the compromised release and you want it now. That is the right instinct, but notice what has happened: you decided that this specific update, at this specific moment, was worth treating differently from every other update of the same age. The rule did not make that decision. You did, using information the rule never had.

Or you could do what the proxy was standing in for, actually look at the releases in question, check who published them, read what changed, and act accordingly. The point here is not that you should have been doing this for every update all along. At the scale most organizations operate, with tens of thousands of dependency updates flowing through, individual human investigation of every release is economically impossible, and a cheap probabilistic filter is exactly the right tool for the overwhelming majority of them. The question is not why we use the proxy, because at scale we have to. The question is why we let the proxy become the decision, rather than one input to a decision that can still be made properly when something warrants it.

Notice that in all three cases, the actual decision making has to reappear the instant real risk appears. The proxy only worked as a labor saving device precisely because, most of the time, nothing forced anyone to test whether it was tracking the thing it claimed to track.

5. Proxy metrics are not worthless, they are just not the goal

None of this means age is a useless signal. It is a genuinely useful, cheap filter for one specific attack shape, the loud compromise that a community catches within hours. Used that way, as one layer among several, it earns its place. GitHub is explicit that this is the intended scope: a cooldown should be one layer among several, alongside lockfiles, disabled install scripts, scoped tokens, and review before merge. That framing is honest about what the proxy is for.

The fallacy of managing proxy metrics is not in using age as a signal. It is in letting a cheap, legible proxy quietly become the default definition of risk, so that the harder work of actually assessing an update, on its content, its provenance, its severity, its blast radius, stops happening except in the moments when the proxy visibly fails and someone has to step in and do the real evaluation anyway. At that point the organization has not simplified risk management. It has just deferred it to the worst possible moment, the middle of an incident, when there is the least time to do it well.

6. Cybersecurity is full of proxy metrics

Dependabot’s cooldown is a small, contained example, and I have spent most of this article on it because the substitution is unusually visible there. But the same move runs through nearly every part of enterprise security reporting, and once you start looking for it the pattern becomes difficult to unsee.

Every item in the left column is cheap to compute, easy to put on a dashboard, and straightforward to report to a board or an auditor. Every item in the right column is what the organization actually needs to know, and every one of them is expensive, contested, and resistant to being reduced to a single number. That asymmetry is exactly why the left column exists, and it is why the left column tends to win.

The CVSS row deserves a word, because the criticism is not of CVSS itself. CVSS was designed to describe standardized technical severity characteristics and it does that job honestly. The failure happens downstream, when a score gets read as a statement about organizational priority. A remotely exploitable 8.1 sitting on internet facing payment infrastructure can matter enormously more than a 9.8 buried in build tooling that nothing reachable ever executes, and no amount of staring at the two numbers will tell you that. The metric is behaving correctly. The organization is asking it a question it was never built to answer.

None of these metrics is useless, and the point is not that measurement is a mistake. The mistake is allowing the measurable thing to quietly replace the thing we actually wanted to know, so that the left column stops being evidence about the right column and starts being the target in its own right. There is a name for what happens next, Goodhart’s law, and it holds in security as reliably as anywhere else: when a measure becomes a target, it ceases to be a good measure. Coverage percentages get optimized by counting differently rather than by protecting more. Mean time to remediate improves by closing easy findings first. Vulnerability counts fall because the scanner scope narrowed. In each case the number moves in the right direction while the underlying exposure does not move at all.

7. So what: the cost of measuring the real thing is falling

All of this diagnosis is worth very little unless it changes what an organization actually does, so it is worth stating the practical conclusion plainly. Proxies took hold because measuring the real thing was prohibitively expensive, and that constraint was genuine. No organization could read every diff, verify every publishing account, or trace which of its dependencies were actually reachable in production. Age was cheap and everything else was not, so age won, and for years that was a defensible trade rather than an intellectual failure.

What has changed is that most of the five factors are now mechanically computable at a cost that no longer justifies the substitution. Change risk can be evaluated automatically: a diff that introduces install scripts, new network calls, new binaries or new permission requests is machine detectable, and it is a far stronger signal than elapsed time. Provenance is now verifiable rather than assumed, through signed builds and attestation that tie a release to the build process and account that produced it, which means an unexpected publisher or a changed key can be flagged without anyone reading code. Threat urgency can be resolved against exploitation catalogues rather than severity scores alone. Exposure is the hardest of the four, but reachability analysis has moved far enough that an organization can increasingly distinguish a dependency it merely declares from one it actually executes in a privileged, internet facing path.

The reason this shifted is worth naming directly, because it is largely about AI. Reading a diff and forming a view about whether it looks like a normal maintenance change or something structurally unusual, in the specific context of what that package has historically shipped, is exactly the kind of judgment that used to require a scarce human reviewer and now does not. A language model can read every diff in a dependency tree, summarize what changed, flag the introduction of new execution surfaces, notice that a package which has never made a network call suddenly does, and do it across thousands of updates at a cost that bears no resemblance to a human review queue. The same applies to correlating a publishing pattern against a maintainer’s history, or reasoning about whether a given code path is actually reachable from your entry points. None of this is perfect, and none of it should be treated as an oracle, but it does not have to be perfect. It only has to be better than a timer, and the bar a timer sets is genuinely very low.

That is what makes the economic argument for the proxy weaker than it was. The defence of age as a control has always rested on the claim that the real assessment does not scale. That claim was true, and it is becoming less true quite quickly. An organization that is comfortable using AI to write its code and review its pull requests, but still governs dependency risk with a three day timer, has an inconsistency worth confronting.

So the honest challenge to a three day cooldown is not that waiting is wrong. It is that waiting is a placeholder for an assessment we are now capable of making, and every additional day of delay is time bought with lost currency rather than information gained. A policy that says hold every release for three days is doing something strictly worse than a policy that says release this update immediately because the diff is trivial, the provenance is intact and nothing reachable executes it, and hold this other one indefinitely because the publishing account changed last week and it runs in our payment path.

The adaptation this asks of a security function is uncomfortable but specific. It means investing in the ability to measure code risk directly, through diff analysis, provenance verification, exploitation intelligence and reachability, and then demoting age from the decision to what it always should have been, one input among several. It also means accepting that the resulting picture will be messier than a timer and harder to put on a slide, because a real risk position does not reduce to a green tick. That is the price of measuring the thing rather than its shadow, and it is a price worth paying, because the alternative is an organization that knows precisely how old its dependencies are and almost nothing about whether they are dangerous.

There is one more reason to do the hard work rather than settle for the adjacent measure, and it is the reason this matters more in cyber defence than in most other domains where proxies are used. In most fields the proxy fails randomly, because reality is noisy and the correlation is imperfect. In security the proxy fails adversarially, because there is someone on the other side whose job is to find the gap between what you are measuring and what you actually care about, and to operate inside it. An attacker who knows about a three day cooldown simply waits four days. An attacker who knows the scanner’s scope operates outside it. Every proxy metric you rely on is a published description of where you are not looking, and unlike a noisy measurement, an adversary will deliberately seek out and exploit precisely that gap. That is why the usual defence of proxies, that they are right often enough on average, carries much less weight here than it does elsewhere.

8. What the discipline actually requires

The discipline worth keeping is to hold three things apart that organizations routinely collapse into one. Patch age is a signal. A cooldown is a control. Neither of them is the risk. A signal is evidence about the risk, a control is an action taken in response to that evidence, and the risk itself is the thing that will actually hurt you, which remains whatever it was regardless of what your signals report or which controls you have switched on. The moment those three blur together, the dashboards can get greener while the organization gets more dangerous, and nothing in the reporting will tell you it is happening.

In practice this means treating any proxy metric, age, popularity, download count, time since last commit, as a shortcut you are allowed to take when the stakes are low enough that being wrong occasionally is cheap. The moment the stakes rise, the proxy has to be set aside and the actual question asked directly: what does this specific change do, and what is it actually going to cost me if it is wrong. A policy that cannot answer that question when it matters most was never really managing risk. It was managing the appearance of managing risk, and hoping the gap between the two never got tested.