Legal Zero-Days: A Blind Spot in AI Risk Assessment

Summary

• What’s the problem: LLMs processing vast legal texts could discover ‘legal zero-days’ – unforeseen vulnerabilities in complex legal systems.
• The stakes: One legal vulnerability lay hidden in Australia's Constitution for 116 years before causing 18 months of government disruption. AI could find many more with minimal resources, enabling system-clogging lawfare.
• Bottom line: Policymakers should consider pre-release evaluation for this capability, to anticipate and prevent legal vulnerability exploits.

‍

Current AI safety frameworks evaluate for certain 'canonical' dangerous capabilities – like chemical, biological, radiological and nuclear (CBRN) weapons, cyber operations, and misinformation. Despite growing efforts to evaluate frontier AI models for these known dangerous capabilities, we may be missing entire classes of unknown threats. 

Legal Zero-Days are one previously overlooked category – a novel threat vector through which advanced AI systems, either through misuse or loss of control, might bypass safeguards or accumulate power by exploiting unforeseen vulnerabilities in complex legal systems.

An example helps illustrate the idea. Since 1901, the Australian Constitution has prohibited parliamentarians from having an “allegiance to a foreign power”. Complex citizenship laws have meant that a place of birth or a relative could confer a citizenship right on an Australian, and hence an allegiance to a foreign power. 

In July 2017, a Perth barrister highlighted this conflict and provided evidence that a specific Senator was also a citizen of New Zealand. The crisis spiralled, leading to fifteen sitting politicians being ruled ineligible by the High Court or resigning pre-emptively. Included in this was the Deputy Prime Minister. The ‘eligibility crisis’ caused practical disruption to the Australian Government for almost 18 months, including a period of acute disruption to parliamentary and executive decision-making for the first several weeks of the crisis.

We define a Legal Zero-Day as: (1) a novel discovery about the implications of a law or the interaction between multiple laws; (2) having immediate effect, with ramifications that impact real-world systems without requiring subsequent litigation, lengthy legal processes, or discretionary action; (3) emerging from existing law – disruptive legislation or executive actions do not qualify; (4) causing significant disruption that meaningfully impairs governmental or regulatory operations or other societal functions; and (5) being time-consuming to rectify, lasting weeks or months and resisting simple administrative or discretionary corrections.

Our Legal Zero-Days evaluation asks whether AI systems can detect these vulnerabilities. We tested six frontier AI models on expert-crafted legal ‘puzzles’. The current solve rate of the best model is around 10%, which we judge to be a nascent and likely increasing capability with concerning implications. In practice, we may soon see AI systems that can identify exploitable gaps in contract law, regulatory frameworks or electoral systems. Ongoing work is required to evaluate this risk for increasingly capable models (particularly those with capabilities like ‘deep research’, chain-of-thought, memory and larger context windows). 

Simplified Zero-Day Evaluation Puzzle

If risks like these exist in legal systems, they likely exist across other complex domains. In principle, every complex system that advanced AI can interact with – legal frameworks, financial regulations, supply chains, and emergency protocols – becomes a potential attack vector requiring specialised assessment. 

Take financial regulations as an example. A sufficiently capable AI might identify interactions between obscure securities laws and tax provisions that create opportunities for exploitation like those in the CumEx scandal.

Or consider emergency response protocols, where an AI could discover that conflicting jurisdiction rules create exploitable gaps in disaster response coordination similar to the murder of Alexander Joseph Reed.

Yet comprehensive risk mapping demands domain expertise for each field, custom evaluation frameworks and coordination efforts that stretch far beyond current resources. Meanwhile, AI capabilities advance faster than our ability to discover and evaluate these new risk vectors, creating a widening gap between what we can assess and what we should be assessing.

Recommendations

We recommend four key actions:

The Australian citizenship crisis took 116 years from the Constitution's drafting to materialise – and that was with human-level intelligence searching for vulnerabilities. AI systems that can process vastly more legal text, identify subtle interactions between provisions, and reason about edge cases could accelerate this discovery process dramatically. It’s important for us to take action now.