A Blueprint for Evaluating AI in Public Administration

This brief presents a blueprint for how government agencies can effectively evaluate AI tools in high-stakes public services.
Key Takeaways
Common AI evaluation methods are insufficient to help governments evaluate AI tools in practice. Despite existing legal mandates, government agencies often struggle to implement effective AI evaluation processes.
Agencies already possess “hidden benchmarks” — administrative records of past task performance that can be used to train and evaluate AI models to meet public sector needs. If released, such benchmarks could both measure and drive AI progress.
Agencies should require third-party evaluations as a condition for procurement and conduct randomized AB tests to understand AI impacts as systems are deployed. Collaborating with universities and other research institutions can offer a cost-effective way to draw on independent outside expertise.
Effective post-monitoring deployment will require rethinking traditional quality assurance approaches to include human-AI collaboration on oversight.
The Problem
In June 2026, autonomous AI agents gained unauthorized access to the Australian Medicare statistics portal, marking the first-known instance of an AI agent hacking a government website. This, and other recent cybersecurity incidents, illustrate the propensity of AI systems to function in ways that deviate from intended behavior and the dangers of lack of oversight. They have also inspired renewed policy proposals and legislation on auditing and evaluation. In California, for example, Governor Gavin Newsom signed Executive Order N-9-26 to accelerate the creation of an independent verification organization framework to evaluate AI systems. Private industry has likewise called for embedded evaluators in an effort to pace the frontier. In response to these incidents, Nvidia CEO Jensen Huang urged frontier labs to allocate 80% of their engineering resources to verification and 20% to development, in contrast to industry norms.
AI evaluation is not a new concern; researchers and policy experts have been developing robust approaches to AI evaluation for years. However, recent incidents have brought critical questions surrounding human oversight of AI into mainstream public debate, particularly with regard to high-impact deployments. Nowhere is this concern more pressing than in the government, which is rapidly becoming one of the largest deployers of AI tools in important domains such as healthcare, education, social services, and law enforcement. In 2025 alone, 56 federal agencies reported 3,611 individual AI use cases within government operations, a 69% increase from the previous year.
As government agencies build chatbots and policy drafting tools, and move toward integrating more agentic AI systems into public services, two categories of risk are becoming particularly salient. The first is safety risk: AI tools can be manipulated or attacked, and agentic systems with access to government data and databases can take unanticipated actions with far-reaching consequences. The second is performance risk: AI systems may fail to perform their intended functions, or fail to deliver the intended benefits after deployment.
This brief addresses how agencies can verify the reliability of an AI tool within the highly consequential setting of public administration, where tools are used to deliver public services more effectively and advance agency missions. We outline the limitations of current approaches to AI evaluation in government and present a blueprint for effective, risk-informed evaluation. Finally, we offer recommendations for how the government can play a leading role in shaping standards and practices for evaluating increasingly powerful AI systems in mission-critical areas.








