Why Local Governments Must Build Their Own AI Knowledge Infrastructure – What new research reveals, and why it matters for counties like ours
A state-of-the-art artificial intelligence system was recently tested on 14,782 questions about U.S. counties, questions about local programs, governance, history, and community life. The best-performing model answered just 56.8% of descriptive questions correctly. When asked to work with local numerical data, performance collapsed to 15.5%.*
Making the models larger didn’t help. Connecting them to web search sometimes made results worse, reducing accuracy by as much as 11.4% for certain systems. The researchers concluded that the limiting factor isn’t computational power. It’s the visibility, structure, and stewardship of local knowledge itself.
This isn’t an abstract technical problem. It’s a question of institutional capacity that will determine whether residents can get reliable answers about their own communities.
The Four Gaps That Break AI
When researchers analyzed where AI systems fail on local questions, four categories accounted for over three-quarters of all errors:
Factual Knowledge Gaps (31.8%): The AI simply doesn’t know the specific details that run local programs; which consulting firm manages your transit study, what year the stormwater ordinance was amended, who operates the weekend food distribution program.
Cultural Misunderstanding (23.4%): A lack of on-the-ground context. The system might know your county has a festival but misunderstand its significance, confuse informal place names, or miss the lived reality of how services actually work versus how they’re described in policy documents.
Geographic Confusion (12.4%): Conflating your county with a neighboring jurisdiction, misidentifying local landmarks, or referencing the wrong community entirely when names overlap across regions.
Temporal Misalignment (9.1%): Providing outdated rules, event timelines from previous years, or information that was once true but has since changed, often because the only indexed version is from 2019.
Each of these failures points to the same root cause: the knowledge that runs county government is distributed across departments, embedded in meeting minutes, encoded in the institutional memory of long-serving staff, and rarely structured for machine readability. No amount of corporate AI development will solve this, because no web-scale training run includes your planning commission’s discussion from last month or the history of why your transit routes run the way they do.
Why the Internet Can’t Save Us
For two decades, digital government meant getting information online: PDFs on websites, scanned documents in repositories, meeting agendas posted for transparency. That approach assumed human readers would do the interpretive work: finding the right document, understanding context, connecting related pieces of information across departments and years.
AI systems expose the limits of that model. When information is locked in PDFs, scattered across departmental silos, or simply never published because “everyone here already knows that,” it becomes invisible to the AI systems residents are increasingly using to understand their community. The internet is not a well-indexed filing cabinet for county government. Local information is fragmented, inconsistent, and often absent entirely.
The research on AI performance makes this concrete. In many cases, web search degraded results because the only available information was generic, outdated, or about a different place with a similar name. When a system retrieves low-quality information, it can perform worse than if it had simply said “I don’t know.”
This means the next phase of digital government is about structuring knowledge itself; treating every policy, program, and procedural fact as a queryable data object with clear provenance, so both people and machines can reliably use it.
What a Knowledge Infrastructure Actually Means
A county knowledge infrastructure would operate on three principles:
Information updated once, trusted everywhere: Every fact about a program, policy, or service becomes a structured data unit. When it changes, it changes once, and that update flows to every system that uses it. Residents asking questions six months from now get current information, not whatever happened to be online when a model was last trained.
Answers that cite their source: Moving toward systems that show exactly which county record, meeting minute, or program description generated an answer. Transparency isn’t just an ethical principle, it’s a technical requirement for residents to trust and verify what they’re told.
Local governance over local information: County staff need the ability to review, correct, and audit how their community’s knowledge is being represented. If an AI system gives an outdated answer about permit requirements, someone with institutional responsibility should be able to see it, flag it, and update the underlying record.
This isn’t theoretical. The same research that documented AI failures also tested a solution: when community members contributed structured narratives about local places and events, and those narratives were integrated into the AI’s knowledge base with clear provenance, accuracy improved measurably on the exact types of questions that had previously failed.
From Consumers to Stewards
Without local knowledge infrastructure, counties will rely on tools shaped by some other communities’ data and priorities. A general-purpose AI trained on national datasets will default to generic answers, big-city assumptions, or information about whichever jurisdiction happened to be better documented online.
With knowledge infrastructure, counties become active stewards:
Preserve institutional memory: Operational knowledge that currently exists only in the experience of long-serving staff can be captured, structured, and made accessible before retirements create information gaps.
Increase organizational capacity: Reduce staff hours spent searching across disconnected systems, re-answering the same questions, or tracking down information that should be centrally maintained.
Ensure accurate representation: Rural and smaller communities get represented accurately rather than through generic assumptions borrowed from larger jurisdictions.
Through our work at the Digital Innovation Collaborative Exchange (DICE) in rural Van Buren and St. Joseph counties in Michigan, we are already building toward this foundation. The work isn’t about chasing the latest AI technology. It’s about ensuring that when technology is used to answer questions about our communities, the answers are grounded in knowledge we govern.
The Stakes
We invest in roads, water systems, and other physical infrastructure because they’re essential to community function. Knowledge is becoming infrastructure of the same order. When it’s fragmented, services slow, decisions get made on incomplete information, and public trust erodes because residents can’t get straight answers to simple questions.
Counties are not behind in this work, we’re essential. We steward some of the most complete local datasets that exist: land records, public health data, court systems, elections. Meaningful AI in civic life depends on the quality and governance of this local information.
In the years ahead, communities that organize, govern, and confidently share their knowledge will be positioned to serve residents effectively. The rest may find that when someone asks a question about a local program, the answer comes from an AI trained on someone else’s county entirely.
That is not a viable way to run local government.
*About the Research
The findings in this article draw on two peer-reviewed studies that systematically document AI’s local knowledge gaps and test community-driven solutions.
LOCALBENCH: Benchmarking LLMs on County-Level Local Knowledge and Reasoning (Gao, Xu & Thebault-Spieker, 2025) created the first comprehensive benchmark for evaluating how well AI systems handle county-specific information. The researchers assembled 14,782 validated question-answer pairs covering 526 U.S. counties across 49 states, drawing from Census data, local news archives, and community forum discussions. When they tested 13 leading AI models—including GPT-4, Claude, and Gemini—the results revealed systematic failures across factual recall, cultural understanding, geographic precision, and temporal accuracy. Crucially, the study found that simply making models larger or connecting them to web search did not solve the problem, pointing instead to fundamental gaps in how local knowledge is structured and made accessible.
Collective Narrative Grounding: Community-Coordinated Data Contributions to Improve Local AI Systems (Gao, Yousufi & Thebault-Spieker, 2025) tested a solution. Through participatory mapping workshops with 24 community members, the researchers developed a protocol for capturing local stories and converting them into structured “narrative units”—data objects that retain the richness of community knowledge while making it queryable by AI systems. When these narrative units were integrated into AI responses with clear source attribution, accuracy improved on precisely the types of questions that had previously failed. The study demonstrates that community-sourced knowledge, when properly structured and governed, can directly address the factual, cultural, geographic, and temporal gaps identified in LOCALBENCH.
Both studies are available as preprints: LOCALBENCH at arXiv:2511.10459 and Collective Narrative Grounding at arXiv:2601.04201. The research was conducted at the University of Wisconsin-Madison, UCLA, and Georgia Tech.
Thank you Zihan Gao, Mohsin Yousufi, Jacob Thebault-Spieker,and Yifei Xu.