The Black Box on the Corner: Who Audits the AI Deciding Who Looks 'Suspicious'?

OPINION: We are currently witnessing a dangerous handover of public safety infrastructure to AI startups that operate with virtually zero external oversight. As police departments integrate automated surveillance into their daily patrols, we have to ask: who is actually auditing the algorithms that decide which citizens look 'suspicious' on a street corner?

Reporting from WIRED, as first reported by the outlet, has pulled back the curtain on the latest tools from Flock Safety, revealing a system that allows law enforcement to move beyond simple license plate tracking. Through a suite of software, officers can now utilize an AI-powered watchlist to cordon off specific city areas and run continuous, automated searches for any individual fitting a written description. This capability, which Flock calls "FreeForm," allows officers to input plain-language descriptions—such as "person wearing scrubs"—to scan camera footage for matches.

According to WIRED, the mechanism is a statistical game of numbers. Every image captured by a camera is stored as a set of numbers; when an officer types a description, that text is converted into numbers, and a model ranks the footage that most closely matches. To refine these results, a "Smart Sort" feature allows officers to approve or reject images, which then reshuffles the ranking to prioritize footage resembling the approved images.

But here is the problem: the actual logic governing these searches is a black box. WIRED's analysis of the software found that the critical components governing searches happen on Flock's servers, hidden from the police departments using the tool. While Flock claims to have safeguards, the reality is that these guardrails are designed to record abuse, not prevent it.

Even more concerning is the lack of transparency regarding what the AI considers "sensitive." WIRED reports that Flock's system scores written descriptions against categories of sensitive content, including race and religion. However, "political, social, and cultural expression" is the only category that does not stop a search. Tom Bowman, policy counsel with the Center for Democracy and Technology's security surveillance project, told WIRED that this is especially concerning since the First Amendment provides some of its strongest protections to political and cultural expression.

When WIRED questioned Flock Safety on these findings, the company issued a statement claiming its tools help police find relevant information while maintaining "clear safeguards." However, the company failed to explain how its AI model was built, the instructions it follows, or the criteria used to reach its verdicts.

This lack of transparency creates a convenient loophole for accountability. While Flock cautions officers that results may be incomplete or wrong and shouldn't be the sole basis for action, the company explicitly shifts the "risk of any inaccuracies" onto the officer conducting the search. Meanwhile, experts cited by WIRED argue that no technology can reliably perform this kind of automated screening at scale, yet no one outside of Flock can measure how often the system is wrong or identify the nature of its errors.

The potential for abuse is not theoretical. Two members of Congress have already written to Flock CEO Garrett Langley following an incident where a Texas deputy used the system to search more than 83,000 cameras for a woman who had an abortion. Additionally, Illinois discovered that Flock allowed federal immigration agents to access state camera data, violating state law.

In response to the backlash, Flock announced a package of changes to be mandatory by the end of the year, including shorter data retention periods, required case codes for searches, and automated auditing. But as WIRED's analysis shows, these are mere speed bumps. The core of the system—the algorithm that decides who is flagged—remains proprietary and shielded from public or independent audit.

We are trusting the fundamental rights of citizens to a company that refuses to disclose its model's instructions and a police force that is told the technology might be wrong, but is then tasked with the consequences of those errors. If we are going to allow AI to patrol our streets, the logic behind the "suspicion" cannot stay locked on a private server.