In 3 years, Chicago police have tripled their use of a secret, computerized “heat list.”

In July of 2013, the Chicago Police Department began using new software to statistically comb through the city’s population. Based on each person’s past police encounters (and the past encounters of people connected to that person), the system initially flagged the top 400 or so people that it judged to have the greatest chance of shooting someone else or getting shot themselves.

Journalists love this story: it seemed eerily similar to Minority Report, the science fiction film where police punish people for predicted crimes that haven’t actually happened yet.

At the outset, there were some basic questions: Who’s on the list? It’s a secret. How’d they get there? Also secret. And how will police use the list? The cops weren’t sure. Harper’s magazine quoted Yale professor Andrew Papachristos (whose research, drawing parallels between crime and disease, inspired the new system):

“When you look at a geographic map and it says, ‘Here’s a hot spot on the corner of 63rd and Knox,’ you know what to do…You send police cars to that area to deter crime or cool things down. But when you say, ‘This is a group of people who are in a really high-risk social network,’ it’s not clear exactly how to interpret that for policing.” Putting someone on a hot list, Papachristos pointed out, is far simpler than getting someone off of one.

Basic questions still aren’t answered.

The criteria for putting people on the list have changed over time and have never been fully public. Factors that have mattered in the past, and may still matter now, include “the extent of a person’s rap sheet, his or her parole or warrant status, any weapons or drug arrests, his or her acquaintances and their arrest histories — and whether any of those associates have been shot in the past,” according to the Chicago Tribune. Press coverage suggested that most of the lists’ members were involved in gangs in the city.

It also remains unclear whether this police outreach feels like help or like punishment. “The strategy calls for warning those on the heat list individually that further criminal activity, even for the most petty offenses, will result in the full force of the law being brought down on them,” reported the Tribune. “At the same time, police extend them an olive branch of sorts, an offer of help obtaining a job or of social services.”

We learned one thing on Monday: Police are now channeling far more resources through this list than they were at first.

On Monday, the New York Times published a fresh story on the program, which included an interview with newly appointed top cop Eddie Johnson, who took command in the wake of the Laquan McDonald shooting. The new chief is expanding his department’s use of the list.

At first, police were focused on the “roughly 400 people” most at risk. But now?

“[A]bout 1,400 people are responsible for much of the violence, [new chief] Johnson said, and all of them are on what the department [now] calls its Strategic Subject List. . . . Over the past three years, police officers, social workers and community leaders have gone to the homes of more than 1,300 people with high numbers on the list. Mr. Johnson, the police superintendent, said that officials were increasing those visits this year, adding at least 1,000 people.”

Of course, the “length” of Chicago’s heat list is really just a shorthand for how many of the computer’s top picks the city’s police are choosing to act on. What’s really changing is that instead of letting the computer shape their interactions with just 400 highlighted names, Chicago’s police are now taking this intensified approach to at least 1,000 more people.

This remains a small number of people — but it’s a growing fraction of police resources, targeted by a computer the public doesn’t understand.

The Times reports some impressive-sounding statistics:

“So far this year, more than 70 percent of the people who have been shot in Chicago were on the list, according to the police, as were more than 80 percent of those arrested in connection with shootings.

In a broad drug and gang raid carried out last week amid a disturbing uptick this year in shootings and murders, the Police Department said 117 of the 140 people arrested were on the list.”

Assuming that first statistic means what it implies — that 70 percent of shooting victims were already on the list, before they were shot — that’s significant: It shows at least that this system can help predict who will be victims of those shootings the Chicago Police will detect in the future.

At first glance, that might seem to mean that the police have found themselves a big data crystal ball. But in Chicago, or in any city, the police don’t learn about all the shootings. (That’s why there’s a market for automated gunshot detection systems like ShotSpotter.) In 2012, the Justice Department, which runs the largest survey of crime victims in the U.S., found that “[m]ore than half of the nation’s violent crimes …. went unreported to police” between 2006 and 2010. So in order to know whether the heat list is improving CPD’s performance, we need to consider how the heat list may have changed not only who CPD focuses on, but also which (and how many) of the city’s shootings they detect. One possibility is that, by focusing more attention on people on the heat list, the police may be learning less about other people and other shootings. To answer those questions, we need a baseline, independent of police, for how many shootings are happening where — data that may simply not be available today.

Similarly, the fact that police are arresting many of the people who already appear on the heat list is not a meaningful statistic: It might turn out that being on the heat list predicts wrongdoing, or it might turn out that being on the list simply predicts being raided by the police. Look again at the Times’s wording — most of the people arrested in a recent “broad drug and gang raid” were already on the heat list.

I’m not sure exactly what a “broad drug and gang raid” is, but it sure doesn’t sound a careful and specifically justified arrest of an individual who committed a particular crime.

There’s a dangerous temptation here to assume that a computer can’t be biased, or at least can’t be any worse in terms of bias than the cops who take its guidance already are. But in fact, the statistics this system has are police statistics. Those numbers are greatly influenced not only by offending, but also by the enforcement process itself.

So, how could we find out what the heat list’s impacts really are? It’s a tough question, and it’ll take a lot more than just measuring the detected crime rate or the arrest rate. Other data, like surveys of victims, or treatment information about gunshot wounds, might also have a vital role to play. Disentangling bias in the numbers from real changes on the ground is a tough job best done by statisticians who aren’t trying to sell a product to police departments.

Regardless of whether they really reduce crime, and regardless of whether they operate fairly, predictive policing systems like the heat list do have one trait that some may view as a powerful selling point: they concentrate more police encounters on a smaller group of designated “high risk” people. As the Times put it, “[s]upporters of Chicago’s list say that it allows the police to focus on a small fraction of people creating chaos in the city rather than unfairly and ineffectively blanketing whole neighborhoods.”

There are obvious upsides to abandoning “blanket” tactics that unfairly target the innocent. But that shouldn’t stop us from asking hard questions — and demanding clearer answers — about how these systems may intensify coercion of likely bad actors who’ve yet to do anything wrong.