Over the last couple of months, amidst all the reckoning, three of the biggest companies in tech, IBM, Amazon, and Microsoft, announced that they will stop offering facial recognition software to law enforcement agencies.

IBM led the way with a letter to Congress, in which its CEO, Arvind Krishna, called for “shared responsibility to ensure that Al is tested for bias, particularity when used in law enforcement, and that such bias testing is audited and reported.”

Soon after this announcement, leading news companies like the New York Times, The Washington Post, BBC, and Quartz put out articles stating that IBM has ended its facial recognition program and credited this partially to a landmark research conducted in 2018.

The 2018 Gender Shades research

This influential research by Joy Buolamwini and Timnit Gebru, titled Gender Shades, brought the issue of bias in facial recognition algorithms to the public spotlight by analyzing the accuracy of gender classification products provided by IBM, Microsoft, and Face++.

Buolamwini talks about what led her to do this research and summarizes its findings in this short and insightful video:

The key finding was that these algorithms had a very high accuracy rate overall, but when broken down by gender and color, some disturbing statistics emerge.

The algorithms all performed better on males than females, and on lighter subjects than darker subjects. For example, IBM had an accuracy rate of 96.8 percent on lighter subjects, but only 77.6 percent on darker subjects. The worst affected group was darker females, who had accuracy rates as low as 65 percent.

Owing to these startling findings, it is natural to assume that this research had a key role to play in IBM’s and Microsoft’s decision, but the events that followed raise further questions.

The authors sent IBM the findings of the research in January 2018 and the company responded within 24 hours. In its statement, IBM acknowledged what the research showed but stated that the firm had substantially increased the accuracy of its algorithms in a new version of the software different from the one used by the study.

When the company ran tests on a dataset similar to the research, it found that the new software had an error rate of merely 3.46 percent on darker females. The research, on the other hand, found an error rate of 34.7 percent on this subgroup. With an almost ten-fold reduction in error rate, IBM’s claim meant that it made great leaps in solving the issue of bias, at least when it comes to gender classification.

The follow-up research

Of course, a company’s own report of accuracy should be taken with a grain of salt. And that’s what authors Joy Buolamwini and Inioluwa Deborah Raji did when they published follow-up research titled Actionable Auditing in which they analyzed the impact of the original research. More specifically, they sought to find out if publicly calling out the biased performance of commercial AI products did any good. And, fortunately, they found it did. The companies targeted in the first study were found to have addressed the bias and showed significant improvements within 7 months.

The follow-up study from August 2018 showed that IBM’s new algorithms yielded an error rate of 16.97 percent for darker females, half of what was found in the original research. Microsoft had an even better improvement. It was able to bring down the error rate from 20.8 percent to 1.5 percent for darker females. Both companies explicitly referenced Gender Shades in their product update releases.

These improvements suggest that the companies were closing in on the bias problem, making it all the more hard to understand why they would stop selling these products now.

Factors other than algorithmic bias

If IBM was able to improve by ten-fold (according to its own claim) or even two-fold (according to second research) in seven months, it’s not too far fetched to assume that they could’ve have made even more meaningful improvements in the last year and a half.

Perhaps, it was a business decision. As CNBC points out, IBM facial recognition business did not generate significant revenue for the company.

Perhaps, IBM is not actually exiting this market entirely. The letter merely says that IBM no longer offers “general purpose” facial recognition software. “It’s worth bearing in mind that a lot of the work IBM does is actually customized work for their customers, so it’s possible we’re not seeing the end of IBM doing facial recognition at all, they’re just changing the label,” Eva Blum-Dumontet, a senior researcher at Privacy International, told Business Insider.

Perhaps, the media is blowing this out of proportion. Microsoft does not sell its facial recognition services to police departments. All it did in its recent announcement was reaffirm this stance.

Or perhaps, the real danger is not with bias in the algorithm, but rather with the algorithm being too good. “A perfectly accurate system also becomes an incredibly powerful surveillance tool,” Clare Garvie, a senior associate at Georgetown Law School’s Center on Privacy and Technology, told CNET last year.

Is a perfectly accurate system the problem?

The stance taken by the three companies makes more sense when seen in this context. In IBM’s letter, Arvind calls for a “national dialogue on whether and how facial recognition technology should be employed by domestic law enforcement agencies.” In Amazon’s announcement, the firm calls for “stronger regulations to govern the ethical use of facial recognition technology.” And Microsoft vowed to continue no selling its facial recognition technology to police departments until there is a “national law in place, grounded in human rights, that will govern this technology.” These three stances reflect a concern on how this technology will encroach people’s privacy and democratic freedom rather than algorithmic bias.

This is not to say there is no bias. While the algorithmic bias has significantly reduced over the years, bias clearly exists in how the technology is being used. For example, in most law enforcement applications, any footage is compared to mugshot databases. But these databases contain a disproportionate amount of black people because black people have a higher arrest rate than white people for the same crime. For example, black people are four times more likely to be arrested for marijuana possession even though the use rates are about the same for white and black people. This makes identifying a black person easier with facial recognition technology and “exacerbates racism in a criminal legal system that already disproportionately polices and criminalizes Black people,” writes ACLU President Kade Crockford. He also points out that more surveillance cameras are installed in black and brown neighborhoods. Debiasing technology is within arms reach, but the same cannot be said for people.

It’s more than a racial issue

The concerns surrounding facial recognition stretches far beyond race. Microsoft President Brad Smith outlined some of these in a detailed blog post back in 2018. For example, he says, public establishments could install cameras connected to the cloud that run real-time facial recognition services, intruding the privacy of people giving them no choice. Footage from multiple cameras like these can be combined together to form longer-term histories on a person. “Stores could know immediately when you visited them last and what you looked at or purchased, and by sharing this data with other stores, they could predict what you’re looking to buy on your current visit,” writes Smith.

A far more concerning scenario is if a government uses such technology for continuous surveillance of its people. In a democracy, this deprives people of their fundamental freedoms, and in non-democracies, it can be used as a weapon to crush those who dissent the ruling regime. Today’s technology has reached a level that makes Orwell’s 1984 possible. “It could follow anyone anywhere, or for that matter, everyone everywhere,” writes Smith.

Back in 2017, an algorithm claimed that it can distinguish between gay and straight men based on headshots with an accuracy of 81 percent. “Imagine for a moment the potential consequences if this flawed research were used to support a brutal regime’s efforts to identify and/or persecute people they believed to be gay,” Ashland Johnson, the Human Rights Campaign’s director of public education and research, told Vox.

Most recently, the misuse of this technology was showcased on the very protests that sought to curb it. Police departments across the US ran protest footage and photos through facial recognition software to identify BLM protestors. This is not without precedent. In the past, the police used this technology to identify and target protestors in the Freddie Gray protests, arresting those with outstanding warrants.

Facial recognition technology is dangerous when it isn’t accurate. But it’s far more dangerous when it is. For this reason, the recent announcements made by Microsoft, IBM, and Amazon, are noteworthy but they are only the tip of the iceberg. There are a countless number of smaller companies that continue to develop and provide facial recognition technology that is far more sophisticated to governments and organizations across the world. One such company is Clearview AI, which has a database of over 3 billion photos! The company lets its clients upload any photo and find matches against this database. If you’re wondering how bad could this be, read this article titled I Got My File From Clearview AI, and It Freaked Me Out or watch this segment by John Oliver.

Thank you for reading. If you enjoyed this article, you can sign up for my monthly digest — a curation of my most popular articles.