Stanford HAI 2019 Fall Conference – Race, Rights and Facial Recognition

– My name is Daniel Murray I’m the executive director of Stanford’s Center for Comparative Studies in Race and Ethnicity We’re a research and teaching center here committed to advancing racial justice through the comparative understanding of race Before we get started, I’d like to begin by acknowledging that the land that we are standing on here at Stanford at the Hoover Institution, indeed in Silicon Valley, is the ancestral, unceded land of the Muwekma Ohlone tribe I think it’s important for us at our center and for us here at Stanford to recognize that every member of the community has benefited and continues to benefit from the use and occupation of this land I’d also like to thank HAI, the organizers of the conference, Rob Reich, Fei-Fei Li, John Etchemendy, and Celia Clark Thanks for putting together a great couple of days to talk about some very important issues here at Stanford and in the world At CCSRE, our center, we’ve recently launched a race and technology initiative of which this panel is a part And for this work we’ve recently won a national challenge grant, competition for public interest technology hosted by New America, funded through Ford, Hewlett, and Rick’s foundations, and through this an in partnership with HAI and other Stanford partners, like the Stanford Digital Civil Society Lab and others, will be hosting a cohort of practitioner fellows starting in January throughout the year We will have a conference in Spring on digital technology and racial justice, and an array of other programs, research and educational opportunities throughout the year So we hope you will join us for that You can find out more information on our website and follow us on Twitter and social media @stanfordccsre Now to our panel In our discussion today, we will focus on racial justice in the development and deployment of AI technology, with some specific focus on facial recognition technology As we’ve heard throughout the conference, there are dangers in biased data, technology, and implementation Joy Buolamwini, who unfortunately is unable to join us today, but, hopefully some of you were able to hear her talk yesterday, if not, can find it online, illustrated how facial recognition technology works great, so long as you look like me, in her words, “pale and male.” Now at present, this flawed technology leads to challenges from inconvenience to wrongful arrests to much worse So do we solve this particular problem through better data sets? But this raises concerns too In one troubling recent example, Google contractors in Atlanta attempted to address this problem by capturing the faces of homeless people of color without their consent But the problems are not only about flawed technology and better data Increased surveillance, particularly in communities of color that are already over-policed, raises equity and civil liberty concerns And emerging technologies like affect recognition, which purports to determine emotional states, raises another host of concerns This technology can be used in classrooms, workplaces, and hiring to determine whether someone is engaged and other emotional states Stanford’s own psychology professor, Jeanne Tsai, one of our faculty affiliates, and others have shown how culturally-specific emotion is How do we define engaged as an emotional state and how does that relate to one’s ability to do a successful job? Will it lead to more disciplinary action for children of color in classrooms? These are some of the challenging concerns that the new technologies raise They require a diverse interdisciplinary and cross-sectional debate as we design and implement new technologies, as we determine the ethical and legal guidelines, and that’s what this conference and this panel hopes to do today So I’ll introduce each panelist in turn, and they’ll present for about 15 or 20 minutes on their work and research to help ground our discussion Then, we’ll discuss as a panel and open for Q and A with all of you So, with that, I’d like to introduce our first speaker, Matt Cagle Matt is a technology and civil liberties attorney at the ACLU of Northern California, where he works on digital rights issues, including privacy, surveillance, and free-speech Matt focuses on ensuring modern digital systems are publicly debated and implemented with equality and justice in mind This includes work on the use of surveillance technology by local police and the promotion of best practices for online platforms He was instrumental in the recent ban on the use of facial recognition technology in San Francisco in Oakland, as well as the recently enacted California law that blocks police from adding facial recognition technology to body cameras He wrote and helped to pass those laws He’s a graduate of Stanford Law School as well, where’s he’s a member editor of the Stanford journal of civil rights and civil liberties Join me in welcoming Matthew Cagel (audience clapping) – Thank you Hello, everyone, and thanks for having me today

I want to thank the Stanford institute for hosting us today and bringing together this amazing conversation So my name is Matt Cagel, I’m with the ACLU of Northern California, based in San Francisco, and today I wanna talk about the work that the ACLU has been doing, with our coalition partners, to push back against facial recognition and really sound the alarm about this technology and its impact on our society So, with my time, I wanna broadly go over, sort of, the concerns that an organization like The American Civil Liberties Union has about facial recognition and other biometric surveillance, and then talk about two ground-breaking laws that the ACLU has led coalitions to pass this year, here, in California And in doing so, I want to persuade this audience, and folks watching online, to support a strong, restrictive measures, like moratorium, or bans on this technology And that’s for really three simple reasons to sort of outline where I’m gonna go in this talk The first reason is that facial recognition poses an unprecedented threat to civil rights and civil liberties This is the technology that gives the government power it’s never had to track people, to identify people, and to effectively control people with automated surveillance The second thing is that, because this is so powerful and unprecedented, we think it’s very radical and dangerous for us to be allowing governments and companies, for that matter behind them, to be testing facial recognition, affect recognition, and biometric surveillance, on the public right now, which is what is happening in many places where we lack any sort of regulation And then finally, it turns out that policy makers and the public strongly support bans and moratorium on facial recognition and other biometric surveillance and they’re passing them So this is where the energy is right now and I’m gonna talk a little more about that as we go forward So, the slide clicker is Early last year, I was on Amazon’s website for their cloud services product, AWS As one would do, I was browsing in AWS’ website, and I noticed they had a product called Rekognition with a K So, that seemed interesting I didn’t know Amazon had a facial recognition product and all throughout their page there, they were quietly, sort of hidden in plain sight advertising a product that they said could identify and track hundreds of faces in a still image or in a video that could track, quote, “persons of interest,” and that on the website, they also talked about their ability to, they touted a use case that was to add this to officer-worn body cameras That was the first use case, if you think about software makers, they hold up sort of your primary use cases This was right at the top And that rang an alarm bell for us because we have been working with our coalition partners to make sure that body cameras are deployed fairly and that they do their job that they were promised to do which is to hold officers accountable and make them transparent for the public Yet, Amazon, on their website, was advertising the use of body cameras for surveillance And so that just showed to us, it suggested a misunderstanding of how, one, policing worked at the local level, and two, how surveillance could impact communities that the police are supposed to protect These are devices that were promised to make police more accountable in the wake of well-publicized shootings of black and brown men across the United States, and so seeing that set off alarm bells for us at the ACLU So we worked with two affiliates of the ACLU in Oregon and in Florida We crafted and we sent public records requests It’s something that all people can do You don’t have to be a lawyer to do this We sent it to Orlando Police Department and a prominent sheriff in Washington state or in Oregon, excuse me, that we knew to be using this software, because it was listed on Amazon’s website as a customer They were listed as a customer on Amazon’s website So we got back a lot of records and we were surprised to learn that Amazon wasn’t only providing this technology, but that it was providing technical support, it was providing free engineering resources, it was even soliciting ideas for new product features and inviting a person who was sort of a prominent user to Amazon’s conference in Las Vegas, where they could talk to other potential customers And what also striking about this was that, and it is, very cheap So for the cost of a large cheese pizza, a law enforcement agency now can upload hundreds of thousands of images of mugshots or DMV photos or whatever sort of, source photo they have, create face templates, and a matching database that can be used to identify persons, or attempt to identify persons as they’re arrested, but more concerningly,

or a sort of separate concern, is that that sort of data set in that system can be turned and applied to body cameras roaming around our streets It could then be applied to public-facing video cameras that are increasingly in our neighborhoods and that have higher and higher quality video footage and that could allow for the establishment of more sort of, eyes watching us as we go down the street So, the cost matters in part because, you know, the limits on government surveillance, some of those limits have traditionally relied on the cost of surveillance It was something that was sort of infeasible 20 years ago for law enforcement to watch where you go throughout your day or week continuously, they would have had to devote personnel resources, potentially, a van and equipment to follow you and there would have to be a sort of, investment of resources to track where you go Facial recognition helps to eliminate that friction So, I just want to also highlight this, which was in the records we’ve received from one of the law enforcement customers I think this is funny because the internal email chain and the voice of this sheriff’s employees, well they were actually correct The ACLU was concerned and we started a public campaign to demand that Amazon stop providing facial recognition to government entities after that So, yesterday Joy talked about the bias and inaccuracy in facial recognition technology and it’s true that inaccurate systems will result in false arrests, they’ll result in encounters between law enforcement and the government and the public that should not occur, and they will result in the wrongful death of people in our society and in our communities But, even perfectly accurate facial recognition systems pose a uniquely dangerous threat to our democratic system and to our civil rights and liberties So, facial recognition, we like to call it face surveillance because that more accurately conveys what this actually functions as in public spaces That means the power to create a profile of somebody, to identify them without lifting a human finger, to create a persistent record of people’s movements across town, their interests, their potential emotional state or at least their perceived emotional state, there are many products now that purport to identify things about our interior, simply from exterior features, which is snake oil at best in many cases, and very dangerous at worst So, what else does it mean? In this world, if you have facial recognition in a community, that means the government has the power to know who you are without asking for your ID That means the government has the ability to know where you worship and who you worship with without visiting your church That means the government has the ability to log and create a roster of you and your political associates, whether you attend the Women’s March or a march for The Dreamers, and know who you were there with and when you were there, all without infiltrating your political associates or group That’s not just a creepy set of powers that the government gets with facial recognition, as I was saying earlier, it’s very unprecedented in many ways And of course, with the ability to impose more surveillance means the ability to impose more political control and social control This is a screenshot, as was the last slide, from a great series The New York Times has done on facial recognition showing how it’s being used in China I wanna point folks to this slide from one of the features they did, which I think really illustrates something called the chilling effect that facial recognition has It’s not only the government’s ability to know more about your private life, but it’s the government’s ability to impact your ability to freely express yourself, to freely associate, to freely exercise your own religion And so, in this story, the author talks about how after the Chinese government set up facial recognition in a city square, or town square, where they had previously been prayer happening at all times of day, lots of civic activity, after the system of facial recognition was established, the author comes back and he sees, you know, a desolate city square where no one wants to go to Now China obviously has a different system of government and they are, in many ways, much further ahead on their use of face surveillance in public spaces, but the chilling effect that is occurring in this sort of situation right here, is not going to be, that’s not uniquely a Chinese thing In the United States, under the first amendment, one of the main sort of principles or concerns motivating our first amendment protections is this idea that when people are watched, they change their behavior When people are watched, they feel intimidated and that is exactly what will happen if we establish facial recognition in public spaces Many of us will feel less comfortable going to a public protest if we know the cameras there,

whether they’re on officers or on streetlights, are going to potentially log who we are, whether we’re there, and who we’re there with, and that’s going to be true even if we have laws in place that say, okay here’s a complex bureaucratic process the government has to go through before they can use facial recognition, they have to mark X, Y, and Z boxes That chilling effect, I would argue and the ACLU would argue, will exist even if that bureaucratic procedure is in place behind the cameras, because what you will have are cameras that were previously asleep, right now when we walk down the street, we don’t think that camera is actively watching me, we’re gonna have cameras that were previously asleep coming awake in a new way, and that is going to have an impact on our first amendment rights And in many communities, it will be safe to assume the worst, that people are actually watching, that government agents are actually watching, because the history of policing in the United States is the history of racist policing It’s a history of policing being disproportionately aimed and policing resources being disproportionately aimed at communities of color, at immigrant communities, and at activists who seek to speak truth to power So, how do we know this? Well, these systems are vulnerable to mission creep, is one thing we know as well So what does that mean? That means when we build a facial recognition system, we create a system that is vulnerable to expansion This is because, in part, to create a minimally viable facial recognition system, you need a large set of matching photos in order to identify a person who’s arrested when they come into the city jail You need plenty of photos of identified people in order to find a match Well with facial recognition, a few lines of code, a few cameras can be established and added to your system, and suddenly, you have a city-wide surveillance network The friction to creating and expanding these systems, is something we like to call, mission creep, that friction is even lower in the facial recognition context than it would have been, say in the context of buying drones, where you have to spend a few thousand dollars each time you buy a new drone to expand that network How do we know that this sort of system will be abused? History tells us that facial recognition and other sorts of surveillance have and will be abused So this first example is from a set of records that the ACLU released earlier this year and what this was is an examination of license plate readers These are technology that American Law Enforcement have been using for over 10 years now They look at license plates, they snap a picture and tag the car with a GPS, many cop cars have these and streetlights, and over time, the collection of this data allows for the collector to know more and more about the locations of drivers The ACLU and many actors in civil society have been sounding the alarm about license plate readers for many years now But then last year, early last year, we learned through records requests and reporting by The Verge that license plate reader systems, that were previously pitched as a way to find amber alert vehicles, to find stolen vehicles, they had expanded their use, and suddenly, ICE had purchased a subscription to one of the largest databases of driver locations in the United States And so with that, ICE gained a new ability to target, locate, and deport immigrants living in our communities We obtained records showing that ICE was talking about somebody’s habits They were searching the system to find out somebody’s habits of where they went to work, where they went home We saw the tracking of the associates and family members of immigrants using these systems and these systems, this ALPR system, license plate reader system, was originally pitched for something completely different Over time, the system grew, and the new uses were found, and profitable ones at that We’ve also seen it in the facial recognition context already So, in 2016, the ACLU, we obtained records showing the use of social media monitoring software, so it’s a software that scrapes social media, and then provides this sort of Google Maps sort of interface to law enforcement so they can conduct investigations We uncovered the prominent use of this software and in some of the records, we found that there was a use of facial recognition Law enforcement in Baltimore pulled images from social media, they ran facial recognition against those images, and then they went and arrested people who were protesting at the Freddie Gray protest in Baltimore So this is a protest following the police killing of a black man Law enforcement used facial recognition to target protesters at that event and to arrest people And so, we don’t need to speculate, we don’t need to theorize, to understand the sort of, writing on the wall when it comes to what happens when we build sophisticated surveillance systems and we let them grow and we let them be deployed

against our communities So, the problems in facial recognition are not just limited to the technical problems that Joy discussed yesterday, you have your inaccuracies in prominent products, there are bias issues in many prominent products We can point to the problems in facial recognition even if it’s perfectly accurate I talked about the privacy issues, the free of speech and first amendment issues that exist even if these technologies are perfected, even if they’re perfectly accurate Another set of problems that exist, even if the technologies get to a perfect state, is misuse by humans Facial recognition like any other surveillance technology is deployed in an imperfect world Even perfect facial recognition is going to be deployed in an imperfect world We see this in Florida where there’s a case brought by a man named Willie Lynch The ACLU and the Electronic Frontier Foundation have filed a brief on his behalf because he was not told about the use of facial recognition in his case to identify him when there lacked an eye witness Until just before his pre-trial hearing, he was not provided crucial information about the use of the facial recognition technology He was not provided the other set of possible matches that were given to law enforcement before they went and arrested him and placed him in prison for eight years, and he wasn’t provided the opportunity to scrutinize the algorithm, to cross-examine the law enforcement agent who operated the system, to ask questions about whether they were even trained properly So, again, perfectly accurate facial recognition and I’m not assuming this particular product was, but even if we say this is perfectly accurate, there are tons of problems there Another situation in records uncovered by the Georgetown Center for Privacy and Technology Earlier this year, they released a report showing law enforcement simply feeding in the wrong data into facial recognition systems So in one instance, the NYPD, not being able to find a match based on the blurry image on the left, gave the facial recognition system a picture of Woody Harrelson who they, in there sort of, guesswork, decided was the closest-looking person to the person in the blurry image Facial recognition systems compare pixels, and so, what the facial recognition told them was not based on the sort of human analysis of well does this person look alike? How much do they look alike? In fact, they just took the results, and then they went and arrested that person So, what are we doing to fight back and to push back? You know, policymakers and the public really do understand these threats that I’m discussing right now Facial recognition resonates with the public and policymakers in the way I’ve never seen a surveillance system resonate with the public or policymakers in my seven years at the ACLU So we know this is resonating, like I said earlier, because of two recent campaigns that the ACLU has lead here in California So, in San Francisco, we put together and led a coalition of 25 civil rights groups to pass what is the first ban on the government use of facial recognition by governments in the United States or perhaps the world Inside of city hall, I think this is a really important point when we talk about this sort of, where regulation should fall Inside of city hall, the ban was uncontroversial What was controversial was this idea that policy makers, or excuse me, police and manufacturers themselves, acting alone, should be the ones to decide whether a new surveillance technology is deployed against the public That’s radical What is reasonable and makes complete sense to policymakers is the idea that the community should decide and residents should decide whether or not a new surveillance technology is deployed against them and what the sort of proper public safety framework and solution should be That argument, that sort of frame resonated with policymakers and this passed by a 10 to one vote It was nearly unanimous in San Francisco and there were hardly any sort of public discussions about the sort of, ways to weaken the facial recognition ban or anything like that It was a very sort of, I think consensus based approach to this regulation And so, that has triggered a nationwide movement in many ways, Oakland and Berkeley have passed their own bans, there’s a ban passed in Somerville, Massachusetts, led by our allies and partners at the ACLU of Massachusetts, and there are bans proposed now in Portland, Oregon, we’ve heard potential bans in Southern California, in New Jersey, and other cities in Massachusetts So, building on the momentum from the San Francisco ban, we also led a coalition to pass a statewide law this year that addresses a sort of pressing issue in the facial recognition and biometric surveillance space So, again I talked about body cameras and how they were promised to communities for keeping officers and police, officers and sheriffs transparent

and accountable to communities But those cameras are at risk, in many places, of being turned into surveillance devices roving through our communities And we had seen chatter from prominent vendors talking about the potential addition of facial recognition and other biometric surveillance to these cameras We had seen patents filed, and so, we banded together with the coalition of partners across California working on an immigrant’s rights issues, working on behalf of the unhoused, working on behalf of criminal defendants, and we brought a bill forward that said law enforcement in California may not use biometric surveillance which includes facial recognition, affect recognition, voice recognition, recognition of the unique way you walk, they may not use that in conjunction with officer cameras, so the cameras attached to them or the cameras they carry This bill was just signed into law by Gavin Newsom three weeks ago And so, it goes effective on January 1st, and that would make California the largest state yet to pass this sort of measure, arguably the largest market for officer-worn cameras, and so we’re really excited to see it made effective here in California And so, I’ve taken a bunch of time I apologize, but I just wanna conclude by making the point that I think is often lost on those of us working in this space, which is that this technology, whether it’s facial recognition or another sort of surveillance technology being deployed or pitched in our communities, it is not inevitable that facial recognition has to be deployed, simply because the technology has improved We as a Democratic society, not technology companies, not police, acting alone, we’re the ones who get to decide whether and when these technologies are ever going to be deployed against us And I think what you saw in San Francisco and in other local communities as well as California really does show that democratic policymakers or democratically-elected policymakers I should say, because these were bipartisan efforts are listening, they’re hearing, and they’re excited to bring forth legislation that draws a line in the sand and at a minimum presses pause on the use of these dangerous new technologies And just to close, I think I wanna highlight that none of this would have happened if we had not banded together with allies who don’t traditionally work on privacy issues to work on these legislations In San Francisco, at the California State House, we brought together coalitions, like I said, immigrants’ right, racial justice, the unhoused, Muslim-Americans, communities who could stand up before policymakers and explain the impacts of historical surveillance and how it harms people That resonated with policymakers in a really meaningful way, because on one side we had sort of theoretical benefits of how infinitesimally, rare situation facial recognition could help address potential crime On the other side of that, we had actual people who talked about what happens when law enforcement infiltrates a mosque and uses a license plate reader to scan everyone who’s parked at the mosque Or people of color who could talk about the use of social media surveillance to infiltrate Black Lives Matter and seek to shut down those movements Those voices carry the day and they are why California right now is really leading on this legislation and this sort of topic So, thanks for having me, and I look forward to the discussion (audience clapping) – Thank you, Matt So Matt has grounded our conversation in facial recognition technology, thank you Our next panelists are gonna expand our thinking on this To think about biometrics and other ways in recognition more broadly So first up, Dr. Bridget Algee-Hewitt is a senior research scientist at CCSRE, Center for Comparative Studies in Race and Ethnicity, and a biological anthropologist with over 20 years of experience in social justice work in the U.S. and abroad She merges machine learning, data modeling, and hands on laboratory approaches for the estimation of the personal identity parameters, like sex, ancestry, stature, and age, that are essential components of the biological profile used in forensic identification of unknown human remains She also applies cultural anthropology and critical race theory to understand how biology and culture interact to shape personal or group identity and reflect individual experience Dr. Bridget Algee-Hewitt (audience clapping) – Good afternoon First, I’d like to thank HAI for having us here today

and I’d like to thank Daniel and the center for inviting me to speak I have to also say how excited I am to have such a diverse panel I think we talk in academia a lot about the value of interdisciplinarity and I think more than ever dealing with issues like facial recognition where it involves those who are the academics, who are developing the methods, those who are the computer programmers who occupy that liminal space between academia and industry, and then those who are the policymakers need to be all agents of discussion and change and development as we move through the thorny issues and come to an understanding of what we don’t know and then how we ultimately deal with what we do know once we reach that point, in terms of regulation, policy, and moving forward So this is, I think, a very exciting letter-shed moment to have all of us here to discuss collaboratively, drawing upon our strengths as policy and academic professionals So, Daniel asked me to give an overview of my work so that in case you’re not all familiar with the minority of us who do this kind of research So, I have a multi-pronged research program as Daniel said I’m a biological anthropologist, which means I study what it is to be human, biologically and culturally, and I use that information distilling it down to make identifications in the individual So we look at large, worldwide patterns, develop mathematical models for understanding those patterns and then use those models on the micro-scale for coming to make an identification of the individual So that means that a large part of my work is situated within the field of computational biology I cover both the skeletal, genetic, and the phenotypic areas of what it is to be a person, and I combine those with demographic information, so how we see ourselves within society, so what our labeling systems are, and also those numbers within society that help us situate the math of doing identifications, so population density, issues of diversity, are all very important when we think about developing models And then as I said, I look at that from the global perspective to understand how we vary We’re all different in terms of populations is what we talk about as anthropologists, which means we have differences that are often geographically located But we also know that in our genetic, in our phenotypic, those are what make up how we look, or what is expressed by our genes, and in our skeletal system We have commonalities and those commonalities form clines, so we have distributions, so while we’re all very unique, and then combined as populations that cover the globe, globally we’re all very similar as well, so teasing apart those different levels of identity is important and that’s how I’ve come to AI and machine learning because it gives me a very powerful tool for developing much more nuanced models that allow for that type of analysis to happen And then, in addition to doing the mathematical work on that more theoretical perspective dealing with lots of big data sets that have been collected over long periods of time, I also deal with the immediacy of the identification problem So, I merge what we know from human variation and from these large data sets that cover that swath of diversity with what we can gleam from individuals So what we study in the lab when we’re given a forensic case so that micro-analysis and the goal there is to use these models, or develop better models that take that information that we have in the laboratory and we merge them together so that we can make a positive identification for an individual for whom we have nothing but a DNA sample, for example, or their skeleton So it’s to bring social justice to the identification process So, the ultimate goal as Daniel said, is to estimate those identity parameters so that we can say from all that remains of your biology, the things that would have been important to you in life

or socially important in life and the things that then would have then been reported on a missing persons report, or told to law enforcement, or a NGO if you’re working in a humanitarian context, about you So I always tell my students that we like to think, we wanna relay what’s on your drivers license We wanna say all those phenotypic and demographic information that may help define you as who you are so that we can take these unknown remains and match it up with the known individual and ultimately serve social justice by returning those remains to the family So, I also then work in the medical legal system, I’ve taught and aided law enforcement in understanding how this process works and being able to produce the reports that can aid them in their investigation, and then I also serve as an expert witness, especially in asylum cases and in the documentation of trauma and humanitarian aid contexts And then, as hopefully you’ve gleamed as I’ve been talking, social justice runs hugely through my work and is a great motivator for why we do these kinds of analysis, model building, it’s what inspired development that applies AI and ML technologies So, I apply cultural anthropology and critical race theory, as Daniel said, to understand that intersection between biology and culture, because if I want to speak to who you are and how your family or your friends would have defined you, then I need to speak to all of you, and as a person, you’re both your biology and you are shaped by the culture in which you exist So important to that, then, is understanding that interaction between what is innate in your skeleton or in your DNA that reflects your ancestry and how that can be translated into more social categories, such as geographic categories for populations, or more commonly, what gets translated into race and ethnicity So there’s always that thorny allusion between biology and culture and its something that needs to be probed deeply and treaded upon lightly And my actual research integrates a lot of these complexities of dealing with biology and culture and also geo-politics as I work on the identification models for improving the work of those people who are on the border The boots on the ground, very amazing people who I work with in Arizona who generously afford me the data and the opportunity to work with them in their medical examiner offices, on the identification of missing migrants who have made the perilous journey from Central America and Mexico and have perished only at the point at which they’ve reached the border In that, since then I work with refugee populations and I also work with the living It’s important to be able to understand the patterns of the dead We need to understand the patterns of the living So, being able to have the opportunity to understand migration movements and how people self-define and trends that shift in that self-definition and in those migration patterns over time are critical for me to build models that accurately reflect the people who are crossing and dying on the border And then, more recently, I have work that is also focusing acutely on social justice concerns and how geopolitical situations actually change our biology by causing trauma and that trauma is played out in our epigenetics and those traumatic events and those consequences of those events can be passed on inter-generationally So understanding how, what happens to us culturally affects our biology is huge in terms of being able to create models that allow us to serve social justice in the best way So then how did I end up on a panel that deals with facial recognition? Because I haven’t mentioned that yet in anything that I do My specialty is actually the estimation of ancestry and ancestry is something that reflects largely continental variation that goes back to our evolutionary history, and the skeleton is a really great source of population history information

because while we do have heritable traits in other parts of our body, the skeleton is very, and the cranium in particular, is highly heritable, which means that it is very stable, so it reflects neutral genetic variation, which is essentially what we need to study in our models to be able to estimate ancestry So the cranium, as I said, retains that kind of ancestry information very well And how do we study the cranium? We take measurements, and we’re taking measurements of the underlying structures for which facial recognition is ultimately built upon So, the sort of fundamentals of biometrics are similar in terms of facial recognition and ancestry estimation from the cranium So the processes, the mathematics of it are complementary and the issues that surround it, the problems of partitioning people up by labels and imposing estimates of what we think someone’s biology tells us are equally problematic when you’re using facial recognition for estimating someone based on their outward appearance, so not to make a match, but to estimate someone’s skin color and then extrapolate from that what the racial identity would be The same issues can arise when we try to make that allusion between ancestry and race, which is problematic and complex So we share in some of these same struggles as a field as we move forward So then as I said, craniometrics then are a form of biometrics, and we need to be cautious as we approach biometrics from a facial recognition perspective looking back to the long and very torturous history of how craniometrics have been used in science So we need to need to think of Blumenbach and phrenology and the racialization of immigrants using measurements of the body, and in particular, the cranium So we can learn a lot as we move forward with what is a new technology by looking backwards at something that has been around a long time and people like myself are hopefully doing a much better job at bringing this established history of anthropology and biometrics forward into something that does good work in promoting social justice So then, as I said, like biometrics, craniometrics have had that fraught relationship with race, but they’re also like biometrics, in the facial recognition realm, used for individual identification So we can take measurements of the cranium and aspects of the craniofacial shape and we can match them between the unknown individual for which we have that source of information whether it’s a radiograph, a CT scan, or a collection of measurements, photography is also very common, and then match that to a record that’s on file for a potential match, and you can see on the screen that we have an image of the skull and then a photograph and we use photogrammatographic superimposition as a way to identifying corresponding points and by the number of those congruent points, we can determine whether or not that’s most likely that individual So think of it the same way as a DNA match is made, a K unknown to a Q to a question, or a fingerprint, where you’re taking a known and an unknown print and you’re looking for concordance in the shapes So then, I assume that as I’ve been talking, in the backs of your minds, you’re hearing my concerns for ethical issues, how we hold ourselves to account and how we hold the academic community and the larger industry to account in terms of doing good science So, when Daniel invited me, he asked me to think about my grand challenge, and without question, the grand challenge that came to mind to me was ethical How we deal with the ethics of doing scientific research and making sure that that work is being used for the best purposes So, I thought then today, I would go through and choose a few of the ethical issues that I think, together,

create a large grand challenge and we can hopefully deeper dive into some of these as we move into the discussion So first and foremost, I think it’s critical that we are critical of ourselves That we ask, as scientists, what our responsibility is for actually doing the work When do we call halt and say that, as scientists, we are developing things that may be misused, abused, and dangerous for people? When do we accept that we only do the science and that is enough? Or we need to recognize that we do the science and that science is disseminated, and therefore, that trail back to us is always there So, as the progenitors, we have a responsibility to determine either collectively, as a university, of what we accept and what we don’t accept in terms of breaking technological bounds with the risks that come with that, or what we believe to be the ultimate gains by advancing science and what infrastructure or what systems we have in place that will be the checks and balances for us So, as an example, and harkening back to what Joy spoke about yesterday, that we have in facial recognition then, the issue of identification based on skin color, so that the successful identification of darker faces suggests higher rates of misidentification But, what happens when we get better at it? So, ultimately we wanna develop better technologies, right? So we do much better as scientists, then, at improving the technology and we can now, not only, identify lighter skins, but we identify darker skins with equal probability Where does that leave us? It then leaves us with another technology that can potentially be used to marginalize, disenfranchise, and cause harm to people of color So there’s a push towards a scientist for improving what we do to do better science, but in that act of doing better science, are we doing more bad than good? And where do we draw the line in terms of what is the limit that we should place for responsibility on us as scientists and the responsibility falls then on policymakers or does it fall on technologists who distribute the software that enables this to happen? So there are many levels of discussions that need to occur That brings me to the dangers of dissemination So, we have these technologies, we produce them within the cloistered environment of academia where it’s safe, it’s safe to be radical thinkers, but then those radical technologies are transmitted into the public, and who is the safeguard, or the safety check, or the educator of the public Well we educate our students and we hope that then they go out and in their jobs, we’ve given them a good foundation to be able to transmit quality information But, we have to think about the role that these types of technologies play within industry So ultimately, when something is distributed sufficiently, it’s sufficiently popular, you have lots of technology, lots of competing companies producing softwares Then, you have it distributed widely Do we always have the best education? Are the people who are employing these technologies always the best trained? You can have the optimal technology that makes very few errors itself, and then it has a very easy application threshold So that means an analyst is gonna do a pretty good job at applying it, but then there’s always that issue of interpretation What weight do we put on the information that we get from it? So this is nothing to do with the software, nothing to do with the algorithm, but it’s in what do we do with that information afterwards? And there’s a lot of literature right now that looks at standards for how law enforcement are trained and using these kinds of software, and there’s not a lot of evidence or sharing in terms of the types of training, what the approaches are to the training, and how these vary between law enforcement offices, between states, between countries So, the idea that these softwares are being distributed,

and then we have the public as well, who is maybe not as best informed about what they do, creates a lot of confusion and chaos So it’s critical that we not only educate the people who are using this software, we need to educate those who are consuming that information from the software, and then we also have to have the public aware of what’s going on in terms of the software use, what that actually means Because ultimately, they are serving as the jurors when law enforcement cases are brought to court And, as an example, Rashida Tlaib, congress spokeswoman, was at a law enforcement office, and she made the comment that’s quoted up here, and while her intentions are correct, that there is a bias in the process and the software and the way that it’s often used, she’s placing that issue on the actual analyst without a larger understanding of the whole mechanism This suggests that, as she is a representative of the people, and yet she is maybe not as informed as we would like her to be, and that is incumbent upon us as academics to do a much better job to teach people It’s incumbent upon the industry to educate more So, we need to make sure that we are investing in not just production of tools, or safeguarding the use of the tools, but also in the education around them This also brings me, as the mathematical person to geek out for a moment, that we also need to ask mathematically or computationally or probabilistically, what it actually means to get a match? Sometimes you can get matches and it doesn’t mean they’re useful So we have to weigh that risk against what the ultimate reward is So, think about the fact that this software does its absolute best job on people of light skin, and we know that the US is populated mostly by people who self-identify as white, who we would extrapolate generally have light skin So that means, it does really well on the majority of the population So then if you make an identification of someone who is white, based on the facial recognition, making that identification match, what does that actually mean? Probability-wise, the evidentiary value means very little So now we’ve created this technology that allows us to do these amazing things, but what it actually outputs doesn’t actually give us anything that improves a case, in terms of advancing social justice of the judicial system So, there’s an important balance that needs to be hit between doing technology for improvement, and does that improvement actually improve things? So to be thinking constantly, critically And then, speaking to the issue of the court, we need to ask, are we at risk of a new CSI effect? We need to be thinking about how jurors, or the public who are our future jurors, consume this information So, as someone who has testified in court, in terms of DNA evidence, we know that there’s evidence to suggest that jurors now, on account of exposure to scientific technologies, have expectations of what will be delivered in a court room What they expect that law enforcement should have done to demonstrate that they did due diligence, that the case is believable, that there’s adequate evidence So, if we are presenting this technologies without teaching the public that there are problems with them, then we can run into the risk that they become an expectation of what is good science and what is a good court case So we need to be always cognizant of how these are being consumed in the public sphere in terms of perception Often as a academics, we think of, or as scientists in particular, we think of numbers Statistically significant Do we have high probabilities? Without also taking into account that perception often trumps numbers How we think, or believe, or what we feel we innately know is sometimes way more important in convincing people than what a statistic does

And then lastly, we run also the risk of what I’ll call the compound technology effect, where we have surveillance technologies that then get combined and we are regulating them independently, but what happens when we start creating systems that pool all of these technologies together So, as an example, there’s this project called The Face of Litter in China It analyzes and collects DNA profiles of litterers using an American software program called Snapshot DNA from which they use forensic phenotyping, which is to take your DNA and to try to recreate what you look like physically, just from a DNA sample, and they combine that with demographic information, so who they expect lives in a particular area and what they know about the habits of that particular area in Hong Kong And, that means that they’re one step towards then being able to reconstruct that individual’s whole person a picture essentially of that person based on their DNA sample that they’re getting from their litter, and then, they’ve taken those pictures and they’ve put them up on billboards So now, anyone walking around has pictures to view of what this potential litterer looks like, and then, if you scroll down to my slide, you’ll see that there’s evidence where police officers are wearing facial recognition glasses to identify known criminals So now we have pictures of known litterers, translate that to criminals, and then we have someone who has a database in their glasses of known criminals walking around waiting to identify them So, we have this combination of DNA technology and facial recognition technology that’s coming together to create this mass system of surveillance And while we talk about China in sort of in the abstract, as some place far that doing these extremes things in terms of surveillance, we are still producing the technologies that are going into supporting this, so we are ourselves still invested in this process, we may not be doing it, but we are not that far from having to grapple with these issues, and these are issues then that we should be grappling with, with the recognition that it could be us some day So, we need to be thinking forward And I will leave it there, thanks to you for your attention and to Daniel for the opportunity (audience clapping) – Thank you Bridget, definitely raised some important challenges for us to discuss in a moment here! But before we get there, I’d like to introduce our last panelist Dr. Wendy Hui Kyong Chun is Simon Fraser University’s Canada 150 Research Chair in new media in the school of communication She’s studied both Systems Design Engineering and English literature, which she combines and mutates in her current work on digital media She’s author of a number of books including, “Control and Freedom,” “Power in Paranoia in the Age of Fiber Optics,” and co-author of “Pattern Discrimination.” She’s been a visiting professor at AI Now at NYU and currently serves on their academic advisory board, and a visiting associate professor in the History of Science department at Harvard of which she is an associate Dr. Wendy Chun (audience clapping) – Hello, everybody, it’s been a very intense two days, and it’s really lovely to be here, thank you Daniel and Jennifer for the invitation, and HAI for bringing us all together So today I’m gonna talk really broadly about the question of race, rights, and recognition by thinking through one of my current projects on discriminating data, and as Daniel mentioned, this project emerges from my new group at SFU called the Digital Democracies Group, which was founded by a Canada 150 Research Chair So this, for all of you who don’t know about the Canada 150 Research Chair Program, was basically Canada deciding, now is the time to try to poach foreign academics and get them to come to Canada And so, what we’re doing is setting up a really

multidisciplinary group, so we’re bringing together research in the humanities, the social sciences, engineering, and data sciences, in order to address issues of equality and democratic dialogue And we’re really unusually interdisciplinary, so we’re working with a global network of network and data scientists, quantitative political scientists, media theorists, political theorists, artists, dramaturges, as well as computer programmers We’re also working across sectors, we’re partnering with some researchers at Intel as well as non-profits, such as Social Science Research Council And we’re working across these discipline and sectors as many of you people have been talking about today because we have to Because in order to take on the really tough problems that face us, we need to be really unusual and a little unholy in our alliances, and so what we wanna do is help counter the proliferation of equichambers, abusive language, discriminatory algorithms, and miss and disinformation, and I’m happy to talk to you about more about the lab and our other projects during the Q and A So, discriminating data explores how algorithms currently encode legacies of segregation, eugenics, and multiculturalism So, it looks at the way in which identity categories, such as race, gender, and class, mutate and persist through algorithms that are allegedly blind to them and are arguably designed to blind us to them So how do categories such as gender, sexuality, race, and class persist and mutate through as many others, including Oscar Gandy many, many, years ago, have shown through proxies that launder discrimination, and this is taken from the Marshall Project, and they’ve shown how categories such as, doesn’t have a high school diploma, which is used to determine the risk of recidivism and thus sentencing in parole in programs such as Compass inscribe race and class in an intersectional manner What’s interesting is their getting at that intersectionality of race and class These categories are also inscribed through latent factors, or principal components, that discover or naturalize sexual difference, so here’s one finding from the notorious Gaydar AI, which produce prototypical gay and straight faces by analyzing snapshots from an online dating site There’s a ton to say about this, which I won’t say right now, but what’s fascinating is that they used machine learning to discover and naturalize the sexual politics of facial hair And these categories are also reinforced, as I’ll discuss in more detail, through network neighborhoods that are premised on and that produce segregation So, social networks are based on the notion of homophily, the idea that birds of a feather flock together, right And so you’re clustered into network neighborhoods based on how you deviate from the norm like others And so, echo chambers aren’t a side effect, they’re the goal So, discriminating data follows the following five-step program The first is to expose and investigate how ignoring differences amplifies discrimination both currently and historically So how crossing out race actually perpetuates racism So how dreams of cyberspace and AI is solving the problem of human discrimination actually perpetuates discrimination And there you see the article sort of introducing Compass in the Washington Post as something that would actually dissolve the problem of discrimination in courts, all right Also, looking at how dreams of machines as our slaves or servants, so self-driving cars and chauffeurs, recommendation engines as everybody’s butler drives fears of the coming singularity, and Jenny Reed has written a great book sort of documenting the intersection of race, class, and gender, with robotics And so, one thing I’ve been thinking through is the dangers of hopeful ignorance, the repeated dangers of this hopeful ignorance Step two is to interrogate the default assumptions

and axioms that ground algorithms and data structures So homophily, again, underlies collaborative filtering So collaborative filtering groups, users, and objects into neighborhoods based on boundary defining likes and dislikes And not surprisingly, U.S residential segregation is often given as evidence that homophily really exists, right, and here you see an excerpt from Easley and Kleinburg’s really excellent textbook on network science But let’s ask ourselves, under what conditions and why has this become a default, right? Because the ties between homophily and residential segregation are deep and profound So, homophily was actually first coined as a term in 1954 by sociologists Lazarsfeld and Merton, and they also coined the term herterophily, which is something that’s been completely erased, but it’s also clear that this is one of the most cited articles in the literature, but also clearly, one of the least read Not only did they also cite heterophily, at the heart of this study, is actually a study of two housing projects One which was a white only co-op in New Jersey, and another which was a biracial yet segregated housing project in Pittsburgh, and they didn’t assume homophily to be axiomatic, but they rather asked what other dynamic processes through which similarity or opposition of values shape the formation, maintenance, and disruption of close friendships, and what’s important to remember is that they were working in the late ’40s This was a time of great income equality It was also a time when there was 80% support within the U.S. public for public housing It was also a time when public housing was predominantly white, and what they noted in their studies of these projects is that homophily actually wasn’t the norm, and what they argued, though, was two circumstances in which you did see status homophily, which was race and gender This was actually due to what they called value homophily, or the sharing of values And to do this, and to prove this, they concentrated on the over-selection of white liberals and white illiberals, of friends, where friend was considered to be one of your three closest friends Now, to make this point, as you see here, they actually threw out the responses of all the black residents, on the argument that they were all mainly liberal So there was no idea ever of putting the black and white responses together, right So, liberal meant that you believed in integrated housing and you thought that the races got along in the housing project Illiberals thought the opposite, and ambivalents thought that housing projects should not be integrated, but the races did get along in the project So, a prior racial segregation, grounds homophily, and in their subsequent analysis, a model of friendship formation, they also ignored the white ambivalence who actually constituted the largest category of white residents So they threw out most of their data and they openly speculated So here at the heart of functionalist sociology is this wonderful acknowledgment of fiction, in terms of thinking through and institutionalizing homophily, and what’s also interesting is that they never published the data from this survey, because if you did as you see here, and we’ve gone through the archive, the over-selection of illiberals for other illiberals is the difference of two people And so, what’s important is that they also have this incredible other data So, they also asked people do you have acquaintances of the other race? Do you have friends of the other race? And based on those questions, the majority of people said yes So the data was there to support a very different vision, and it’s not just homophily we need to rethink, but it’s also correlation So the early 21st century celebration of correlation isn’t new, right, and here you see two quotations taken,

one, from the beginning of the 21st century, one, at the beginning of the 20th century, declaring how correlation was going to change the face of science And the piercing correlation coefficient, which is used in many of these algorithms to determine similarity and linear regression, were actually developed to prove the truth of eugenics So they’re developed as tools to understand human heredity, and so Francis Galton, who was the father of eugenics, developed linear regression, which he first called liner reversion, while studying two problems One was the identification of criminals, and the other one was to understand the transmission of human heredity So he wanted to understand how genius was or wasn’t transmitted culturally, biologically, amongst his schoolmates in Oxbridge And Galton was a biometrician rather than a Mendelian, so he believed that traits followed a normal curve rather than Mendelian genetics So exceptions, such as genius were statistical outliers, right, so they were located at the ends of the curve, and so, Galton developed linear regression, which is now used to determine the best fitting line between scattered points in order to understand the transmission of deviation And so this is very different from how we understand linear regression now, right The point of linear regression is to minimize deviation Come up with a line that minimizes standard deviation, but what Galton wanted to do was try to understand how to maximize deviation So he formulated the notion of linear reversion by studying the deviation in heights between people of exceptional height and their offspring, and what he did was draw a line and try to figure out a slope that linked their deviations, and what he showed was that deviation, unless carefully bred, decreased over time So according to Galton, his graph showed that we were regressing towards an ancestral mean, and so the point was to constantly stir the system so that deviation would be produced And it’s not just linear regression, it’s also SVM But to be clear, my point is not that any use of these techniques are eugenicist Clearly statistics and proxies are important to understanding things, such as global climate change, but my point rather, is that if the world now seems closed, it’s because these methods focused on breeding a human future based on an unchanging past, so in this rule, there was no nurture, only nature Intelligence, Pearson famously argued, couldn’t be learned, it could only be bred So the future could be calculated and had to be calculated, but it couldn’t be radically different from the past So if you think about it, it’s really perverse, that a system that believed that learning was impossible has become the basis for machine learning Which leads me to step three, to apprehend the future machine-learning algorithms put in place, focusing on when, why, and how the predictions work So what world are we living in, that these predictions seem to make sense, and what sort of role do they put in place? So again, in terms of regression and correlation, how did a system that denied the possibility of learning, again intelligence could not be learned, only bred, become the basis for machine learning? And how did this system that was determined to shut down disruption, right, so a system designed to shut down the future, to make the future repeat the past, how did that become the basis for disruption? Or maybe, the shutdown is actually what’s so disruptive because if knowledge now means knowing the future, it’s because in these models, the past and the future coincide So, most basically as we all know, these algorithms are trained on and tested against past data, which means if the past is racist, they’ll make racist predictions,

but also that if the past is racist, they won’t be verified as correct unless they make racist predictions So, as Reed Hoffman pointed out yesterday, these programs can lock in-scale inequality in a non-transparent way Yes, so what should we do? Well maybe, we need to be a little perverse Maybe we need to be a little perverse towards our attitudes towards these discriminatory algorithms, and what if we used existing AI assistance to diagnose current inequalities? What if we corralled discriminatory predictions as evidence of current and historical forms of discrimination? Right, so the example that everyone knows, of course, is Amazon’s secret hiring tool, which they’ve stopped using because it knocked your score down if you had woman anywhere in your CV So they’ve stopped using it, but what if instead we sort of thanked Amazon for meticulously documenting their discriminatory hiring practices? So in other words, rather than these models automating the mistakes of the past, what if we used them to diagnose and address past mistakes? So in other words, what if we used them and treated them like global climate change models? Right, so global climate change models tell us the most probable future if we keep repeating our certain actions, right, and when the global climate change models shows us that things are going up by two degrees, we don’t then say let’s fix the model, we say let’s fix the world So in other words, these predictions of the future, not to close it, but rather to open it up so another future could emerge, right So what if we also devised, and this is the last point, what if we also devised different kinds of algorithms in new ways to verify machine-learning programs so we displace these disturbing eugenic and segregationist histories, right? And this is something that was touched on really nicely earlier in the conference So, heterophily, the notion that opposites attract, what if those lines represented heterophily? This is how electromagnetism works How many of you know people who are heterosexual? Apparently, you can have heterophilis networks It exists Think of something like indifference What if indifference was actually the grounds that tie? Think of everything you need to be indifferent to in order to sit in the seat that you’re sitting in, right What if indifference or infrastructure became a way of understanding it? So, basically, the goal is to reach out and to engage the rich spectrum of human experience so we can build better models and systems in the world Thank you (audience clapping) – Thank you, Wendy and thank you, Bridget and Matt as well I told you Bridget and Wendy were gonna expand our thinking today and I think they’ve delivered So we have about a half an hour left, I have some questions here, but would love to hear your questions too So, I’ll ask a couple and then, you know, we’ll do some back and forth here, and there’ll be some microphones going around the room as well So, one thing that’s, whether we’re talking about the design or the deployment of AI systems, creating the frameworks to regulate them, who is in the room and at the table matters So we talked about this panel having different disciplines, different sectors coming together, this conference has attempted to do that in certain ways as well, Wendy, you described the diverse group of academics in your digital democracy lab, also about the challenges and problems of echo chambers, you called your group an unholy alliance, which I appreciated Matt the coalition of civil rights groups and others who are not traditionally or haven’t traditionally been focused on privacy and similar issues to come together in the table, and that was really transformative I’d like to dig a little bit deeper into that and ask you to reflect on how its been important in your work to have this diverse set of perspectives So what are those, you know, there are some of those moments that you have really illuminated new perspectives in your thinking, new approaches on how we might be able to do legal or legislative or litigation work, and how do we ensure that this is the norm going forward? Matt, we’ll start here Go down the line – Sure, sure, yeah thank you Being able to collaborate with our racial justice partners has meant that we have been able to pass better policy basically

To give a particular example, each year at the ACLU, we sit down and we do strategic planning at our affiliate in San Francisco and thinking about what are we gonna be working on in the next year, what sort of annual goal do we wanna sort of push towards, and what are the strategies we’re gonna need to get there, so we like to work on a variety of integrated advocacy strategies, whether it’s public records work, legislative work, litigation in the courts, we don’t just choose the tool that we wanna chose but we choose the one that is the best for getting towards the goal At the beginning of this year and coming out of last year’s legislative year, we’ve sat down deliberately with our racial justice partners including Malkia Cyril at The Center for Media Justice, simply now Media Justice, and we talked about where surveillance in communities was moving and what we saw together as some particular threats that needed to be addressed So out of that conversation came two really, sort of, definitive things that set us on a course to where we are now That was one, the understanding that these body cameras that had been deployed in many communities in California and elsewhere, were one, not delivering on the promise of transparency and accountability, but two, were at risk of being turned into roving surveillance devices And so Malkia and her deputy Steven, now the co-executive director of Media Justice, they said this is something we need to be focusing on, and having that conversation right there sort of narrowed the aperture towards this issue of body cameras, and that moved into legislative work which they were there alongside us the whole time Another thing that was really important for having the discussion early and often is to develop a narrative like why do we want this change that we want to achieve in the world and sitting down with Media Justice and other partners, we were able to break down, I think a traditional, a narrative that has traditionally been a barrier to passing civil rights reforms when it comes to surveillance, which is, on one side you have security and safety, which is typically owned as this sort of narrative by law enforcement, and the other side, you have civil rights and civil liberties and that sort of balance is not a productive one for conversations with policymakers, who are reasonably and understandably concerned about safety as well as the other things What we did by sitting down with our partners and a broader sort of coalition of groups is we sought deliberately to re-own the public safety narrative Not only to literally mention the word public safety or the phrase, whenever we could in our work, but to think about the ways, which history provides plenty, where unrestricted secretive surveillance doesn’t make people safer And so, not only the sort of concept of what the legislation would be, but the messages were a direct result of simply convening folks that we worked with and wanted to work with about and talking about what we were concerned about and how we were gonna talk about it – Bridget, you talked about a diverse group of collaborators, whether that’s an interdisciplinary group of academics and researchers, but also Boots on the Ground, how is that diversity of voices and participants in the work that you do shape the work and led to challenges potentially as well? – Yeah so I actually have a project now that I think probably is a really good example of the value of reaching across the aisle to a whole variety of stakeholders, so I mentioned that I work with Boots on the Ground, groups in Arizona in particular, I have a very generous partner who is in Pima County who I’ve worked with there who also shares their data So I have a project in which I have data on unidentified border deaths, and then subsequently, they have been identified So I have the ability to test the models and then actually verify them against the known information and using that information and applying the machine-learning algorithm, I’m able to estimate with very high probability, like in the 80s and 90 percents, where each of those individuals originated And while on the one hand that seems very negative and counterintuitive because it is a form of surveillance, right, that we’re able to make those types of determinations, it’s to the people who are working Boots on the Ground every day trying to take those remains and get them sent back to the right place, to their families of where they came from This kind of work is critical to be able to say, no this person didn’t come from Mexico or this person came from this state in Mexico, rather than just sending a file to a consulate that stays on a shelf indefinitely So, to be able to know what the needs are of the communities that you serve and think critically

about what you’re doing and broadly about what they need, to be able to bring those two perspectives together, I think is really important, because sometimes we get really lost in what we think other people need from us, or what we think is the type of research or good science that should be done and we miss the potential to actually make a contribution thinking about what the applied context is I also will say as, so as Matt was talking about the very practical, I’m gonna go back to the very academic theoretical that interdisciplinarity is talked a lot about and it’s very rarely practiced, and it is critical that we need to think interdisciplinarily so that we are creating students, student opportunities that celebrate the fact that we are good at a bunch of things, that we have multi or intersectional perspectives, that we are able to talk about not only what we do, but appreciate what someone else does, because how can we ever make judgments on the quality of work if we don’t have an understanding of how our work interacts with others? And to understand that the work that they’re giving us is properly contributive to the types of questions that we are seeing as important to answer So, we need to have systems in place as we’re teaching students to think that way, we need to have systems that allow for that to flourish, and then we need to also hopefully in doing that, create a culture in which this is the way that we think globally, that we are not promoting the fissuring of academia, or the partitioning of policy versus academics, but that we’re thinking connectively, where we’re creating a network of people working towards social justice, and I think that’s what so great about CCSRE is that it’s in locus here at Stanford that really celebrates bringing in many people from different departments, celebrating those perspectives and allowing those voices to be heard, and those voices are all voicing, work towards a common social justice goal, and I think we’re very lucky at Stanford to have that – So, Bridget you’ve talked about interdisciplinary, Matt, you’ve talked about trying to create a shared narrative, Wendy, one of the big challenges in doing this kind of work is language, right How do we communicate across disciplines or just even understand, what computer scientists are saying and scholars of literature are saying or critical theorists, this often, you know, sort of over each others heads In your unholy alliance, how have you been able to communicate effectively to do collaborative work? – Well this is gonna sound bizarre but I think what, in order to communicate across disciplines, what you need to do is actually go higher than these methodological issues and ask yourselves, what matters? So this all started because Brown’s like most academic institutions sometimes it feels like “Survivor” for academics A new president comes in and decides, we’re gonna have these little groups that are faculty-led and they’re gonna come up with these initiatives So, I was put on the big-data initiative, which gives you a sense of when it was, with the head of CS and someone in bio-stats, and we looked at each other and we said, we’re not gonna work together unless there’s a problem that you can’t solve without me and I can’t solve without you And the problem that we came up with is what’s the relationship between what’s real and what’s true? That if almost any correlation can be shown to exist, how do we know which matter? And this actually came from the person in bio-stats who’s working on AIDS research And then we added a second question was, so what if we know it’s true, what actually inspires political action? And this is what the engagement with global climate change but also asking ourselves what worked as simulations actually do So I’ve often found, and then to give you one more current that we’re working on right now is the question of misinformation, right a lot of people have shown that accuracy is important, known verification is important but it’s not enough, right Corrections don’t reach the same audience, people don’t care if a story is correct They find it compelling, and to be true, irregardless of whether or not it’s verifiable And so what we’re doing is working with somebody to build a model of authenticity So authenticity is different from facts It also has a very dramatic history, in the notion of the tragic hero, someone who lies a lot is considered to be authentic And so, what we’re thinking through is what would an authenticity measure, as opposed to an accuracy measure look like? So using machine-learning for that but then also working with people in these workshops, where they produce scripts, based on their experiences with misinformation,

which we then perform We’re also turning them into web extensions, but also trying to work with people in theater and performance studies to come up with other forms of interactions and scripts – So, we’re here at a panel on race rights and facial recognition and each of you talked about it in different ways in your discussion At this conference, we’ve heard some conversation about race, Joy certainly focused on it in her talk and other have discussed it as well, but I’d like you to help us reflect a bit more on why you think it’s important to center race and specifically racial justice in discussions about facial recognition technology in particular but also AI in general How have you worked to do this in your work, challenges you’ve encountered and people who don’t really wanna frame the conversation or the work in that way, thinking Matt, starting here, you’ve done work in the Bay Area, having a conversation here is very different than, about racial justice in a legal context is different than having it, say in other parts of the country Whether you, sort of lead with that argument or not is taken differently How does that manifest in your work? – Yeah, so, it’s been important to center racial justice in our work because of the sort of the historical arc of how surveillance technology has been used in communities of color and against communities of color, against immigrants and against activists, as I was saying during my presentation We’re to the point now where we have plenty of historical examples of what happens when a new surveillance system is built and what will the regime, or the sort of government in power, will do to use it? And who it will be used against? And so, the writing is on the wall, then we need to be paying attention to this issue and more and more, the intersectional conversations are happening in the civil rights communities, where we realize together that, we can’t be talking about privacy as a sort of high-minded concept because one, that’s not persuasive to policymakers, that’s not persuasive to the members of all of our groups as talking about real people’s lives are So, really it’s a, I think sometimes just more valuable to be talking about people’s lives and addressing the history and the patterns that have occurred And you’re right that the Bay Area is different than a lot of other of other places, but we’re seeing this narrative and sort of conversation expand across the United States right now Over a dozen cities in the United States have passed laws Not all of them have facial recognition bans, but many of them have a requirement that the community has to be a part of the conversations about surveillance technology, and so you have places like Nashville, Tennessee, St. Louis, considering and adopting legislation that requires that all communities members have an opportunity to say no to surveillance technologies and to say what the safeguards should be I was also, as you were saying that question, thinking about a meeting we had last December with the general council of Amazon as a coalition, I flew up to Seattle and local groups in Seattle working on racial justice issues came to that meeting and besides the sort of disappointment of not having this sort of productive conversation around racial justice with Amazon, who I think and we think, has really utterly refused to engage with the sort of impacts that their technology can have on real human beings What was striking, the most striking moment in that meeting for me was hearing a gentleman from a Japanese-American advocacy organization sit across the table from the general council of Amazon and talk about Japanese internment, and how the history of what happened there, which I mean, amongst many other things, the government used a very superficial feature, the Japaneseness of Americans, to round people up and put them in internment camps without regards for real dangerousness or anything more than skin deep level, and that was just an amazing, and I thought, impressive reminder of why it’s so important to be talking about this with as many people at the table as possible because that sort of reminder has not only stuck with me, it stuck with everyone in that room I know it stuck with the Amazon executives as well, and it’s a different perspective than what a community member in San Francisco or Oakland might bring to the table – Wendy, you think about it sort of give us a genealogy, challenge some of these histories of development of methodologies, how does that connect with racial justice? Why is it important to center racial justice in conversations that we think about, that history of science and what the future brings – Because, what it brings out is the fact that race is central It’s not something that people have added on to the discussion, it’s actually been there and in terms of Japanese internment camps, one of the interesting things in terms of looking at homophily and studies of homophily was it wasn’t

just this integrated, yet segregated housing project When they were trying to think through it, they said, okay we’re interested in this because it’s social engineering, they were blatant about this is social engineering, and one of the examples that they drew from and tried to understand was Japanese internment camps, as well as Indigenous reservations So, it’s not, why it’s so important to engage questions of race, is that you realize that if we’re in this situation now, it’s not an accident, and that we need to engage communities and think through what other things were possible People in the Japanese internment camp did not experience things the ways in which the sociologists who were studying them did So it’s by engaging these communities and histories and making them the basis for different kinds of models and understanding, that’s absolutely key – And Bridget, we’re thinking about race as a marker, along with age and stature Why is it important to think of it as more than just a specific variable or marker but think about and have conversations and do the work in terms of racial justice? – ‘Cause it’s always already all about race and so how we, who we chose to have as a partner is racialized That then has effects on races may not exist, but biologically, but races have biological consequences So to think about how race influences every aspect of our biology So to understand how through our biological shifts and biological patterns, we need to understand how race operates within the world, and, I will say that I feel like I’ve been super negative about facial recognition and science and I’m actually a very positive scientist that I would say that doing innovative exciting work, is a lifeblood of what we do and that’s important And I will say that from my perspective, there can be very positive outcomes of issues of race and its intersection with facial recognition We know that eye-witness testimony is terrible and having the opportunity then in certain cases to be able to provide another source of documentation may prove helpful To be able to have information on where someone might have been if we’re dealing with a missing persons case, a human trafficking case, that there are potential good uses of it As I said earlier, we just don’t know what we don’t know, and we’re in such an area of change and flux within the technology and within its uses that often the potential for the good needs to be regulated by the, or weighed against the immediate risk right now for its misuse So I would hope someday that we might reach a point at which policy and risk versus value will balance out, where we can have opportunities where it can be used in places where it would actually do the work of social justice – So, in our last few minutes here, our last ten minutes or so, we’d like to open it up for some questions from the audience I believe we have some microphones How about up front here, David – Thank you, it was a really fascinating and great discussion I guess one of the categories I was thinking about, not just around the difference between truth and reality, for example, was what counts as authoritative? Right, so there’s so much of what happens with this technology is that there’s this presumption that technology produces authoritative results, and there’s a question and it’s an ethical conundrum, what are the correctives against that, not just the data, but the mechanisms that will allow to correct for what is considered to be authoritative, so cultural literacy, you know, you said everyday people, Matt, right, like what are communities that are being targeted around these technologies, like what are their bull works against that? – So, I can talk just quickly about the question, so clearly authenticity and authority come from the same root as well, so that’s part of trying to think through that authenticity route, but I think that the bigger question that you raise is verification How are we going to talk about verification if we don’t want verification of a model to be the fact that it’s results happen? So like for instance, global climate change modelers, they don’t wanna be proved right, right The idea isn’t to wait to see if their predictions are true, right

So there, you know, this is why you have hindcasting and different forms of trying to understand, oh can it do a different data set, but I think that that whole question of what is verifiable versus what is outside of simply repetition is absolutely key and it’s something that’s facing us And also, one reason why we like the example of global climate change models is because it’s unclear they’re convincing either, given the debate that we’re having in the U.S., right, so I mean, I think thinking through that and that relationship is really important – And I will just add that this idea of one version I think, of what I was hearing you say is that people trust machines in some sort of sense, we’re just more likely to trust machines than we would have trusted an eye-witness or a human telling us the same thing, so in the field that not only can result in the wrong decisions being made when the systems themselves are incorrect, which is life and death questions sometimes, but I do also think that it results, that sort of bias towards trusting machines, I’ve also heard it called automation bias, right It also causes us to sometimes reflexively think that surveillance technology is going to be the answer to societal problems that could be addressed through other policy mechanisms or other discussions And so, you’ll see this every time there’s a tragedy, every time, unfortunately now there’s a school shooting, the reflects of response frequently is going to be more surveillance, more security, I mean this is just a tried and tested pattern unfortunately But I think that is in some ways influenced by our sort of willingness to trust machinery and processes, when in fact, they don’t address the root causes of the problems that are harming people in our society – All right, let’s go back in the middle, yeah – So, are there other interdisciplinary examples where people are using machine-learning to challenge bias in non-linear, non-binary ways that are visual For example, there’s a project here with ID cards for the medical school, maybe that doesn’t have to be the standard kind of mugshot ID Are you aware of any other projects like that that we’re doing? – So there’s a really fascinating computer scientist based at the University of Washington who’s using Word2vec so this can, this is textual, but in order to diagnose and redress discrimination So using Word2vec to see how certain words are placed close to each other in certain corpuses, in order then make the case for certain kinds of both implicit and explicit bias – To just build off that, some of my own work using text mining methods which are inherently machine-learning as well, studying how academics talk about issues of race So, if we removed the word race, are there patterns in which we speak about race and it’s an encoded structure that we don’t actually have to say the word, but that we have a language and the language that signals to people reading it, that we’re talking about race or that there’s this implicit bias in the way that we’re describing things, and in being able to tease those patterns out, we can then do better ourselves and teach our students how these latent trends exist and the way people consume these latent trends and how they’re effective of particular sentiments or reactions in people when they read our work For me, coming from initially a biology, and we talked about this a bit, about how there are different languages, there’s the language of science, there’s a language of law, there’s the language of humanities, and to be able to recognize that the way you’re presenting something could encode a different message to someone else, and if we’re really trying to embrace interdisciplinarity, we also have to be able to talk in a common language and to appreciate how what we’re saying could potentially be, or not be, taken as we intended it, so there are exciting machine-learning ways to also get at that as well – Others, why don’t we do here and here – Maybe him first – All right, let’s collect a couple of questions and we’ll have our panelists answer them Sure, we got a mic back here, go for it – Thanks, just a quick one, I think this is mostly for Wendy and Bridget but I’d be interested to hear anyone’s thoughts on it

So, Wendy you mentioned this kind of leap into fiction in that sociological article you were talking about as something that could potentially, at its best, sort of shed light on the made up-ness of the facts involved, right And Bridget you closed talking about these AI-generated faces based on DNA that are, you know, in a real sense fictional faces that are being used to guide policing in this litter project And so I was just wondering if you had any thoughts about, as someone who works on literal fictions, I work on 19th century literature, on if we can think about AI as producing fictions, about machine-learning models as kind of generating fictions and whether that kind of ontological shift would have an impact on, kind of, what we think about the goals of these models being? – All right let’s hold that one, and we’ll collect a couple here, how about up in the front – Hi, so I have a question mostly for Bridget which is like a lot of critics of biometrics or facial recognition jump to like a historical analogy with phrenology, and I think this is because like phrenology did a lot to systematize and worsen racial injustice, and also, because it was kind of bad science like they thought they could detect people’s personality from like these craniological measures, they had that time and they were fooling themselves, and a lot of people think that like with adversarial examples and not being able to generalize machine-learning has a lot of maybe similar flaws, so I’m wondering like, as someone from your background, what do you think about that historical analogy? – All right, can we get one more in there? Up front here Michelle, so we’ve got fiction, NAI, and phrenology – So much, just a quick question, this is such a great panel, so exciting, I want to just pick up on something I heard, first, congratulations on passing that legislation, Matt, I think that’s kind of amazing, and all of you have sort of have talked also about educating the public or the industry, Bridget was pointing out educating the public, but you also notice that you know, that there’s tech PR and lobbyists that have the ear of influencers who are elected and not and I know you’ve all sort of dealt with them, and so in the collaboration with interdisciplinarity, of course, there’s also all these competing interests and incentives and commercial self-interest as well too, so I’m wondering, when you were talking, Matt, about shaping the public imagination, and you were, I’m really excited also to think about how the arts and humanities are shaping perception then too It’s sort of, I also heard Reed Hoffman the other day and also have heard Eric Smith talk about how they don’t support this legislation, so I’m wondering as this plays out, if recruiting narratives will be successful given the enormous amount of money in not regulating those kinds of emerging technologies in the way that you were suggesting – Thank you Let’s hear about phrenology – Yeah, let’s do phrenology Yeah, do that first So, I will say first that I am a question, question, question person So when I first got into anthropology and started thinking about ancestry and that issue of phrenology, it of course came to mind, how am I doing something different than phrenology? Why is this different, why are all these critics who are, a lot of them who are anthropologists, biological anthropologists, wrong in when they say this is just an extension to phrenology So actually, that was like the thrust of my dissertation project so I’m very passionate about this, good question – Good question – I decided that within my dissertation, for my dissertation I was going to turn the mathematical models on their head and instead of using models for testing whether or not we get the same answers, just with different data sets, in terms of, can we organize people into geographic groups or is there, what we call, biologically structure, so there’s some organization in human biology that follows along some cultural or geographic lines Instead of introducing new data, ’cause that’s always been like the solution like more data, more data, more data, better models What if we completely alter how we think about the model, and take the classifier out of the classification? And this is something that’s very similar to what’s also done in genetics So, I stripped all the data down so that it had no identifiers, and I wrote an algorithm that would look at the data structure alone,

those of you who are maybe familiar with text analysis, there’s like this concept of bag of words where you sort of take everything away and you just look at the word patterns I took all of those reference information away, the metadata, created an algorithm that just created patterns based on the measurements alone, and it could have determined that there was no pattern whatsoever So it would have been one great big glump of everyone all amassed or it could have assigned each person, based on their measurements, to their own little category So that instead of having one group of 25,000, I would 25,000 mini groups Instead, it actually grouped people into evolutionarily and social-historically relevant groups So then, it tells us that yes, there are aspects to biology that are shaped by culture, that are shaped by our evolutionary history, that are driven by climate, that are driven by social norms, rules against assortative mating et cetera, that have changed how our biology is expressed So there is a way then to get at that question of phrenology and still do it in a mathematical way using, you know, cutting-edge technology that doesn’t necessarily require you to take a position I essentially let the data speak, and then use what I knew contextually, as an anthropologist, as someone who studied history, and geography, and cultural anthropology to make sense of that, and I think that also gets to your comment about authority and authenticity that numbers are numbers until we bring context So, there’s the opportunity then to bring that context to the numbers, to flush out the picture, to be able to determine whether or not it actually presents something that’s authentic or real, and, as I’ve said in my talk, something can also be authentic and real and it doesn’t mean it matters, so just because you’ve expended all this energy on defining something doesn’t mean you should, and doesn’t mean that it was worthwhile in the end or should be assigned some value just because it’s present or absent or whatever So again, context in that regard is also important – AI and fiction, AI as fiction perhaps – That’s such a great question and I think that the relationship between fiction and technology is such a large one, I mean So, just a clear example is something like how did the internet ever become cyberspace? Right, you know, read “Neuromancer” then look at “Gopher,” whoa Two do not map onto each other, there was no jacking and there was, you know, so I think that the generative power of fiction is important, especially if you look at something again like authenticity as opposed to verifiability Just telling a bunch of dead heads that you know there’s a difference between jacking in and “Gopher” that’s not gonna do anything The question was why and how did it become felt as true? Like what was that capacity there? And I think there is this generative moment that we need to engage and think through – And I’ll just add that I think, so in terms of your question and in terms of literary studies, I think there’s great value in studying fiction, because how people write about people and present the world tells us a lot about what we think and feel and believe and know and it helps us to come to that better understanding of that context So sometimes the numbers in the scientific articles don’t tell us as much as a description of the world in something that is fiction Sometimes fiction can be more true or of greater value in studying that in terms of flushing out those pictures of the understanding of that context – And finally, how can we challenge and reshape the narrative and the face of massive PR, powerful PR machines? – Yeah, I would say that, you know, right now, big tech lacks an affirmative vision of how the world would be a better place if facial recognition were pervasive in our society And the fact that they lack that vision is not lost on legislators, and legislators when given the opportunity to vote against facial recognition and for what the voters want, are so far falling behind the voters vision of things I would also say in addition to that, that this sort of idea that platforms are not going to be responsible for the ills that they cause in the world That is something, I think, big techs sometimes clings to, still but it doesn’t get any traction to policymakers now People, policymakers, I mean you’re seeing this on the news all the time, whether it’s Russian interference

or some other thing that the tech companies have had a role in, they’re rejecting, policymakers are rejecting this idea that a company like Amazon should have the ability to throw its hands up when it sells an API to a law enforcement agency who goes and uses it to harm people There’s not the disconnect that I think, tech would prefer to be there, so that is also playing a role, and then having pased legislation in California, in Massachusetts, and here in the state of California now, the status quo and this sort of gold standard is moratorium or a ban So the burden is now on proponents of the technology to articulate that affirmative vision and to show how it actually is good for communities and not bad for civil rights – With that, I’d like to thank you all for joining us today, (audience clapping) and thank you Wendy, Bridget, and Matt, for sharing your work