Executive Summary
Automated decision-making has become the unseen hand of discrimination, preventing the most vulnerable members of society from achieving important life goals. More evidence is emerging that discrimination is borne out when people seek a mortgage, apply for a job, credit, school, or government benefits. It happens when we don’t realize it, such as scrolling through our social media feeds. And it’s usually lower income individuals, people of color, females, religious groups, or those with disabilities who suffer the most as a result of automated decisions. But there is an opportunity to stop discrimination via automated decision-making under California’s soon-to-be-drawn regulations for the new California Consumer Privacy Act (CCPA). And that hinges on what automated decision-making applies to and what about it is disclosed. Under the CCPA, the state privacy agency is charged with writing broad rules requiring disclosure of, and allowing consumers to opt out of, automated decision-making, including profiling.
When voters enacted Proposition 24 in 2020, they directed the agency to “adopt regulations to further the purposes of this title,” including: “Issuing regulations governing access and opt-out rights with respect to businesses’ use of automated decision-making technology, including profiling and requiring businesses’ response to access requests to include meaningful information about the logic involved in those decision-making processes, as well as a description of the likely outcome of the process with respect to the consumer.”
The statute defines profiling as: “any form of fully or partially automated processing of personal information…to evaluate certain personal aspects relating to a natural person and in particular to analyze or predict aspects concerning that natural person’s performance at work, economic situation, health, personal preferences, interests, reliability, behavior,1 location, or movements.”
And indeed, personal information is defined as:1 “information that identifies, relates to, describes, is reasonably capable of being associated with, or could reasonably be linked, directly or indirectly, with a2 particular consumer or household.”
Specifically, the privacy agency should align automated decision regulations closer to Europe’s data privacy law, the General Data Protection Regulation (GDPR). The agency should write regulations stating that disclosure and the right to opt out of automated decision-making should apply to, “a decision based on fully or partially automated processing, including profiling, which produces legal or significant effects concerning the consumer.” This will capture a range of discriminatory behavior, including in financial lending, education, and the job market, as well as allow innovation that will benefit consumers.
Further, the privacy agency must write new disclosure rules for companies to reveal the logic behind their automated decisions under the CCPA. It should be no mystery to consumers as to how they are judged by an automated decision. A consumer should know the personal data that was processed, the automated decision’s consequences for the subject, the important factors used to formulate a decision, and their importance in the decision. Essentially, consumers should know what the algorithm making the decision is considering when formulating a decision. If an automated decision is based on gender, sexual orientation, geolocation, religious preference and race, a consumer should know about that.
When disclosing information to consumers about how automated decisions are determined, entities should not merely provide information, but meaningful explanation. Input data for profiling and the mathematical formula used are important for disclosure purposes, but if people are only given a bunch of metadata they can’t understand, then the law is useless. Disclosure should be in clear, explanatory terms. Such information is crucial for consumers to understand their situation and be empowered with the appropriate knowledge if they choose to opt out.
New disclosure rules about the logic behind automated decision-making should apply to mortgages, student loans, credit applications, the job market, gig economy workers and school applications.2 The California Consumer Privacy Act (opens in new tab)
“The privacy agency should align automated decision regulations closer to Europe’s data privacy law, the General Data Protection Regulation (GDPR). “ Automated decision-making is an appealing service for businesses and governments because it markets efficiency, equality, and savings. But it often relies on flawed data pools that punish the most vulnerable members of society. By one study’s account3 80 percent of businesses are using automated decision-making as of 2020, but 85 percent of algorithms throughout this decade will provide false analysis because of4 Taking these two figures into account presents a frightening scenario of a bias.
society that chooses cost and speed over fairness. The results are often a racist or classicist algorithm, a sort of digital redlining that occurs instantaneously. For example, home mortgage lenders in 2019 were found to give out loans 40-80 percent more to White people over people of color in scenarios where both had5 similar financial characteristics. In the courts, one sentencing risk assessment345 recommended longer sentences for Black people than White people convicted of6 the same exact crime. In 2019 Facebook agreed to enter into a settlement with the ACLU for deploying an algorithm that targeted men and excluded women from7 the audience for traditionally male job openings, like truck drivers. In 2022 the Department of Justice warned that AI hiring practices discriminate against those with disabilities. Courts have ruled that automated decisions that deny or reduce benefits without proper notice or explanation infringe on the due process rights of8 the beneficiaries.
With regulations being drafted in the coming months, California is positioned to lead the way in forcing needed disclosure of the algorithms that control too many aspects of Americans’ lives.
We look to GDPR for guidance, where caselaw has come down in favor of consumers who seek to exercise their rights under the law, including automated decision-making and disclosing the logic of algorithms.678
Introduction
The tech world was founded on utopian ethos, positioning itself as the key to addressing many of society’s problems. AI tools like automated decision-making are just one of the latest examples tech evangelists cite as a way to eliminate human flaws such as bias and error. But we’re seeing more and more evidence of how that’s not true. In fact, the supreme irony is that automated decision-making is often exasperating discrimination, not fixing it.
“With opaque, unknowable algorithms, systemic racism will be even harder to address.”
The New York Times recently reported that Black taxpayers are at least three times as likely to be audited by the Internal Revenue Service thanks to the algorithm used to determine who is chosen for an audit. However, it’s not completely clear why. The program skews toward auditing those who claim certain earned-income tax credit, but Black Americans are still selected for audit more, even in comparison to other9 groups who also claim the tax credit. The algorithm also targets less complex returns instead of ones that include business income. With opaque, unknowable algorithms, systemic racism will be even harder to address.
And the unseen hand of automated decision-making is everywhere, feeding off a10 personal data trade that is worth hundreds of billions of dollars. Over the next five years, 1 in 4 organizations plan to start using or increase their use of automation, according to the study. A total of 75 percent of c-suite executives reported fears that their company would be in peril if it didn’t scale automated11 decision-making by 2024. And with the economic downturn, various sectors are only more likely to turn to automation in order to cut costs. In 2021, for example,12 the federal government invested more than $6 billion into artificial intelligence. Overall, about half of our daily tasks can be automated with current13 technology.
The following sections address concerning areas of discrimination though automatic decision-making.910 Robin Bloor, Permission, April 6, 2020.111213
Job Hiring
It’s a common feeling: Imagine being qualified for a job you really want, applying for it, but you can’t even land an interview. Increasingly, your fate rests with an algorithm that, for several reasons, weeded out your resume. Those reasons could be due to gaps on your resume because of a pregnancy to certain key words or phrases you don’t have on your resume. Or maybe it had nothing to do with your resume and you were excluded because of a joke you made on Twitter seven years ago.
In other words, there is a hidden AI hiring process in which we don’t know what the employer is even looking for. That has resulted in discriminatory hiring practices based on the color of a person’s skin, their gender and economic status.
“There is a hidden AI hiring process”
In addition, programmers can create algorithms based on past or current favorable employees. Instead of looking for skillsets, the algorithm looks for word patterns in resumes to see if the prospective candidate is a fit for the company. But issues could arise if the data pool used by algorithms is too uniform and constitutes certain gender, race, or age. If, for example, the benchmark resume is of a white male, it may exclude other key words or phrases like, “First female president of the Asian American Bar Association,” and therefore skip over Asian and/or women applicants. In 2019 Facebook agreed to enter into a settlement with the ACLU for deploying an algorithm that targeted men and excluded women from the audience14 for traditionally male job openings, like truck drivers.
California’s predecessor data privacy law in Europe, the General Data Protection Regulation, offers insight on how automated decision-making has been viewed in the eyes of the courts. A job application assessment used by a German government entity automatically assessed and ranked job applicants according to predetermined criteria. Applicants’ names, addresses, gender and severe disabilities was among the personal data used for the assessment, which was the only way applicants would be invited for interviews. A court concluded that there was profiling and automated decision-making, because the decisions taken lacked15 meaningful human intervention and significantly affect applicants’ rights. Almost 1 in 4 organizations use AI to support their HR-related activities, according16 to a 2022 study by Society for Human Resource Management. Nearly half of large employers with over 5,000 employees use automated decision making in their HR departments. And ninety-nine percent of Fortune 500 companies use AI for17 hiring.
Some agencies are catching on. In 2022 the Department of Justice warned that AI18 hiring practices discriminate against those with disabilities. Since common work-related AI tools include resume scanners, monitoring software that ranks workers based on keystrokes, and video interviewing software, such technology could filter out people with speech impediments or those who type or work slower due to physical or mental impairments, according to the DOJ.1415 Future of Privacy Forum, Sebastião Barros Vale and Gabriela Zanfir-Fortuna, May 2022.161718
Loans
The financial lending industry is a scary peek into how people are trapped into poverty. Discriminatory algorithms exist in the mortgage lending world, handicapping people of color from building generational wealth. An investigation by The Markup found home mortgage lenders gave out loans 40%-80% more times to white people than people of color in scenarios where both groups had similar19 financial characteristics. In addition, high-earning Black applicants with less debt were denied loans more than high-earning White applicants with more debt. This digital redlining is based on a dated algorithm that emphasizes traditional credit from the 1990s and not criteria such as payments made on time.
“The algorithm can easily misidentify people entirely”19 It’s just as much a nightmare in the rental world, where tenant screening reports rely on an unregulated algorithm. Those who are merely arrested, and not even convicted of a crime, or have an eviction merely filed against them, are disqualified. And Black renters are more likely to be targeted by police and20 homeowners.21 There is also a high error rate in these screening reports. A disabled Connecticut man was denied permission to move in with his mother after the algorithm deployed by a tenant screening company said it had found a “criminal court action” because of a shoplifting arrest. But the arrest never ended in a conviction and the charge was ultimately dropped. One federal lawsuit claimed the screening22
company RealPage submitted 11,000 inaccurate tenant background checks.
“RealPage submitted 11,000 inaccurate tenant background checks”202122
The algorithm can easily misidentify people entirely, based on the same or similar names. For example, over 12 million Latinos share only 26 last names, according to U.S. Census numbers. People with common names are disproportionately victim to algorithmic error.
Automated decision-making in the world of credit and lending has grown thanks to a boost from major corporate consultants, who have advised clients that credit-decision-making models boost profits.
“Using new credit-decisioning models is not only a powerful way to boost profits but also a business-critical competitive imperative,” said the influential consulting23 firm McKinsey in a 2021 report. “Banks need to implement more automated credit-decisioning models that can tap new data sources, understand customer behaviors more precisely, open up new segments, and react faster to changes in the business environment.”
And banks have listened to devastating effect. A program used by banks to predict the probability that someone will pay back credit card debt favors wealthier, white24 applicants. Predatory lending is now optimized to better identify and target low-income people. High-interest, payday loans are taken out by 12 million Americans per year, whose average income $30,000. The average interest rate is 650% and25 over half of borrowers struggle to meet monthly expenses.
However, courts have weighed in on these issues. A Finnish data regulator enforcing GDPR found that a financial credit reporting company could not use age26 as an automatically excluding factor from having a credit application analyzed.23242526
Employed by the Algorithm
Gig economy workers such as those who work for Uber, Lyft, Doordash or Grub Hub essentially work for an algorithm. Amazon warehouse workers also live and die by an algorithm that monitors their every move. Much of their working lives are dictated by an automatic decision, from how much work they receive, to their schedule, where they travel, down to the penalties and pay they receive. But the nature of these automated decisions are often shrouded in mystery. Uber uses a blackbox algorithm to calculate fares based on “several factors,” including estimated trip length, duration, demand, and surge pricing. But Uber has not addressed if there are more factors.
However, there has been some guidance from courts in Europe who have made rulings regarding automated decisions and gig workers. An Amsterdam District Court ruled in March 2021 that automated decisions which impose fines or reduce fares for drivers based on the performance data it collects on them “significantly” affected the driver. The court ruled the rideshare company in question, Ola, could
not make such decisions.
“The nature of these automated decisions are often shrouded in mystery”
School
Higher education has been flirting with automated decision-making for a few years now as a tool to help meet important financial goals. One vendor promises a “10% increase in enrollment” through its scholarship optimization algorithm. Admission offices see such tools as a way to make better guesses about scholarship funding and27 how it increases the probability of student enrollment.
One AI lending platform charged higher fees to borrowers who attended Howard University, which is historically black, or majority Latino New Mexico State28 University, than it charged those who went to New York University.
And the admissions process is also guided in some cases by an AI that scans29 prospective enrollees in areas such as openness, motivation, and “neuroticism.” According to a presentation for the software company Kira Talent, which bills itself as “the unbiased assistant,” prospective students are analyzed based on a number30 of traits, including “cooperative,” “cold” and “aggressive.” New York University has used a company named Element451, which gauges potential for success based on how prospective students interact with the school’s website and respond to31 messages. And decisions once made by faculty are being made by AI. The company ElevateU uses an algorithm to administer different learning mechanisms to students, deciding to have a student learn visually or via the written word. There is evidence that the reason there is a high percentage of college students who drop out because of too much debt is due to the algorithm. One survey of 1,000 dropouts found that those with loans held an average of $14,000 in student loan32 debt, and just under half were making payments. And according to the Center272829303132 for American Progress, 90% of those who defaulted on loans had received a Pell33 grant.
Companies that specialize in enrollment optimization have said they can target what an applicant is willing to pay while doling out the minimum amount of financial aid in order to maximize tuition. From the advanced analytics platform Othot: “With advanced analytics, you can analyze how to invest the minimum amount of aid necessary to meet and exceed your goals. Or, even better than investing in more aid is prioritizing and focusing your recruitment and communication resources on the right students at the right time to increase the34 likelihood they will enroll.”
“There is evidence that the reason there is a high percentage of college students who drop out because of is due to the algorithm.”
too much debt3334 There is a strong argument to be made that due to enrollment optimization algorithms, it is targeting those at the edge of financial stability, therefore contributing to the high percentage of students who drop out over debt concerns. In one court case pertaining to the European data privacy law, CNIL, the data protection authority in France, looked at how French universities automatically ranked applications based on residence, the order of their wishes, and their family35 situation. Based on that ranking, the schools automatically made an offer. CNIL found this sort of automated ranking of prospective students by university admissions was illegal.35
How Automated Decision-making Rules Should Be Drawn
California’s bellwether privacy law, the CCPA, calls for the ability to opt out of automated decision-making and profiling of sensitive personal information, the rules of which are currently being contemplated by the agency tasked with drafting regulations and enforcing them, the California Privacy Protection Agency (CPPA). Specifically, the privacy agency should apply the law’s protections to automated decisions with legal or significant effects, aligning the law with Europe’s data privacy law, the General Data Protection Regulation. The new CCPA should say, “The consumer shall have the right not to be subject to a decision based on fully or partially automated processing, including profiling, which produces legal or significant effects concerning the consumer.”
This will capture a range of discriminatory behavior, including in financial lending, education, and the job market, as well as allow innovation that will benefit consumers.
“Legal effects” would occur when someone’s legal rights are affected, such as decisions impacting a contract.
The new CCPA
“Significant effects” would be a decision that impacted a
should say, “The
person’s circumstances or behavior, such as decisions that affect someone’s financial situation, denies
consumer shall have
employment or access to education.
the right not to be subject to a decision
“Significant effects” generally would not encompass marketing, however, it depends on other factors such as
based on fully or
intrusiveness, how people are tracked via other websites,
partially automated
and an individual’s situation. For example, someone in a
processing, including
difficult financial situation who is targeted with advertisements for high-interest loans signs up for the
profiling, which
offer and incurs further debt. This sort of targeted,
produces legal or
behavioral advertising, which is the main driver of our
significant effects
modern surveillance economy, should be considered automated decision-making that significantly affects a
concerning the
person. And consumers should know with specificity
consumer.”
why they are seeing an ad and opt-out of such automated decision-making.
Everyday uses that are also automated decision-making technology, such as GPS,
spam filters, spellcheck and other low-risk, widely used tools should not be subject to the opt out right.
Under the CCPA statute, automated decision-making encompasses profiling, which is defined in the law as, “any form of automated processing (opens in new tab) of personal information…to evaluate certain personal aspects relating to a natural person (opens in new tab) and in particular to analyze or predict aspects concerning that natural person (opens in new tab) ’s performance at work, economic situation, health, personal preferences, interests, reliability, behavior, location, or movements.”
And personal information is broadly construed in the CCPA statute as, “information that identifies, relates to, describes, is reasonably capable of being associated with, or could reasonably be linked,36 directly or indirectly, with a particular consumer or household.” The law further defines personal information as precise geolocation, and characteristics of any protected class, including race, religious creed, national origin, physical or mental disability, mental disability, sex, age, or sexual orientation. Personal information also includes identifiers such as name, address, social security number and biometric data.36
Disclosing the Logic Behind Automated Decisions
In 2022, a Swedish bank was deploying automated decision-making for credit applications of its customers, as well as seeking to detect cases of fraud or money laundering. In its data protection notice, the bank only went as far as saying it used contact, identification, and financial information for its automated decisions. However, it did not explain what criteria may lead to a negative credit decision. This led the Swedish Data Protection Authority finding that the bank violated requirements under GDPR that companies disclose the logic behind automated decision-making. The regulator said, “The requirement to provide meaningful information on the logic behind an automated credit decision entails the indication of the categories of data that are crucial in the context of an internal scoring model and the possible existence of circumstances that always lead to a refusal37 decision.”
It should be no mystery to consumers as to how they are judged by the automated decision. A consumer should know the personal data that was processed, the automated decision’s consequences for the subject, and the most important factors used to formulate a decision. Like in the Swedish example, consumers should know the categories of data as part of a scoring model and any circumstances that lead to a negative credit decision.
Essentially, consumers should know what the algorithm making the decision is considering when formulating a decision. If an automated decision is based on gender, sexual orientation, religious preference or race, a consumer should know about that. Such information is crucial for consumers to understand their situation and be empowered with the appropriate knowledge if they choose to opt out. The privacy agency should fashion regulations so users are not merely buried in inscrutable data they can’t understand, but also provided easy to understand information. Consumers deserve not just meaningful information, but meaningful explanation. Meaningful means contextual, explanatory. Many legal researchers believe a fundamental duty to explain automated decision-making exists instead of38 providing abstract information in favor of data controller secrecy. If people are given a bunch of metadata they can’t understand, then the regulation is useless.3738 This is echoed in GDPR regulations, which state: “The GDPR requires the controller to provide meaningful information about the logic involved, not necessarily a complex explanation of the algorithms used or disclosure of the full algorithm. The information provided should, however, be sufficiently comprehensive for the data subject to understand the reasons for the39 decision.”
In addition, people should know when other people’s personal data is used to make a decision about themselves. For example, a credit card company might lower the credit line for a person, based not on that person’s own repayment history, but based on other customers living in the same area who shop at the same stores. This could result in people being deprived of opportunities based on the actions of
others. People may be given credit lines who cannot afford it, or denied credit when
“The court ruled that it was profiling”
they can. This logic should be disclosed and allow for users to opt out of this sort of automated decision-making.
Businesses should also not confuse a consumer uploading personal information as permission to score the consumer based on that data. The business still has a duty to disclose the logic of such a score. Indeed, this was a ruling made by the Italian40 Supreme Court in 2021, finding it violated GDPR’s transparency obligations. European caselaw also established that a fraud probability score created by rideshare company Ola was considered profiling, and had to be disclosed to drivers, even if an automated decision was not made based on that score. This was the ruling by the Amsterdam District Court in 2021 after drivers requested information about their fraud probability scores, earning profile, and assigned rides and fines. Regarding the fraud probability score, the court ruled that it was profiling under GDPR because, “through the automated processing of personal data of [applicants], a risk profile is drawn up with which a prediction is made about their41 behavior and reliability.” The court did not determine automated decisions were made from this, but ruled, “This does not alter the fact that Ola must provide access to the personal data of [applicants] that it used to draw up the risk profile and provide information about the segments into which [applicants] have been42 classified.”404142
Conclusion
With technology rapidly changing how businesses operate, consumers are long overdue for protections. Americans have had to rely on various civil rights, unfair competition or narrow state and federal statutes to indirectly protect them from the negative and discriminatory effects of automated decision-making. The CCPA grants consumers broad new rights against the damaging consequences of this developing technology. The plain language of the law requires the agency to let Californians know how they are being profiled, and their right to opt out of automated decision-making. That was the intent of voters when they passed
Proposition 24, which endowed Californians with unprecedented control over the use of personal data.
“The law requires the agency to let Californians know how they are being profiled”
