To streamline hiring, many employers have begun using artificial intelligence (“AI”) and machines learning programs to screen employment applicants. However, this has given rise to concerns of AI bias. Indeed, to combat this bias in the use of AI, New York City has just passed an ordinance banning bias in the use of AI and machine learning tools. The ordinance will become effective in 2023.

But, what does AI bias mean? To help understand, it is useful to look at some recent developments in the financial industry. The financial industry is facing some similar concerns about the use of AI and machine learning in the making of lending decisions. We can get a glimpse of the “AI bias” issue by taking a quick look at what politicians and regulators are saying with respect to bias and potential discrimination in the financial industry.

For example, Congresswoman Maxine Waters recently issued a letter, as Chair of the House Financial Services Committee, to five major financial regulatory agencies. See letter here. Essentially, the problems with AI bias concern historical inputs and the use of supposedly “neutral” inputs. The first concern can be summarized by this quote from the letter: “Historical data used as inputs for AI and ML can reveal longstanding biases, potentially creating models that discriminate against protected classes, such as race or sex, or proxies of these variables.” In effect, if the historical data used by an AI program is biased, then the AI outputs are also biased. Likewise, with the second issue — supposedly “neutral” inputs — these too can contain hidden bias. Examples include zip codes and a borrower’s frequent websites and domain destinations. In her letter, Waters suggested that financial regulators focus on several methods of combating AI bias such as demanding transparency and resisting purely automated decision-making.

In that same manner, the New York City Ordinance focuses on transparency by requiring that companies that use AI for employment decisions must conduct a “bias audit” and make the audit results available to applicants. The Ordinance also requires that job applicants be given notice that AI programs are being utilized in employment decisions. Notice also must be given concerning which parts of the job application will be subject to AI processing, what data is being collected, the sources from which data is being collected and used and the employer’s data retention and destruction policies. These notices are to be given 10 days before the AI is being used (but, frankly, it seems it would be better if notice was given before an applicant fills out an online application).

The NYC Ordinance is actually broader than AI and machine learning tools. The Ordinance applies to any use of an “automated employment decision tool” (“AEDT”). Other examples are statistical modeling and data analytics. AEDTs have been used commonly over the last decade for remote videoed employment interviews. Thus, the NYC Ordinance would apply to such interviews. The Ordinance is particularly directed at curtailing bias that is hidden in “simplified outputs” such as scores or rankings.

Companies can be penalized for violating the new Ordinance, but there is no private right of action.

If you have business law questions or questions about consumer privacy, data security or other legal issues related to internet law, contact the trusted internet and business lawyers at Revision Legal at 231-714-0100.

How the NYC Law Works in Practice

Local Law 144, New York City’s automated employment decision tool law, took effect on July 5, 2023 after several delays. The law is enforced by the NYC Department of Consumer and Worker Protection (“DCWP”). Employers and employment agencies that use an AEDT to screen candidates for positions in New York City or to evaluate employees for promotion are subject to the law. The key obligations fall into two categories: bias audits and candidate notifications.

A bias audit must be conducted by an independent auditor prior to using an AEDT and at least annually thereafter. The audit must calculate the selection rate for each race/ethnicity and sex category and compare those rates to identify any disparate impact. The audit results, along with a summary of the data relied upon, must be publicly posted on the employer’s website for at least six months after the AEDT is used to make an employment decision.

Candidate Notice Requirements

Under Local Law 144, employers must provide candidates and employees in New York City with advance notice before using an AEDT. The notice must be provided at least ten business days before the AEDT is used and must include:

• A description of the type of data collected for use by the AEDT and the source of that data
• The employer’s data retention policy
• How the employer will use the AEDT in the employment decision
• Notice that a candidate may request an alternative selection process or accommodation

Importantly, candidates have the right to request that the employer use an alternative, non-AEDT-based selection process. Whether an employer must honor such a request depends on whether a reasonable accommodation is available, but the right to make the request is explicit in the law.

Penalties and Enforcement

Civil penalties for violations of Local Law 144 range from $500 to $1,500 per violation per day. The DCWP can investigate complaints and impose fines. As noted, the law does not create a private right of action — meaning job applicants cannot sue employers directly under the NYC ordinance. However, applicants may still have claims under other laws. For example, if an AEDT produces a discriminatory outcome based on race, sex, national origin, or another protected characteristic, an applicant may have claims under Title VII of the Civil Rights Act of 1964, the New York State Human Rights Law, or the New York City Human Rights Law, all of which do permit private lawsuits.

The Broader National Picture

New York City’s law was the first of its kind in the nation, but it is no longer alone. Illinois enacted its Artificial Intelligence Video Interview Act in 2020, requiring employers to notify applicants when AI is used to analyze video interviews. Maryland passed a similar law. At the federal level, the EEOC has issued guidance on employer liability for AI-driven discrimination, and the agency has stated that existing employment discrimination statutes fully apply to the use of algorithmic tools in hiring.

The Federal Trade Commission has also weighed in, warning that AI tools used in hiring contexts may violate the FTC Act if they produce discriminatory outcomes or if vendors make false or unsubstantiated claims about the tools’ efficacy or fairness. The CFPB has raised similar concerns about AI in credit decisions, signaling a whole-of-government approach to AI accountability.

What Employers Should Do Now

Any employer using automated tools in employment decisions — whether for resume screening, video analysis, skills testing, or performance evaluation — should take the following steps:

• Inventory all AEDTs currently in use, including those embedded in third-party HR software platforms
• Obtain bias audit results from vendors or commission independent audits before using any AEDT for New York City applicants or employees
• Update candidate-facing communications to include the required disclosures at least ten business days before any AEDT is used
• Review vendor contracts to ensure the vendor will provide the data needed for bias auditing and will cooperate with regulatory inquiries
• Train HR staff on the accommodation request process and ensure there is a documented alternative selection procedure available

Given the evolving landscape of AI employment law across multiple jurisdictions, companies with employees or applicants in more than one state should consult employment counsel to map their current AEDT usage against applicable legal requirements. Contact the business lawyers at Revision Legal at 231-714-0100 for guidance.