Amazon Scraps Biased Recruiting AI

AlphaAtlas

[H]ard|Gawd
Staff member
Joined
Mar 3, 2018
Messages
1,713
An AI Amazon built to automate the hiring process reportedly took a turn for the worse. According to the report, the machine learning algorithm was programmed to sort through resumes and pick out the best candidates for a job. Machine learning algorithms learn from the data they're trained on, and the resumes Amazon fed this algorithm were unintentionally skewed towards male candidates. Naturally, the AI picked up on that, and showed a bias against women's resumes that took some time to discover. Amazon has allegedly abandoned the effort, but not before the AI was used in at least some capacity.

Amazon declined to comment on the technology’s challenges, but said the tool "was never used by Amazon recruiters to evaluate candidates." The company did not elaborate further. It did not dispute that recruiters looked at the recommendations generated by the recruiting engine.
 
Seems there are details being left out of the story here lol. Eliminating more subjective, human elements from the process would logically point to a more impartial process.
To interpret the results as biased because it doesn't follow a desired, political narrative is frankly laughable.

The author adding quotes from attorneys of ACLU's Racial Justice Program at the ACLU isn't helping either lmao.
 
Or , or , hear me out on this , it's picking more male candidates because there are more male qualified candidates ?

Now calm down , calm down ,I know it's a long shot , just saying , there is a possibility that could be the truth.


(SJW head explodes)
 
Seems there are details being left out of the story here lol. Eliminating more subjective, human elements from the process would logically point to a more impartial process.
To interpret the results as biased because it doesn't follow a desired, political narrative is frankly laughable.

The author adding quotes from attorneys of ACLU's Racial Justice Program at the ACLU isn't helping either lmao.
Exactly :) apparently the argument they gave was there was more males to review. Unless the criteria given was "give male" then it comes downto the characteristics more favoured males. By scrapping it for some real-world ideology to provide bias based upon sex pretty much prove the entire concept of equality of outcome is flawed.

There was Equality of opportunity, just a certain real-world group did not exhibit enough additional characteristics, characteristics key to the job, to be picked...

unless no women were picked this is SJW looking for some means to express bias when they are the ones showing bias
 
Seems the way they setup the AI to learn was deeply flawed. From what I read in another place, the AI was so stupid it just looked for certain key phrases/words and shit in an application/resume.
 
Or , or , hear me out on this , it's picking more male candidates because there are more male qualified candidates ?

Now calm down , calm down ,I know it's a long shot , just saying , there is a possibility that could be the truth.


(SJW head explodes)

Nope. I know anti-SJW love to troll but this is a no.

What happened here is what happened with apple's face recognition (not being able to recognize non-white faces accurately). The data used to train the AI was a bad sample. It was too limited and those who picked the data were either lazy or didn't take/given the time to fully analyze the problem space.

BTW with commas there's a space after not before; same for question marks.
 
Last edited:
Seems there are details being left out of the story here lol. Eliminating more subjective, human elements from the process would logically point to a more impartial process.
To interpret the results as biased because it doesn't follow a desired, political narrative is frankly laughable.

The author adding quotes from attorneys of ACLU's Racial Justice Program at the ACLU isn't helping either lmao.

Not sure what's hard to understand here.

If you, a human, have a stack of resumes. You then separate them out by levels of qualifications and then feed them to the AI. You, the human, can then compare which resumes were reject and which were accepted - this is basic acceptance testing.
If a discrepancy is found - people of equal qualifications are being rejected more in the case they are black, latino, asian, woman, or male - then you've found a problem.
 
Probably because the AI already did this, but still gravitated towards men because they were still the best choice.

It appears to be fun being anti-SJW to you but more than likely Amazon decided to ditch it entirely so that it can avoid any backlash from the media / government oversight.

Amazon saying this AI didn't work but they are retraining it would result in ACLU, media, possibly government wanting to test it themselves

Amazon saying that they tried something, found it discriminated, and shut it down ends the issue. A year down the line once everyone's forgotten they can try again.
 
It appears to be fun being anti-SJW to you but more than likely Amazon decided to ditch it entirely so that it can avoid any backlash from the media / government oversight.

Amazon saying this AI didn't work but they are retraining it would result in ACLU, media, possibly government wanting to test it themselves

Amazon saying that they tried something, found it discriminated, and shut it down ends the issue. A year down the line once everyone's forgotten they can try again.

No it wouldnt. Amazons only talking about this to distract you from the bad press they got over cutting their employee benefits. Or maybe something else thats come out or about to come out. If they really wanted to fix this they would have just quietly done it and nobody wouldve known any better.
 
Seems the way they setup the AI to learn was deeply flawed. From what I read in another place, the AI was so stupid it just looked for certain key phrases/words and shit in an application/resume.
HR employee emulator?
 
Seems there are details being left out of the story here lol. Eliminating more subjective, human elements from the process would logically point to a more impartial process.
To interpret the results as biased because it doesn't follow a desired, political narrative is frankly laughable.

The author adding quotes from attorneys of ACLU's Racial Justice Program at the ACLU isn't helping either lmao.

We can inadvertently inject our own biases into our machine learning models. For example, there was an issue where LinkedIn "discovered that high-paying jobs were not displayed as frequently for women as they were for men" because there was an existing relationship between people with those jobs already and gender. That's not eliminating the subjectivity of an interviewer and replacing it with objective elements, it's making those existing biases systemic.

As others point out, there's a model training problem here, not a political narrative one.
 
Seems the way they setup the AI to learn was deeply flawed. From what I read in another place, the AI was so stupid it just looked for certain key phrases/words and shit in an application/resume.

That's actually how all of our federal government jobs are determined (seriously). USAJobs.gov applications are based ENTIRELY on having keywords in your resume.

My wife works for the government (as a contractor) and took a claas they offerred on how to apply for USAJobs (non-contract) - And thats basically what they taught her... it's entirely based on keywords. So read the application, fit in the application words into your resume somehow (but not obviously) and you're usually good to go.
 
No it wouldnt. Amazons only talking about this to distract you from the bad press they got over cutting their employee benefits. Or maybe something else thats come out or about to come out. If they really wanted to fix this they would have just quietly done it and nobody wouldve known any better.

This was already out so nothing to hide. Amazon had to make a decision and decided to avoid as much negative press as possible. This isn't anything new.
 
Remove source data that contains gender.
Problem solved, why scrap?


Because I think we have misunderstood the goal ..... to utilize AI, removing the human element, in order to achieve recruiting goals selected in order to reduce the likelihood of legal actions targeting hiring practices.

In the legal world, to my understanding, it's not enough to ensure that you will win. In order to reduce wasting money you need to eliminate the risk of a suite all together. The goal is, hire as many of each as is needed in order to eliminate the risk of being suited to the highest degree achievable over time.

In order to do this, fair has nothing to do with it, you just need to play the numbers right.
 
This is easily fixable, I do not understand the reason to scrap it completely. There is no reason to feed gender data into a model like this. Gender should not have any effect on the impact of each individual qualification. Why would the ML model require such input? It is like giving it the applicant's name as input, which would skew the model towards hiring people with more popular names. Especially if their label is who was hired previously and the contents of their resumes.

It seems to me that it is an input validation/preparation problem. Validate better, re-train, and keep on using it. If Amazon is having trouble with ML models, I have no idea who would be able to do it properly.

Edit - Clarity
 
That's actually how all of our federal government jobs are determined (seriously). USAJobs.gov applications are based ENTIRELY on having keywords in your resume.

My wife works for the government (as a contractor) and took a claas they offerred on how to apply for USAJobs (non-contract) - And thats basically what they taught her... it's entirely based on keywords. So read the application, fit in the application words into your resume somehow (but not obviously) and you're usually good to go.

Been a contractor for about 20 years, this can get you a phone call, maybe an interview, it usually doesn't get you the job. That being said, you usually won't get the job without the interview and the preceding phone calls so ..... it's simply understanding of what your resume is supposed to be doing for you, and that is mostly getting you the interview.

Next, you need a resume that you float on the job boards, Monster, USAJobs, etc, and this one is written in order to attract interest for the kind of jobs you want. and are qualified for. This is important, it's a let down to a potential employer/recruiter for you to suggest that you have skills that are not really there. And because most recruiters seem to target keywords, filling your resume with all the great keywords you can think of just ensures that you will fill your inbox and your voicemail with an endless stream of recruiters ..... who all sound like they are here on an H1B Work Visa.

Make sure that the keywords you select are focused on the job you want.

Then if you are actively searching for a job and target one, tailor your resume to address the job description as recommended above. I'm not saying to make shit up, just address each major point in a way that let's them see how you measure up against their job as written. Familiarity is one thing, exposures, experience with, etc. If you get to an interview, expect questioning along all these lines.

You can't plan for a recruiting unit that doesn't know what they are doing or for a company that doesn't know how to recruit,, and if you come acrossed this, well it'll come down to how well they like you more than how good a fit you are.

Keywords are important, they can work for you, they can work against you, all depends on how you use them.

As for relevance to this article, keywords make sense, it's how the people do it so why shouldn't it be how they want the AI to approach the issue.
 
This is easily fixable, I do not understand the reason to scrap it completely. There is no reason to feed gender data into a model like this. Gender should not have any effect on the impact of each individual qualification. Why would the ML model require such input? It is like giving it the applicant's name as input, which would skew the model towards hiring people with more popular names. Especially if their label is who was hired previously and the contents of their resumes.

It seems to me that it is an input validation/preparation problem. Validate better, re-train, and keep on using it. If Amazon is having trouble with ML models, I have no idea who would be able to do it properly.

Edit - Clarity


You've answered your own question and just don't know it yet.

Try the logic, if you wanted to eliminate the possibility of gender bias then you would eliminate gender from the criteria right? You said it, it's a no brainer.

Which is why, logically, that is not their goal. Logically, their goal must be something that is dependent upon gender or at least, gender is an element that they can't eliminate and achieve their goal.

Therefore they must intend an outcome that is at least partially dependent upon gender.

I don't think they want an truly unbiased outcome. I think they want an outcome they appears unbiased and provides other desired benefits as well.
 
You've answered your own question and just don't know it yet.

Try the logic, if you wanted to eliminate the possibility of gender bias then you would eliminate gender from the criteria right? You said it, it's a no brainer.

Which is why, logically, that is not their goal. Logically, their goal must be something that is dependent upon gender or at least, gender is an element that they can't eliminate and achieve their goal.

Therefore they must intend an outcome that is at least partially dependent upon gender.

I don't think they want an truly unbiased outcome. I think they want an outcome they appears unbiased and provides other desired benefits as well.

From a data science perspective, this does not make any sense. Assuming that you are asserting that they are trying to achieve a biased outcome leaning towards hiring more women - even if the qualifications do not support it - this is a baffling way to achieve that. It is obvious that they are training with previous hiring as labels. Assuming they have more male than female employees (fair assumption), it means that their model would automatically skew towards suggesting male employees. If they used previous hiring data as negative labels, then the model would have skewed towards hiring more female employees (but this is obviously not happening).

Which can be explained in two ways. Either they had ulterior motives to hire more females, but implemented their model in the most opposite logic to this goal as possible, which requires incredible amounts of stupidity (especially if they KNOW what their goals are). And which would be easily fixable on their end, if they wish so. And it would have been so easy to debug the logic of it ("Our model was supposed to suggest more females, but it is suggesting more males. There is an issue that we must fix."). Or they just fed the complete resumes without much data cleaning/validation. My Occam's razor tells me that the second explanation is much more likely. Just looking at it objectively, as a machine learning engineer (and as someone who knows what the most common mistakes in the field usually are).
 
From a data science perspective, this does not make any sense. Assuming that you are asserting that they are trying to achieve a biased outcome leaning towards hiring more women - even if the qualifications do not support it - this is a baffling way to achieve that. It is obvious that they are training with previous hiring as labels. Assuming they have more male than female employees (fair assumption), it means that their model would automatically skew towards suggesting male employees. If they used previous hiring data as negative labels, then the model would have skewed towards hiring more female employees (but this is obviously not happening).

Which can be explained in two ways. Either they had ulterior motives to hire more females, but implemented their model in the most opposite logic to this goal as possible, which requires incredible amounts of stupidity (especially if they KNOW what their goals are). And which would be easily fixable on their end, if they wish so. And it would have been so easy to debug the logic of it ("Our model was supposed to suggest more females, but it is suggesting more males. There is an issue that we must fix."). Or they just fed the complete resumes without much data cleaning/validation. My Occam's razor tells me that the second explanation is much more likely. Just looking at it objectively, as a machine learning engineer (and as someone who knows what the most common mistakes in the field usually are).


The only assumption I am making is that they are not actually incompetent idiots, and if that is the case, then gender must have been a desirable characteristic that they required in order to achieve what they wanted from this solution. No more and no less. If you read more into it than that, that's up to you.

Their failure could be for a couple of reasons and I'll leave someone with more experience in the field to posit potential reasons for this.
 
  • Like
Reactions: Madoc
like this
Fair enough, but if their goal was to get more male than female suggestions, then they are already achieving that. If their goal is the opposite, that is easy to fix and achieve. I just cannot see any plausible way that the gender is what they are actually trying to focus on here. Because if that was the case, it is substantially easy to fix that issue (and as I said, debug). Most plausible reason is that they messed up the data cleaning/validation. And again, this aligns very well with what I am seeing in AI/ML model implementations as that is exactly my field. Incompetence seem to be more of an issue in the field. Though I must say, Amazon should have better people, especially on this field, since they are trying to lead it. Maybe I should apply :)
 
This is easily fixable, I do not understand the reason to scrap it completely. There is no reason to feed gender data into a model like this. Gender should not have any effect on the impact of each individual qualification. Why would the ML model require such input? It is like giving it the applicant's name as input, which would skew the model towards hiring people with more popular names. Especially if their label is who was hired previously and the contents of their resumes.

It seems to me that it is an input validation/preparation problem. Validate better, re-train, and keep on using it. If Amazon is having trouble with ML models, I have no idea who would be able to do it properly.

Edit - Clarity
You've answered your own question and just don't know it yet.

Try the logic, if you wanted to eliminate the possibility of gender bias then you would eliminate gender from the criteria right? You said it, it's a no brainer.

Which is why, logically, that is not their goal. Logically, their goal must be something that is dependent upon gender or at least, gender is an element that they can't eliminate and achieve their goal.

Therefore they must intend an outcome that is at least partially dependent upon gender.

I don't think they want an truly unbiased outcome. I think they want an outcome they appears unbiased and provides other desired benefits as well.

I'm under the impression they are not doing anything so complex, simply trying to replace the human element in the recruiting process. Once they realized the gender skewed results, they scrapped the process.
Maybe the realization that led to this was that filtering out indicators of gender in existing data was too complex a process, and not so basic as to remove the applicant's name.
Gender lean in resumes may not be so objective as I (and possibly Amazon) had thought.
 
I'm under the impression they are not doing anything so complex, simply trying to replace the human element in the recruiting process. Once they realized the gender skewed results, they scrapped the process.
Maybe the realization that led to this was that filtering out indicators of gender in existing data was too complex a process, and not so basic as to remove the applicant's name.
Gender lean in resumes may not be so objective as I (and possibly Amazon) had thought.

That is very fair, and very likely to have happened. But I still do think it should be achievable to fix the issue of removing gender indicators from the resume. May not be as straightforward as just removing a name, as you mentioned. There are things like likely gender pronouns, somewhat different thought patterns in forming certain sentences and so on, I'm sure. But it is an interesting problem and personally I am interested in taking a shot at doing it properly :) It sounds like a nice challenge.

My biggest gripe in this is from a professional standpoint. Amazon is legitimately trying to be a leader in this field, so I do think they have a responsibility in doing a better job, and not giving up.
 
This is total nonsense! If the AI is basing it's decision to hire men by looking at the M F box then remove the M F box. Then the AI will be forced to hire based on the best qualified regardless of sex. BUT, since the fix is trivial and yet it obviously still hired more men than women. Then they MUST stop using AI because it only cares about the best qualified and seems to be ignoring the SJW dogma.
 
When does the AI do the interview and discover that 90% of the resumes are all lies?
 
From TFA, 90% of the resumes from the 10 year period of the data fed to the AI were from males. The AI reached the conclusion that male resumes were better. Didn't see any mention of what percent of Amazon hires over that 10 year period were male. If the ratio of hires closely matched the ratio of applicants, then the AI screwed up. If say 98% of hires were male, then the AI reached the proper conclusion. If only 50% of hires were males then the AI really screwed up.
Face it, for most job openings, most resumes represent failures that aren't hired. Amazon might have gotten better results by only feeding in the winning resumes instead of the complete pile which has mostly failures.
 
From TFA, 90% of the resumes from the 10 year period of the data fed to the AI were from males. The AI reached the conclusion that male resumes were better. Didn't see any mention of what percent of Amazon hires over that 10 year period were male. If the ratio of hires closely matched the ratio of applicants, then the AI screwed up. If say 98% of hires were male, then the AI reached the proper conclusion. If only 50% of hires were males then the AI really screwed up.
Face it, for most job openings, most resumes represent failures that aren't hired. Amazon might have gotten better results by only feeding in the winning resumes instead of the complete pile which has mostly failures.

But that begs the question...why are male and female resumes different? If you take name + gender off the resume all thats left is the skills etc. That should be the same regardless of gender. So training the AI should teach it to recognize the best skills not which resume is male vs female. Either their algorithm is flawed or males write less sucky resumes.
 
Maybe the machine wasn't biased at all and was just picking the best candidates, who tended to be men.
 
Last edited:
Maybe the machine wasn't biased at all and was just picking the best candidates, who tended to be men.


Most likely the case, but Title IX, Affirmative Action, etc., etc., are things that exist and I don't think the Govt. or lawyers care much about if it was an AI or a human doing the hiring process, if those quota's aren't filled, it's gonna cost them big time.

Programming an AI to find the best candidates is probably easy, but programming an AI to pick the best candidates and meet all the rules and regulations (some of which are quite illogical) is probably pretty damn hard.
 
  • Like
Reactions: Madoc
like this
Most likely the case, but Title IX, Affirmative Action, etc., etc., are things that exist and I don't think the Govt. or lawyers care much about if it was an AI or a human doing the hiring process, if those quota's aren't filled, it's gonna cost them big time.

Eh I dont know about that. Not only is it generally hard to prove that there is discrimination in hiring (this case is probably easier) but the enforcement really sucks. The last major company I remember reading about that got fined was target in 2015 and their fine was 2 million I think. A drop in the bucket for them. In 2017 the EEOC only got about 400 million in total damages across the entire set of companies in the US...a drop in the bucket. Their website says they launched 184 court cases last year which means the average settlement per case was about 2 million...

Plus the law limits damages. For employers with more than 500 employees, the limit is $300,000.
 
Back
Top