Amazon Scraps Biased Recruiting AI

Discussion in '[H]ard|OCP Front Page News' started by AlphaAtlas, Oct 11, 2018.

  1. AlphaAtlas

    AlphaAtlas [H]ard|Gawd Staff Member

    Messages:
    1,044
    Joined:
    Mar 3, 2018
    An AI Amazon built to automate the hiring process reportedly took a turn for the worse. According to the report, the machine learning algorithm was programmed to sort through resumes and pick out the best candidates for a job. Machine learning algorithms learn from the data they're trained on, and the resumes Amazon fed this algorithm were unintentionally skewed towards male candidates. Naturally, the AI picked up on that, and showed a bias against women's resumes that took some time to discover. Amazon has allegedly abandoned the effort, but not before the AI was used in at least some capacity.

    Amazon declined to comment on the technology’s challenges, but said the tool "was never used by Amazon recruiters to evaluate candidates." The company did not elaborate further. It did not dispute that recruiters looked at the recommendations generated by the recruiting engine.
     
  2. Mav451

    Mav451 [H]ardness Supreme

    Messages:
    4,443
    Joined:
    Jul 23, 2004
    Seems there are details being left out of the story here lol. Eliminating more subjective, human elements from the process would logically point to a more impartial process.
    To interpret the results as biased because it doesn't follow a desired, political narrative is frankly laughable.

    The author adding quotes from attorneys of ACLU's Racial Justice Program at the ACLU isn't helping either lmao.
     
  3. Madoc

    Madoc Gawd

    Messages:
    902
    Joined:
    Mar 10, 2006
    I wonder if it would show a bias in favor of women making sammiches for the menfolk? Mmmm... Sammiches!
     
    AlphaQup, dvsman and RaxusCraxurFace like this.
  4. Dahkoht

    Dahkoht Limp Gawd

    Messages:
    415
    Joined:
    Jan 2, 2014
    Or , or , hear me out on this , it's picking more male candidates because there are more male qualified candidates ?

    Now calm down , calm down ,I know it's a long shot , just saying , there is a possibility that could be the truth.


    (SJW head explodes)
     
  5. naib

    naib [H]ard|Gawd

    Messages:
    1,141
    Joined:
    Jul 26, 2013
    Exactly :) apparently the argument they gave was there was more males to review. Unless the criteria given was "give male" then it comes downto the characteristics more favoured males. By scrapping it for some real-world ideology to provide bias based upon sex pretty much prove the entire concept of equality of outcome is flawed.

    There was Equality of opportunity, just a certain real-world group did not exhibit enough additional characteristics, characteristics key to the job, to be picked...

    unless no women were picked this is SJW looking for some means to express bias when they are the ones showing bias
     
  6. percydaman

    percydaman Limp Gawd

    Messages:
    276
    Joined:
    Jun 26, 2011
    Seems the way they setup the AI to learn was deeply flawed. From what I read in another place, the AI was so stupid it just looked for certain key phrases/words and shit in an application/resume.
     
  7. Seventyfive

    Seventyfive [H]ard|Gawd

    Messages:
    1,261
    Joined:
    Jul 14, 2004
    In other news, my machine learning AI that I made for fantasy basketball only wanted to recruit 6'7" to 7'6" black guys. Back to the drawing board I guess.
     
  8. kju1

    kju1 2[H]4U

    Messages:
    2,982
    Joined:
    Mar 27, 2002
    Or you know they could have retrained it...
     
  9. HVAC

    HVAC n00bie

    Messages:
    28
    Joined:
    Sep 25, 2015
    Garbage in, garbage out.
     
    commissioneranthony likes this.
  10. Kalabalana

    Kalabalana [H]ard|Gawd

    Messages:
    1,289
    Joined:
    Aug 18, 2005
    Remove source data that contains gender.
    Problem solved, why scrap?
     
    Revdarian and auntjemima like this.
  11. travisty

    travisty Gawd

    Messages:
    724
    Joined:
    Feb 3, 2016
    Nope. I know anti-SJW love to troll but this is a no.

    What happened here is what happened with apple's face recognition (not being able to recognize non-white faces accurately). The data used to train the AI was a bad sample. It was too limited and those who picked the data were either lazy or didn't take/given the time to fully analyze the problem space.

    BTW with commas there's a space after not before; same for question marks.
     
    Last edited: Oct 11, 2018
  12. DukenukemX

    DukenukemX 2[H]4U

    Messages:
    3,995
    Joined:
    Jan 30, 2005
    Probably because the AI already did this, but still gravitated towards men because they were still the best choice.
     
    naib, Revdarian, Seventyfive and 2 others like this.
  13. travisty

    travisty Gawd

    Messages:
    724
    Joined:
    Feb 3, 2016
    Not sure what's hard to understand here.

    If you, a human, have a stack of resumes. You then separate them out by levels of qualifications and then feed them to the AI. You, the human, can then compare which resumes were reject and which were accepted - this is basic acceptance testing.
    If a discrepancy is found - people of equal qualifications are being rejected more in the case they are black, latino, asian, woman, or male - then you've found a problem.
     
  14. travisty

    travisty Gawd

    Messages:
    724
    Joined:
    Feb 3, 2016
    It appears to be fun being anti-SJW to you but more than likely Amazon decided to ditch it entirely so that it can avoid any backlash from the media / government oversight.

    Amazon saying this AI didn't work but they are retraining it would result in ACLU, media, possibly government wanting to test it themselves

    Amazon saying that they tried something, found it discriminated, and shut it down ends the issue. A year down the line once everyone's forgotten they can try again.
     
  15. kju1

    kju1 2[H]4U

    Messages:
    2,982
    Joined:
    Mar 27, 2002
    No it wouldnt. Amazons only talking about this to distract you from the bad press they got over cutting their employee benefits. Or maybe something else thats come out or about to come out. If they really wanted to fix this they would have just quietly done it and nobody wouldve known any better.
     
  16. Skyblue

    Skyblue Limp Gawd

    Messages:
    219
    Joined:
    Jan 2, 2008
    HR employee emulator?
     
  17. Tawnos

    Tawnos 2[H]4U

    Messages:
    3,800
    Joined:
    Sep 9, 2001
    We can inadvertently inject our own biases into our machine learning models. For example, there was an issue where LinkedIn "discovered that high-paying jobs were not displayed as frequently for women as they were for men" because there was an existing relationship between people with those jobs already and gender. That's not eliminating the subjectivity of an interviewer and replacing it with objective elements, it's making those existing biases systemic.

    As others point out, there's a model training problem here, not a political narrative one.
     
    masteralef likes this.
  18. Cyraxx

    Cyraxx [H]ardness Supreme

    Messages:
    4,556
    Joined:
    Feb 21, 2005
    That's actually how all of our federal government jobs are determined (seriously). USAJobs.gov applications are based ENTIRELY on having keywords in your resume.

    My wife works for the government (as a contractor) and took a claas they offerred on how to apply for USAJobs (non-contract) - And thats basically what they taught her... it's entirely based on keywords. So read the application, fit in the application words into your resume somehow (but not obviously) and you're usually good to go.
     
    Revdarian likes this.
  19. travisty

    travisty Gawd

    Messages:
    724
    Joined:
    Feb 3, 2016
    This was already out so nothing to hide. Amazon had to make a decision and decided to avoid as much negative press as possible. This isn't anything new.
     
  20. lcpiper

    lcpiper [H]ardForum Junkie

    Messages:
    10,096
    Joined:
    Jul 16, 2008

    Because I think we have misunderstood the goal ..... to utilize AI, removing the human element, in order to achieve recruiting goals selected in order to reduce the likelihood of legal actions targeting hiring practices.

    In the legal world, to my understanding, it's not enough to ensure that you will win. In order to reduce wasting money you need to eliminate the risk of a suite all together. The goal is, hire as many of each as is needed in order to eliminate the risk of being suited to the highest degree achievable over time.

    In order to do this, fair has nothing to do with it, you just need to play the numbers right.
     
  21. freeloader1969

    freeloader1969 2[H]4U

    Messages:
    2,221
    Joined:
    Dec 13, 2003
  22. arkhenon

    arkhenon n00bie

    Messages:
    4
    Joined:
    Oct 11, 2018
    This is easily fixable, I do not understand the reason to scrap it completely. There is no reason to feed gender data into a model like this. Gender should not have any effect on the impact of each individual qualification. Why would the ML model require such input? It is like giving it the applicant's name as input, which would skew the model towards hiring people with more popular names. Especially if their label is who was hired previously and the contents of their resumes.

    It seems to me that it is an input validation/preparation problem. Validate better, re-train, and keep on using it. If Amazon is having trouble with ML models, I have no idea who would be able to do it properly.

    Edit - Clarity
     
    drescherjm likes this.
  23. lcpiper

    lcpiper [H]ardForum Junkie

    Messages:
    10,096
    Joined:
    Jul 16, 2008
    Been a contractor for about 20 years, this can get you a phone call, maybe an interview, it usually doesn't get you the job. That being said, you usually won't get the job without the interview and the preceding phone calls so ..... it's simply understanding of what your resume is supposed to be doing for you, and that is mostly getting you the interview.

    Next, you need a resume that you float on the job boards, Monster, USAJobs, etc, and this one is written in order to attract interest for the kind of jobs you want. and are qualified for. This is important, it's a let down to a potential employer/recruiter for you to suggest that you have skills that are not really there. And because most recruiters seem to target keywords, filling your resume with all the great keywords you can think of just ensures that you will fill your inbox and your voicemail with an endless stream of recruiters ..... who all sound like they are here on an H1B Work Visa.

    Make sure that the keywords you select are focused on the job you want.

    Then if you are actively searching for a job and target one, tailor your resume to address the job description as recommended above. I'm not saying to make shit up, just address each major point in a way that let's them see how you measure up against their job as written. Familiarity is one thing, exposures, experience with, etc. If you get to an interview, expect questioning along all these lines.

    You can't plan for a recruiting unit that doesn't know what they are doing or for a company that doesn't know how to recruit,, and if you come acrossed this, well it'll come down to how well they like you more than how good a fit you are.

    Keywords are important, they can work for you, they can work against you, all depends on how you use them.

    As for relevance to this article, keywords make sense, it's how the people do it so why shouldn't it be how they want the AI to approach the issue.
     
  24. lcpiper

    lcpiper [H]ardForum Junkie

    Messages:
    10,096
    Joined:
    Jul 16, 2008

    You've answered your own question and just don't know it yet.

    Try the logic, if you wanted to eliminate the possibility of gender bias then you would eliminate gender from the criteria right? You said it, it's a no brainer.

    Which is why, logically, that is not their goal. Logically, their goal must be something that is dependent upon gender or at least, gender is an element that they can't eliminate and achieve their goal.

    Therefore they must intend an outcome that is at least partially dependent upon gender.

    I don't think they want an truly unbiased outcome. I think they want an outcome they appears unbiased and provides other desired benefits as well.
     
    Madoc, Laowai and JoeRChillin like this.
  25. YeuEmMaiMai

    YeuEmMaiMai Death Incarnate

    Messages:
    18,150
    Joined:
    Jun 11, 2004
    So much lolz not enough space....
     
  26. arkhenon

    arkhenon n00bie

    Messages:
    4
    Joined:
    Oct 11, 2018
    From a data science perspective, this does not make any sense. Assuming that you are asserting that they are trying to achieve a biased outcome leaning towards hiring more women - even if the qualifications do not support it - this is a baffling way to achieve that. It is obvious that they are training with previous hiring as labels. Assuming they have more male than female employees (fair assumption), it means that their model would automatically skew towards suggesting male employees. If they used previous hiring data as negative labels, then the model would have skewed towards hiring more female employees (but this is obviously not happening).

    Which can be explained in two ways. Either they had ulterior motives to hire more females, but implemented their model in the most opposite logic to this goal as possible, which requires incredible amounts of stupidity (especially if they KNOW what their goals are). And which would be easily fixable on their end, if they wish so. And it would have been so easy to debug the logic of it ("Our model was supposed to suggest more females, but it is suggesting more males. There is an issue that we must fix."). Or they just fed the complete resumes without much data cleaning/validation. My Occam's razor tells me that the second explanation is much more likely. Just looking at it objectively, as a machine learning engineer (and as someone who knows what the most common mistakes in the field usually are).
     
  27. lcpiper

    lcpiper [H]ardForum Junkie

    Messages:
    10,096
    Joined:
    Jul 16, 2008

    The only assumption I am making is that they are not actually incompetent idiots, and if that is the case, then gender must have been a desirable characteristic that they required in order to achieve what they wanted from this solution. No more and no less. If you read more into it than that, that's up to you.

    Their failure could be for a couple of reasons and I'll leave someone with more experience in the field to posit potential reasons for this.
     
    Madoc likes this.
  28. arkhenon

    arkhenon n00bie

    Messages:
    4
    Joined:
    Oct 11, 2018
    Fair enough, but if their goal was to get more male than female suggestions, then they are already achieving that. If their goal is the opposite, that is easy to fix and achieve. I just cannot see any plausible way that the gender is what they are actually trying to focus on here. Because if that was the case, it is substantially easy to fix that issue (and as I said, debug). Most plausible reason is that they messed up the data cleaning/validation. And again, this aligns very well with what I am seeing in AI/ML model implementations as that is exactly my field. Incompetence seem to be more of an issue in the field. Though I must say, Amazon should have better people, especially on this field, since they are trying to lead it. Maybe I should apply :)
     
  29. Kalabalana

    Kalabalana [H]ard|Gawd

    Messages:
    1,289
    Joined:
    Aug 18, 2005
    I'm under the impression they are not doing anything so complex, simply trying to replace the human element in the recruiting process. Once they realized the gender skewed results, they scrapped the process.
    Maybe the realization that led to this was that filtering out indicators of gender in existing data was too complex a process, and not so basic as to remove the applicant's name.
    Gender lean in resumes may not be so objective as I (and possibly Amazon) had thought.
     
    arkhenon likes this.
  30. Kalabalana

    Kalabalana [H]ard|Gawd

    Messages:
    1,289
    Joined:
    Aug 18, 2005
    Socially forbidden consideration!!!!
    But a real possibility
     
  31. arkhenon

    arkhenon n00bie

    Messages:
    4
    Joined:
    Oct 11, 2018
    That is very fair, and very likely to have happened. But I still do think it should be achievable to fix the issue of removing gender indicators from the resume. May not be as straightforward as just removing a name, as you mentioned. There are things like likely gender pronouns, somewhat different thought patterns in forming certain sentences and so on, I'm sure. But it is an interesting problem and personally I am interested in taking a shot at doing it properly :) It sounds like a nice challenge.

    My biggest gripe in this is from a professional standpoint. Amazon is legitimately trying to be a leader in this field, so I do think they have a responsibility in doing a better job, and not giving up.
     
    Kalabalana likes this.
  32. AtomClock

    AtomClock n00bie

    Messages:
    47
    Joined:
    Jul 7, 2018
    This is total nonsense! If the AI is basing it's decision to hire men by looking at the M F box then remove the M F box. Then the AI will be forced to hire based on the best qualified regardless of sex. BUT, since the fix is trivial and yet it obviously still hired more men than women. Then they MUST stop using AI because it only cares about the best qualified and seems to be ignoring the SJW dogma.
     
  33. Montu

    Montu [H]ard DCOTM x4

    Messages:
    8,005
    Joined:
    Apr 25, 2001
    LOL.

    We're headed for another dark age because of this ignorant quest for equal outcomes.
     
    Madoc and Skyblue like this.
  34. mygreeneggsandham

    mygreeneggsandham n00bie

    Messages:
    40
    Joined:
    Feb 18, 2016
    In other news, Scientists create an AI to determine the best tasting food but then quickly scrap it as it is discovered that the AI has a bias for ribeye steaks and not for khale.
     
    Madoc, naib, Skyblue and 2 others like this.
  35. Eickst

    Eickst [H]ard|Gawd

    Messages:
    1,770
    Joined:
    Aug 24, 2005
    When does the AI do the interview and discover that 90% of the resumes are all lies?
     
  36. Dead Parrot

    Dead Parrot 2[H]4U

    Messages:
    2,172
    Joined:
    Mar 4, 2013
    From TFA, 90% of the resumes from the 10 year period of the data fed to the AI were from males. The AI reached the conclusion that male resumes were better. Didn't see any mention of what percent of Amazon hires over that 10 year period were male. If the ratio of hires closely matched the ratio of applicants, then the AI screwed up. If say 98% of hires were male, then the AI reached the proper conclusion. If only 50% of hires were males then the AI really screwed up.
    Face it, for most job openings, most resumes represent failures that aren't hired. Amazon might have gotten better results by only feeding in the winning resumes instead of the complete pile which has mostly failures.
     
  37. kju1

    kju1 2[H]4U

    Messages:
    2,982
    Joined:
    Mar 27, 2002
    But that begs the question...why are male and female resumes different? If you take name + gender off the resume all thats left is the skills etc. That should be the same regardless of gender. So training the AI should teach it to recognize the best skills not which resume is male vs female. Either their algorithm is flawed or males write less sucky resumes.
     
  38. Cmdrmonkey

    Cmdrmonkey Gawd

    Messages:
    997
    Joined:
    Jul 19, 2004
    Maybe the machine wasn't biased at all and was just picking the best candidates, who tended to be men.
     
    Last edited: Oct 12, 2018
  39. DNMock

    DNMock Limp Gawd

    Messages:
    292
    Joined:
    Apr 16, 2015

    Most likely the case, but Title IX, Affirmative Action, etc., etc., are things that exist and I don't think the Govt. or lawyers care much about if it was an AI or a human doing the hiring process, if those quota's aren't filled, it's gonna cost them big time.

    Programming an AI to find the best candidates is probably easy, but programming an AI to pick the best candidates and meet all the rules and regulations (some of which are quite illogical) is probably pretty damn hard.
     
    Madoc likes this.
  40. kju1

    kju1 2[H]4U

    Messages:
    2,982
    Joined:
    Mar 27, 2002
    Eh I dont know about that. Not only is it generally hard to prove that there is discrimination in hiring (this case is probably easier) but the enforcement really sucks. The last major company I remember reading about that got fined was target in 2015 and their fine was 2 million I think. A drop in the bucket for them. In 2017 the EEOC only got about 400 million in total damages across the entire set of companies in the US...a drop in the bucket. Their website says they launched 184 court cases last year which means the average settlement per case was about 2 million...

    Plus the law limits damages. For employers with more than 500 employees, the limit is $300,000.
     
Tags: