IT Director Job Opening in NC - Apply Now

Discussion in '[H]ard|OCP Front Page News' started by Kyle_Bennett, Dec 6, 2017 at 4:52 PM.

  1. Kyle_Bennett

    Kyle_Bennett El Chingón Staff Member

    Messages:
    47,994
    Joined:
    May 18, 1997
    Mecklenburg County in North Carolina, with a population of around 900,00 good folks, is taking it right in the pooper today thanks to some enterprising hackers. It seems that someone left the door open and has allowed 48 of its 500 servers to be infected with LockCrypt Ransomware. County officials are now kicking around paying the bandits $23,000 in Bitcoin. Guess they had better get those other 452 servers mining quickly! Thanks cageymaru.


    “We are open for business, and we are slow, but there’s no indication of any data loss or that personal information was compromised,” Diorio said. Hackers typically don’t steal data but encrypt it, placing it out of reach of the owners until ransom is paid.

    The county has been in contact with the hackers, she said. “We have not made a decision whether or not to make the payment,” Diorio said. The county hopes to make a decision by the end of Wednesday.
     
    scojer likes this.
  2. Twisted Kidney

    Twisted Kidney 2[H]4U

    Messages:
    2,393
    Joined:
    Mar 18, 2013
    I would love to be the new IT director for an organisation that already has a running chainsaw installed in their collective ass hole.

    I mean, who wouldn't?

    I spent most of today cutting dovetails on 18" hewn logs. It's very soothing and it's a far better use for a chainsaw.
     
    Last edited: Dec 6, 2017 at 6:46 PM
    Bloodystumps and Pringle like this.
  3. MavericK

    MavericK Zero Cool

    Messages:
    29,718
    Joined:
    Sep 2, 2004
    Seems cheaper to just pay the ransom rather than pay someone to clean up this shitstorm. But then, pay someone afterwards to lock your shit down.

    Though, as we found out when a client brought in a ransomware machine and wanted to pay - it's hard to actually buy Bitcoin. Not sure if that's changed, this was back when it was around $4k and rising.
     
  4. RogueTadhg

    RogueTadhg [H]ard|Gawd

    Messages:
    1,077
    Joined:
    Dec 14, 2011
    On the bright side - You know going into the job that that shit hit the fan.

    I sort of want to take on the challenge. And hey, it's where Charlotte, NC is located - which is a pretty large freaky city. So it can't possibly be too bad paying.

    CIO Keith Gregg. Tisk.
     
    Last edited: Dec 6, 2017 at 5:15 PM
  5. Gigus Fire

    Gigus Fire [H]ard|Gawd

    Messages:
    2,033
    Joined:
    Oct 14, 2004
    Why does a county have 500 servers to begin with?
     
    Esso, DF-1, NeoNemesis and 2 others like this.
  6. MrValentine

    MrValentine [H]ard|Gawd

    Messages:
    1,125
    Joined:
    Jul 5, 2004
    I was going to ask the same question!!! WTF!!?? Is the county a front for something?
     
    cyclone3d likes this.
  7. cyclone3d

    cyclone3d [H]ardForum Junkie

    Messages:
    11,826
    Joined:
    Aug 16, 2004
    And lemme guess.. no offsite backups.
     
    Solhokuten likes this.
  8. jojo69

    jojo69 [H]ardForum Junkie

    Messages:
    10,256
    Joined:
    Sep 13, 2009

    sign up at Coinbase

    wire in USD

    buy Bitcoin

    what hard?
     
  9. MavericK

    MavericK Zero Cool

    Messages:
    29,718
    Joined:
    Sep 2, 2004
    I recall at the time it required a lot of verification beforehand, and would have taken more time than there was left on the ransomware. The client also waited awhile to get the machine to us so it left us with very little time to act.
     
  10. viper1152012

    viper1152012 Limp Gawd

    Messages:
    339
    Joined:
    Jun 20, 2012
    What a been not having backups. Reimage and mve on otherwise
     
  11. jojo69

    jojo69 [H]ardForum Junkie

    Messages:
    10,256
    Joined:
    Sep 13, 2009
    ahh, that makes sense

    yes, Coinbase complies with all US KYC/AML requirements, which can take a few days

    localbitcions would have been your next bet for speed, or post up in here and make a deal
     
  12. Dead Parrot

    Dead Parrot [H]ard|Gawd

    Messages:
    1,121
    Joined:
    Mar 4, 2013
    If they are a Microsoft shop, very believable. No matter how much Microsoft crows about how one server can do many things, it rarely works out that way. Need to restart DHCP service, DNS takes a crap. Restart Print Server after driver updates, OS restart needed because vendor 3's driver requires it. It is just simpler to have many servers, each doing one thing. A better question is how many of those 500 are physical machines vs virtual. Restoring virtual servers is often as simple as reverting to a recent snapshot.

    Sounds like they had a backup policy since they said that getting the unlock code and testing it would have taken about as long as restoring the data from backups. Hope they have done restore tests in the recent past. "Where's the DVD with the restore software?, Fred's house? What do you mean he's on vacation for two weeks!"
     
    lcpiper likes this.
  13. tmillszero1

    tmillszero1 Limp Gawd

    Messages:
    222
    Joined:
    May 11, 2011
    We are actually a very large county incorporating many cities including Charlotte. Not really a surprise for 500 servers. Actually its a small number considering...
    Just a "squirrel!" thought... And Merck's global prod datacenter is here in the same county and got spanked too with a crypto virus! So it boils down to budget, experience and prioritization.
     
    Last edited: Dec 6, 2017 at 10:17 PM
  14. haste.

    haste. [H]ard|Gawd

    Messages:
    1,118
    Joined:
    Nov 11, 2011
    Really? It's pretty obvious...

    Porn.
     
    nightfly and Esso like this.
  15. painintheworld

    painintheworld [H]Lite

    Messages:
    64
    Joined:
    Jun 5, 2007
    I live about 100 miles north and west of Uptown and travel there several times per month. It is unreal how much the entire metro area has grown in the last 25 years.

    I really dig Charlotte and have thought of buying a place in the suburbs for when we are in town.

    For the poster above that mentioned Charlotte being freaky...that would be Asheville :)
     
    Gigantopithecus likes this.
  16. Dan_D

    Dan_D [H]ardOCP Motherboard Editor

    Messages:
    51,835
    Joined:
    Feb 9, 2002
    You'd be surprised how many servers a state may have. 500 in a specific county that's relatively large wouldn't surprise me at all. In fact, I've seen it before.
     
  17. nutzo

    nutzo [H]ardness Supreme

    Messages:
    6,115
    Joined:
    Feb 15, 2004
    Backup, Backup, Backup. No need to buy bitcoins.

    Make sure you do test restores.
    Virtualize all your servers. Makes them easy to restore, assuming you have a proper backup application.
    Even if you lose a complete server, it's easy to restore them to another server.
    Even better, do a D2D2T (disk to disk to tape) backup with a product that just backs up the changes. That way you can take snapshots ever couple hours.

    If you have more data than will fit on a single USB drive, then you should be looking at tape, preferably a tape changer.
    Make sure you have 2 copies off-site. If you lose your computer room, you don't want to lose everything due to a bad tape.

    Currently takes me 7 LTO-6 tapes to backup all the servers in my office. I get about 4TB per tape with compression, so that's around 28TB.

    It would take a few days to restore everything, but that's largely due to the 1GB Ethernet limit. Plan to upgrade the servers to 10GBit next year.
     
    Solhokuten likes this.
  18. c3k

    c3k [H]ard|Gawd

    Messages:
    1,392
    Joined:
    Sep 8, 2007
    Hmm, $23,000 in Bitcoins??? Was that TODAY'S Bitcoin equivalent in USD, or a from a few weeks ago? Because, if it's from a few weeks ago, that ransom could be up to $500,000 or more. ;)
     
  19. Solhokuten

    Solhokuten Gawd

    Messages:
    928
    Joined:
    Dec 9, 2009
    This 100%, but If you can afford offsite replication then that will be a nice alternative to dealing with tapes.
     
  20. geok1ng

    geok1ng [H]ard|Gawd

    Messages:
    1,944
    Joined:
    Oct 28, 2007
    if the data was not worth of regular backups, it is not worth paying ransom .
    what happened to the american policy of not negotiating with kidnappers on hostage situations?
     
  21. dandirk

    dandirk [H]ard|Gawd

    Messages:
    1,711
    Joined:
    Jun 5, 2004
    They don't really say how large the county is... I work in one with over 1k servers and that is not counting non-prod.

    As others have said part of it is 1 server = 1 service practice.

    Though I would say a larger factor is just their needs. People don't give government enough credit at times, they have very complex needs because they literally do a little bit of everything. Tons of niche apps and services from software that control street lights to geo-mapping for construction/taxes and planning, jail systems, multi media etc etc the list goes on and on.

    People in the medical field talk about how special their systems have to be, county governments generally have to do that AND everything else as well.

    At least in my county they aren't overpaying and underworking either.
     
  22. gman

    gman [H]ard|Gawd

    Messages:
    1,886
    Joined:
    Jan 24, 2001
    Well sq miles it's bigger than Atlanta. Population of metro area around 1 million. I assume 500 servers is about right.

    They did not pay the ransom. Heard on news this morning they are working it out internally. I live here by the way.
     
  23. nutzo

    nutzo [H]ardness Supreme

    Messages:
    6,115
    Joined:
    Feb 15, 2004
    Should have added that into my list.
    But offsite replication assumes you have a reasonably fat pipe and don't have a massive amount of data.
    I have enough daily change in my 28TB of data, that it would overwhelm my 100GB internet connection.
     
  24. lcpiper

    lcpiper [H]ardForum Junkie

    Messages:
    8,645
    Joined:
    Jul 16, 2008
    If they caught this fast, most of it could have been fixed just by restoring a storage snapshot. The rest by backups ..... if they have backups :unsure:

    Ahh, they did say they could still restore from backup.

    Now, as long as the backups weren't taken after the encryption attack.
     
  25. Exavior

    Exavior [H]ardForum Junkie

    Messages:
    8,794
    Joined:
    Dec 13, 2005
    buy $10,000 worth and just sit on them for a day or two and you might be at the $23,000
     
  26. painintheworld

    painintheworld [H]Lite

    Messages:
    64
    Joined:
    Jun 5, 2007
    That is wayyy off. Metro area is north of 2.6 million people. https://en.wikipedia.org/wiki/Charlotte_metropolitan_area
     
Tags: