IT Director Job Opening in NC - Apply Now

FrgMstr

Just Plain Mean
Staff member
Joined
May 18, 1997
Messages
52,227
Mecklenburg County in North Carolina, with a population of around 900,00 good folks, is taking it right in the pooper today thanks to some enterprising hackers. It seems that someone left the door open and has allowed 48 of its 500 servers to be infected with LockCrypt Ransomware. County officials are now kicking around paying the bandits $23,000 in Bitcoin. Guess they had better get those other 452 servers mining quickly! Thanks cageymaru.


“We are open for business, and we are slow, but there’s no indication of any data loss or that personal information was compromised,” Diorio said. Hackers typically don’t steal data but encrypt it, placing it out of reach of the owners until ransom is paid.

The county has been in contact with the hackers, she said. “We have not made a decision whether or not to make the payment,” Diorio said. The county hopes to make a decision by the end of Wednesday.
 
Joined
Mar 18, 2013
Messages
3,719
I would love to be the new IT director for an organisation that already has a running chainsaw installed in their collective ass hole.

I mean, who wouldn't?

I spent most of today cutting dovetails on 18" hewn logs. It's very soothing and it's a far better use for a chainsaw.
 
Last edited:

MavericK

Zero Cool
Joined
Sep 2, 2004
Messages
31,106
Seems cheaper to just pay the ransom rather than pay someone to clean up this shitstorm. But then, pay someone afterwards to lock your shit down.

Though, as we found out when a client brought in a ransomware machine and wanted to pay - it's hard to actually buy Bitcoin. Not sure if that's changed, this was back when it was around $4k and rising.
 

RogueTadhg

[H]ard|Gawd
Joined
Dec 14, 2011
Messages
1,527
On the bright side - You know going into the job that that shit hit the fan.

I sort of want to take on the challenge. And hey, it's where Charlotte, NC is located - which is a pretty large freaky city. So it can't possibly be too bad paying.

CIO Keith Gregg. Tisk.
 
Last edited:

jojo69

[H]F Junkie
Joined
Sep 13, 2009
Messages
10,874
Seems cheaper to just pay the ransom rather than pay someone to clean up this shitstorm. But then, pay someone afterwards to lock your shit down.

Though, as we found out when a client brought in a ransomware machine and wanted to pay - it's hard to actually buy Bitcoin. Not sure if that's changed, this was back when it was around $4k and rising.


sign up at Coinbase

wire in USD

buy Bitcoin

what hard?
 

MavericK

Zero Cool
Joined
Sep 2, 2004
Messages
31,106
sign up at Coinbase

wire in USD

buy Bitcoin

what hard?

I recall at the time it required a lot of verification beforehand, and would have taken more time than there was left on the ransomware. The client also waited awhile to get the machine to us so it left us with very little time to act.
 

jojo69

[H]F Junkie
Joined
Sep 13, 2009
Messages
10,874
I recall at the time it required a lot of verification beforehand, and would have taken more time than there was left on the ransomware. The client also waited awhile to get the machine to us so it left us with very little time to act.

ahh, that makes sense

yes, Coinbase complies with all US KYC/AML requirements, which can take a few days

localbitcions would have been your next bet for speed, or post up in here and make a deal
 

Dead Parrot

2[H]4U
Joined
Mar 4, 2013
Messages
2,831
Why does a county have 500 servers to begin with?
If they are a Microsoft shop, very believable. No matter how much Microsoft crows about how one server can do many things, it rarely works out that way. Need to restart DHCP service, DNS takes a crap. Restart Print Server after driver updates, OS restart needed because vendor 3's driver requires it. It is just simpler to have many servers, each doing one thing. A better question is how many of those 500 are physical machines vs virtual. Restoring virtual servers is often as simple as reverting to a recent snapshot.

Sounds like they had a backup policy since they said that getting the unlock code and testing it would have taken about as long as restoring the data from backups. Hope they have done restore tests in the recent past. "Where's the DVD with the restore software?, Fred's house? What do you mean he's on vacation for two weeks!"
 

tmillszero1

Limp Gawd
Joined
May 11, 2011
Messages
223
We are actually a very large county incorporating many cities including Charlotte. Not really a surprise for 500 servers. Actually its a small number considering...
Why does a county have 500 servers to begin with?

Just a "squirrel!" thought... And Merck's global prod datacenter is here in the same county and got spanked too with a crypto virus! So it boils down to budget, experience and prioritization.
 
Last edited:

painintheworld

Limp Gawd
Joined
Jun 5, 2007
Messages
143
We are actually a very large county incorporating many cities including Charlotte. Not really a surprise for 500 servers. Actually its a small number considering...


Just a "squirrel!" thought... And Merck's global prod datacenter is here in the same county and got spanked too with a crypto virus! So it boils down to budget, experience and prioritization.

I live about 100 miles north and west of Uptown and travel there several times per month. It is unreal how much the entire metro area has grown in the last 25 years.

I really dig Charlotte and have thought of buying a place in the suburbs for when we are in town.

For the poster above that mentioned Charlotte being freaky...that would be Asheville :)
 

Dan_D

Extremely [H]
Joined
Feb 9, 2002
Messages
58,922
Why does a county have 500 servers to begin with?

You'd be surprised how many servers a state may have. 500 in a specific county that's relatively large wouldn't surprise me at all. In fact, I've seen it before.
 

nutzo

Supreme [H]ardness
Joined
Feb 15, 2004
Messages
7,380
Sounds like they had a backup policy since they said that getting the unlock code and testing it would have taken about as long as restoring the data from backups. Hope they have done restore tests in the recent past. "Where's the DVD with the restore software?, Fred's house? What do you mean he's on vacation for two weeks!"

Backup, Backup, Backup. No need to buy bitcoins.

Make sure you do test restores.
Virtualize all your servers. Makes them easy to restore, assuming you have a proper backup application.
Even if you lose a complete server, it's easy to restore them to another server.
Even better, do a D2D2T (disk to disk to tape) backup with a product that just backs up the changes. That way you can take snapshots ever couple hours.

If you have more data than will fit on a single USB drive, then you should be looking at tape, preferably a tape changer.
Make sure you have 2 copies off-site. If you lose your computer room, you don't want to lose everything due to a bad tape.

Currently takes me 7 LTO-6 tapes to backup all the servers in my office. I get about 4TB per tape with compression, so that's around 28TB.

It would take a few days to restore everything, but that's largely due to the 1GB Ethernet limit. Plan to upgrade the servers to 10GBit next year.
 

c3k

2[H]4U
Joined
Sep 8, 2007
Messages
2,283
Hmm, $23,000 in Bitcoins??? Was that TODAY'S Bitcoin equivalent in USD, or a from a few weeks ago? Because, if it's from a few weeks ago, that ransom could be up to $500,000 or more. ;)
 

Solhokuten

[H]ard|Gawd
Joined
Dec 9, 2009
Messages
1,389
Backup, Backup, Backup. No need to buy bitcoins.

Make sure you do test restores.
Virtualize all your servers. Makes them easy to restore, assuming you have a proper backup application.
Even if you lose a complete server, it's easy to restore them to another server.
Even better, do a D2D2T (disk to disk to tape) backup with a product that just backs up the changes. That way you can take snapshots ever couple hours.

If you have more data than will fit on a single USB drive, then you should be looking at tape, preferably a tape changer.
Make sure you have 2 copies off-site. If you lose your computer room, you don't want to lose everything due to a bad tape.

Currently takes me 7 LTO-6 tapes to backup all the servers in my office. I get about 4TB per tape with compression, so that's around 28TB.

It would take a few days to restore everything, but that's largely due to the 1GB Ethernet limit. Plan to upgrade the servers to 10GBit next year.

This 100%, but If you can afford offsite replication then that will be a nice alternative to dealing with tapes.
 

geok1ng

2[H]4U
Joined
Oct 28, 2007
Messages
2,129
if the data was not worth of regular backups, it is not worth paying ransom .
what happened to the american policy of not negotiating with kidnappers on hostage situations?
 

dandirk

[H]ard|Gawd
Joined
Jun 5, 2004
Messages
1,832
Why does a county have 500 servers to begin with?

They don't really say how large the county is... I work in one with over 1k servers and that is not counting non-prod.

As others have said part of it is 1 server = 1 service practice.

Though I would say a larger factor is just their needs. People don't give government enough credit at times, they have very complex needs because they literally do a little bit of everything. Tons of niche apps and services from software that control street lights to geo-mapping for construction/taxes and planning, jail systems, multi media etc etc the list goes on and on.

People in the medical field talk about how special their systems have to be, county governments generally have to do that AND everything else as well.

At least in my county they aren't overpaying and underworking either.
 

gman

[H]ard|Gawd
Joined
Jan 24, 2001
Messages
1,887
Well sq miles it's bigger than Atlanta. Population of metro area around 1 million. I assume 500 servers is about right.

They did not pay the ransom. Heard on news this morning they are working it out internally. I live here by the way.
 

nutzo

Supreme [H]ardness
Joined
Feb 15, 2004
Messages
7,380
This 100%, but If you can afford offsite replication then that will be a nice alternative to dealing with tapes.

Should have added that into my list.
But offsite replication assumes you have a reasonably fat pipe and don't have a massive amount of data.
I have enough daily change in my 28TB of data, that it would overwhelm my 100GB internet connection.
 

lcpiper

[H]F Junkie
Joined
Jul 16, 2008
Messages
10,611
Why does a county have 500 servers to begin with?

Forty-eight of about 500 county computer servers were affected.

If they caught this fast, most of it could have been fixed just by restoring a storage snapshot. The rest by backups ..... if they have backups :unsure:

Ahh, they did say they could still restore from backup.

Now, as long as the backups weren't taken after the encryption attack.
 

Exavior

[H]F Junkie
Joined
Dec 13, 2005
Messages
9,700
Hmm, $23,000 in Bitcoins??? Was that TODAY'S Bitcoin equivalent in USD, or a from a few weeks ago? Because, if it's from a few weeks ago, that ransom could be up to $500,000 or more. ;)

buy $10,000 worth and just sit on them for a day or two and you might be at the $23,000
 
Top