Image of McAfee Endpoint Encrypted Work Laptop Takes Down ZFS?

[Zarathustra[H]]

Hey all,

I had a rather odd thing happen to me today.

I've been in crunch mode at work, and thus been doing a lot of work from home. I have some applications on my work issued laptop necessary for what I am doing, so it becomes kind of a nuisance, with a lot of work on my home workjstation that has plenty of desktop real estate, and the teeny little laptop on the side.

In a moment of frustration (and in part procrastination) I had a lightbulb moment.

Why don't I image the laptop drive, copy it to my workstation, convert it to a .vdi virtualbox image, and run it from the same machine! It would be so much more efficient, and I would get done so much faster!

The laptop has some sort of full disk encryption by McAfee which pops up a username and password dialog before the windows boot loader starts loading windows 7. I figured this wouldn't be a problem, as it would just allow me to enter the username and password when loading my VM, and it wouldn't be a security concern for IT either, because hey, the image would be just as encrypted as the hardware disk, right?


Attempt 1: (ZFS on Linux on Workstation Mirror)

I grab my handy little Ubuntu bootable USB, boot up my work laptop in Ubuntu, and from there DD an image of the encrypted disk in it's entirety to a USB hard drive dock I have connected to it.

I then reconnect the hard drive dock to my workstation, and proceed to copy the image to my small two disk ZFS on Linux mirror on my workstation. Once copied, I use the vbox manage convertdd application to convert it to a virtualbox compatible .vdi.

This is when all hell breaks loose. About 26GB into the convert process my previously perfect small mirror all of a sudden has a metric ton of read errors, perfectly matched in quantity on both drives in the mirrored set. zpool status recommends I delete the image file and restore from a backup. I dutifully delete the raw dd image and then do a scrub of the disks. No problems detected, 0 repairs. I've since copied multiple gigabytes to and from the mirror with no issues at all.

Odd, I think. A little bit concerned that my setup is starting to act up, I chuck it up to some weird random read or write failure somewhere in the chain, make a mental note to keep an eye on the health of my little local mirror and decide to start over.


Attempt 2: (FreeNAS server)

I reboot the laptop up using my trusty ubuntu USB stick, and this time mount a share on my FreeNAS server (12 disks 2 RAIDz2 vdevs in one pool). Again, I proceed to DD the full encrypted disk, but this time to my FreeNAS server.

First attempt fails after 20 some GB due to some sort of timeout. Weird I think. I reboot the laptop, and try again.

This time the image completes. I go to the share on my workstation, and again, initialize the vdi conversion process. The image in total is about 167GB, but the conversion process stops without error resulting in a 75GB file. I see this and am surprised, but take it as a positive sign. Some sort of magic in the conversion process due to sparse .vdi files, I think, instead of fully populated raw images.

Ready to test how everything will load in virtualbox, I set up my machine, add the converted vdi, and nothing. Not even a boot error. I get a black blank screen in my virtualbox console, that is not responding. A little disappointed, I assume that it is some sort of protective measure to prevent booting in case the wrong people get a hold of the laptop and try to steal data from it.

Disappointed I go to continue my annoying work from home task, switching back and forth between laptop and workstation, when I notice none of my network drives to FreeNAS are responding. Not only that, the FreeNAS server is not responding at all. Not to the web interface,m and not to pings...

My FreeNAS install runs in a virtual environment so I sign on to my ESXi server to investigate. I expect to find the FreeNAS guest running, but somehow having network issues, but instead I find it completely off. All the other guests are running, but FreeNAS is powered off.

I go to the logs, there are some cryptic unintelligible messages about a crash (and an offer for paid support from VMWare )

To be on the safe side I reboot the entire host, and bring all the guests back up again.

So, what do you guys think? Does McAfee Endpoint Encryption do something crazy to try to foil data thiefs, or is this the biggest coincidence of odd failures when tying to do the same thing on different systems?

Other than this one raw image file from from my work laptop, both my Workstation and my server have been rock stable since they were both built, 4 years and 1 year ago, respectively.

Thoughts?

 

[Zarathustra[H]]

This may have something to do with it.

 

[mwroobel]

Separate from issues you may have with the imaging, I have seen problems where images that were installed on boxes/laptops with a TPM chip didn't operate properly when moved to a non TPM box. What OS are you running? Just ssh or remote desktop into it if you have admin rights and open a big window on your desktop monitor.

 

[Zarathustra[H]]

Separate from issues you may have with the imaging, I have seen problems where images that were installed on boxes/laptops with a TPM chip didn't operate properly when moved to a non TPM box. What OS are you running? Just ssh or remote desktop into it if you have admin rights and open a big window on your desktop monitor.

That's a good point. Should have thought of that first

Desktop is running Linux Mint, Work Laptop is on Windows 7. Not sure if Remote Desktop is enabled on Laptop.

 

[Zarathustra[H]]

Either way, I'm less concerned about getting this to work. It was intended to make my life easier until done with work, but has already taken WAY more time than just putting up with the inconvenience would have

My concern is more something like this:

Have McAfee discovered a way to intentionally construct a malformed data block in such a way that it can not be stored on RAID (in general) or ZFS (in particular) storage arrays?

If they have, and are using it, is it only a matter of time until others figure this out?

What could the implications be? Intentionally malformed data segments coupled with DRM so I can't backup my media, for instance?

Or intentional sabotage of production systems by uploading files containing this malformed data in an attempt to harm them or take them down?

 

[TCM2]

Have McAfee discovered a way to intentionally construct a malformed data block in such a way that it can not be stored on RAID (in general) or ZFS (in particular) storage arrays?

If they have, and are using it, is it only a matter of time until others figure this out?

What could the implications be? Intentionally malformed data segments coupled with DRM so I can't backup my media, for instance?

Or intentional sabotage of production systems by uploading files containing this malformed data in an attempt to harm them or take them down?

No.

 

[Zarathustra[H]]

No.

Feel free to delve? This is a safe space

 

[TCM2]

That would be a bug in ZFS that's not even remotely likely.

Also

The image in total is about 167GB, but the conversion process stops without error resulting in a 75GB file. I see this and am surprised, but take it as a positive sign. Some sort of magic in the conversion process due to sparse .vdi files, I think, instead of fully populated raw images.

I detect too much magical thinking.

A possible explanation for this is that the disk is actually an SSD and the encryption driver is passing through TRIM commands, leading to zeroed blocks in the image that would normally not occur with encryption.

I don't know the properties of .vdi files, but a raw sparse file would still show as 167GB in size, while only allocation 75GB worth of disk blocks. Could be that .vdi actually compresses the image to 75GB on disk but uses some metadata to provide the "real" image size to the consuming application.

Anyway, it's more likely that your setup has some other problem rather than McAfee using some "malformed" data blocks. ZFS is a filesystem and stores what you throw at it. No user data can be malformed for any filesystem, otherwise it wouldn't be a filesystem but just trash.

 

[Zarathustra[H]]

Anyway, it's more likely that your setup has some other problem rather than McAfee using some "malformed" data blocks. ZFS is a filesystem and stores what you throw at it. No user data can be malformed for any filesystem, otherwise it wouldn't be a filesystem but just trash.

Normally I would agree here. It was the fact that it happened two times in a row on two separate systems, running separate implementations of ZFS in different setups (one a simple mirror under ZFS on Linux on my desktop using on board sata and one in OpenZFS for BSD on my FreeNAS server with a 12 disk RAID60 equivalent setup using two SAS HBA's) that made me a little suspicious.

Especially since both file systems (which have been perfectly healthy both before and since, with no issues during scrubs) magically suffered read errors which could not be corrected by parity/mirror approximately the same percentage of the way through the file.

A possible explanation for this is that the disk is actually an SSD and the encryption driver is passing through TRIM commands, leading to zeroed blocks in the image that would normally not occur with encryption.

Could TRIM commands saved into an image cause read errors and hard crashes when read outside that image?

I don't know the properties of .vdi files, but a raw sparse file would still show as 167GB in size, while only allocation 75GB worth of disk blocks. Could be that .vdi actually compresses the image to 75GB on disk but uses some metadata to provide the "real" image size to the consuming application.

Oracles vdi files don't have that weird file system behavior that VMWARE has, that quite frankly confuses the crap out of me. They rely on using whatever file system the host uses, thus sparsely allocated files start out as occupying 0 byte on disk, and then grow as used.

I think you might have misunderstood what I meant by this bit. I thought it was complete at this point, but it was actually the FreeNAS server that had gone offline and the operation never completed. (it was odd that Oracles conversion tools didn't give me an error message)


So I guess, to sum it up. I know coincidences happen, but the fact that two very different long term stable systems with different hardware and software had similar odd and unexplained problems with this file, and once removed, all problems disappeared, scrubs show nothing wrong, and several multi GB file reads and writes have been made to test without issue, suggests that something odd is going on.

 

[TCM2]

A TRIM command is not "saved into the image". A TRIM command gets sent to the disk when the filesystem deletes a file for example, and it tells the SSD to forget about those blocks. When reading back those blocks all you get is zeroes. From the SSD point of view it's different than writing a block of zeroes, because zeroes can be valid user data. Sending a TRIM enables the SSD to use that block internally as it sees fit for wear leveling or whatever, which it couldn't do with a block full of "user data zeroes".

Nothing of that matters when imaging an SSD. An image is just data, there are no commands in it that get interpreted by the filesystem or its underlying disk.

As for the main problem, the first thing to do is gather more information rather than coming up with outlandish explanations that are just not likely. Is there any information in the logs etc.?

Yes, the effects are weird, but that doesn't justify inventing weird causes.

 

[cantalup]

my work laptop is McAfee Endpoint Encrypted. I am really hate due on so many issue when updating to new version .
well. I am playing cheat by stopping the update program to launch since we will move to a new building that does not required those craps haha.

my laptop failed once. they need to reimage my laptop and copy back all myfile.
mcafee has a tool to convert from encrypted to un-enc again, but takes long. Mine was 3 days to recover.

true, you can not convert encryp to any VM on my understanding, You have to unencry the whole drive first!.
McAfee Endpoint Encrypted has some protection, I would assume.


Have fun with McAfee Endpoint Encrypted till you are tired hahah.

 

[Randolf]

here is a rough idea of what we do/use at work.

We use Storecraft ShadowProtect. Does a base image while booted into window and creates a password encrypted image. It can then convert into a VirtualBox boot environment for a head start restore or need to spin up a downed server in a hurry.

I use it for taking a clients clean fully configured setup, then building a golden image out of it. strip out all the drivers, etc, etc. Allow the image to be redeployed on new hardware with minimal issues.

but in your case the spinning it up into a VM is what you are really after. catch is their software at the moment only works with VirtualBox 4.something. Also it will not restore to a smaller drive with out a lot of fucking around with it in a VM.

 

[thedge]

I dont know what the issue is but my laptop is McAfee Endpoint encrypted and I dump images of it regularly to a ZFS pool for backup. I use R-Drive Image and just let it compress the image, nothing specific. I am doing it specifically for backup though, no conversion or reimaging process. My image is about 50GB currently.

Do you have any issues on your pool if you write a large file to it from a different system? You could have issues that only present on very large sequential writes.