Partition mystically deleted? (Linux)

[CC_Pirate]

Hi all

I have a NAS/home server running Ubuntu 14.04, with a 128GB Samsung SSD as root (OS) disk and a 4-disk mdadm array for storage. The system has been running fine for 4 years.

I've got a CRON script that regularly checks CPU temp, and shuts down the machine if temps exceed 80C. It also sends me an email before shutting down.

I woke up the other day to such an email. When I tried powering the system on again, I was greeted with a GRUB error screen saying something like

"No such partition found" with a GRUB "rescue mode" prompt.

I booted into an Ubuntu LiveCD, and `fdisk -l /dev/sda` shows an empty partition table for my root disk (the SSD). To emphasize, my boot and main partitions were on this disk, now they are suddenly gone..

I ran a SMART short test using `smartctl -t short /dev/sda` which completed without errors. The results can be seen here. All seems to be fine with the disk.

Anyone have advice on where to go from here? I wanna start attempting data recovery of some sort, but I'm not sure how to start. I have the data backed up of course, so its not the end of the world if I have to re-install, but I'd rather be sure the disk is healthy first. Otherwise I risk something similar happening all over again.

How this data-loss is related to the CPU overheating (or if it was), I have no idea..

Thanks for all help!

 

[darkeen]

Not sure about the cpu relation, but how to start recovery:
use ddrecovery to create an image of the failed drive
use scalpel (https://github.com/machn1k/Scalpel-2.0) or similar to carve up the image and dump out files

 

[CC_Pirate]

Thanks for the tip.

I have the data backed up so thats not actually my main concern.
I'm more puzzled as to what happened (how partitions suddenly disappeared), and worried that the SSD has failed, so that something similar can happen in the future if I re-install the OS.

Looking at the output of smartctl again, I see that the disk has been powered on for 10.000 hrs, has 8192 re-allocated sectors. Is this bad?
Also, I see "Wear_Leveling_Count" is at 1, or 1%, which is the worst possible value. I've read though that this does not necessarily mean it is bad - its just that it has warranty is out, and that it may hold data for shorter amount of time in powered off state..
Any thoughts on this?

 

[CC_Pirate]

Update: Reallocated sector count of 8192 is really high.

Quoting Samsung (its a Samsung SSD), this number "represents the number of sectors that have been moved as a result of a read error, write error, or a verification error".

I'm starting to think this SSD is on its deathbed..

Oh, and the 10.000 powered-on hours reported must be wrong, thats just a year and a half. This NAS has been running 24/7 since dall 2011

 

[darkeen]

That does indeed seem like a failed SSD, best I can guess is that swapping/logging killed it.

 

[CC_Pirate]

That does indeed seem like a failed SSD, best I can guess is that swapping/logging killed it.

Good thought, thanks.
What can one do to prevent excessive swapping? I had the standard partition-setup that is set up when installing Ubuntu to a single disk without LVM, all defaults. No other OSes.

The system has 8GB memory, which I've never seen to be eaten up. Would 16GB seize the need for a swap-partition entirely?

And as for logging, what are "best practices", and are there any other recommendations when installing OS/software on an SSD? This was afterall a nas/server and did a lot stuff 24 hours a day, which of course results in log files constantly being written to disk..

The sensible thing would be to have in-memory logging, being dumped to disk every minute or so..

Interested in suggestions here

 

[cantalup]

That does indeed seem like a failed SSD, best I can guess is that swapping/logging killed it.

I disagree logging kills SSD unless logging sending numerous writing that is illogical in OS scenario

swapping:
when the system uses swap partion/file, the system would very slowlyyyyyyyyyyyyyyyy in response. admin or user will know for sure.
to know swapping are used or not, many tools can be used. I use top to know, got directly to proc live system.

 

[cantalup]

Good thought, thanks.
What can one do to prevent excessive swapping? I had the standard partition-setup that is set up when installing Ubuntu to a single disk without LVM, all defaults. No other OSes.

The system has 8GB memory, which I've never seen to be eaten up. Would 16GB seize the need for a swap-partition entirely?

And as for logging, what are "best practices", and are there any other recommendations when installing OS/software on an SSD? This was afterall a nas/server and did a lot stuff 24 hours a day, which of course results in log files constantly being written to disk..

The sensible thing would be to have in-memory logging, being dumped to disk every minute or so..

Interested in suggestions here

just a hint:
when swapping is used intensively: your system would be slow trashly.

to know swap acitivity, this is an example on one of my sytem:
[centos@remote ~]$ cat /proc/swaps
Filename Type Size Used Priority
/dev/dm-1 partition 1470460 0 -1
[centos@remote ~]$ vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 0 813736 764 130880 0 0 0 0 21 13 0 0 100 0 0

focus on used under /proc/swaps
or
si and so under vmstat command
si -> swap in: memory amount to swap
so-> swap out: amount out to memory

the best practice, more memory is better .

check on vmstat first to know is swap was really invoked!

 

[cantalup]

Update: Reallocated sector count of 8192 is really high.

Quoting Samsung (its a Samsung SSD), this number "represents the number of sectors that have been moved as a result of a read error, write error, or a verification error".

I'm starting to think this SSD is on its deathbed..

Oh, and the 10.000 powered-on hours reported must be wrong, thats just a year and a half. This NAS has been running 24/7 since dall 2011

that number is in hex value, you need to convert to Decimal valu

 

[darkeen]

Good thought, thanks.
What can one do to prevent excessive swapping? I had the standard partition-setup that is set up when installing Ubuntu to a single disk without LVM, all defaults. No other OSes.

The system has 8GB memory, which I've never seen to be eaten up. Would 16GB seize the need for a swap-partition entirely?

Linux has a lot of run time variables. One of which is vm.swappiness which controls how aggressive the system will swap. Setting vm.swappiness to 0 will prevent the system from swapping unless absolutely necessary (100 is swap like mad, I has most of my systems set to 1). Do a quick google to see how to set it in ubuntu.

I would leave the swap partition just because some things do need it (and vm.swappiness works pretty damn well)

And as for logging, what are "best practices", and are there any other recommendations when installing OS/software on an SSD? This was afterall a nas/server and did a lot stuff 24 hours a day, which of course results in log files constantly being written to disk..

The sensible thing would be to have in-memory logging, being dumped to disk every minute or so..
Interested in suggestions here

Logging is a more difficult issue because different programs log differently. Best thing to do is just turn off anything you don't need. That said logging is low IO so it should be a much lower burden on the ssd.

 

[CC_Pirate]

Thanks, good input here.
I'll try to set swappiness to 1. Logging I don't think I'll worry about..

 

[darkeen]

Hope it works out for you. I've had a few old ocz ssds working in that way since 2012 or so (knock on wood)

 

[CC_Pirate]

that number is in hex value, you need to convert to Decimal valu

Well in that case the number is 33170, which crazy high. The SSD is f*****

 

[CC_Pirate]

Hope it works out for you. I've had a few old ocz ssds working in that way since 2012 or so (knock on wood)

Thanks.

From what I've just experienced, running SMART monitor tools (`smartctl`) every now and then to check drive health is crucial - especially if used in a server environment. At least glance at the wear-level and re-allocated sector count to see if it's rising.

When my system is back up I'm gonna write a CRON script to email me a SMART report every couple weeks or so

 

[cantalup]

Thanks, good input here.
I'll try to set swappiness to 1. Logging I don't think I'll worry about..

check your swap usage as I mentioned on my prev post before doing further tweak..

do not minimize logging since very usefull when something happens
use whatever default logging verbose.

I am not worry on swap value since this has to do
wheneve memory is limited, swap is the only way..

if you force to let swap for the last extreme condition, your system would be unstable..
memory<->swap.


once again, check your swap statistical usage...

 

[cantalup]

Hope it works out for you. I've had a few old ocz ssds working in that way since 2012 or so (knock on wood)

the key is., how many writes that you put on your SSD
the SSD spec will tell you the number..

I have 32G samsung SSD sata 3G, still runnig esxi with light writing due on most VMs are having ramdisks .

 

[michalrz]

Your SSD looks damaged. I think you had a fire in your case to be honest
I would seriously take a look at everything, both sides of the mobo, inside the PSU, check the wires that go into the front of the case.
Which sensor did you poll in your script? Could you put some of the relevant code?

Also, I use iotop to see what's up with reads and writes. It shows usage per process, among other things.

It would be cool to run some partition scanner and rebuilder on the SSD and check how much data was lost.