Initframfs on boot

auntjemima

auntjemima

[H]ard DCOTM x2
Joined
Mar 1, 2014
Messages
12,141
I have a 4p server running in my basement. I use a GUI for running BOINC on it. I control it with TeamViewer.

Two days ago the system restarted twice. It didn't indicate why, but when booting in all programs were no longer running and it was on the desktop. My ASSUMPTION was a reboot. Then last night it was offline on TV. I restarted it manually a few times and it never made it to the OS, atleast TV didn't register it.

So, today I went down and hooked up a monitor and keyboard. When starting the system comes up it hits grub but no matter which option I pick, there are four, the system boots to a prompt of <initramfs>.

If this was windows (go easy on me), I would boot up to a windows usb and repair the OS... And before the full shut down I would have just checked eventviewer for hints.

Where do I go from here in Linux? I assume something needs to be rebuilt. If I throw this drive into an enclosure will windows read it just like Linux will read an NTFS partition?

It's fairly important that I repair this install.

Edit: big oof, forgot the OS. It's running Ubuntu 17.10.
 
Windows won't likely read your ext4? partitions no.

I would boot of an ubuntu live usb/cd and see if you can read some logs.

You can try booting into Ubuntu recovery mode as well.
https://wiki.ubuntu.com/RecoveryMode

The easiest fix though is probably a live boot... mount your system drive.... then use chroot. From their you could rebuild your initfram or whatever else you need to do.

So boot from a live media... then run
lsblk
So you can figure how which drive houses your install.
Then mount it with...
sudo mount /dev/sdx /mnt/system
Mount your drive where x is the right drive of course and you can give the mount folder whatever name you want.
Then use chroot to mount your system (pretty sure chroot would be on an ubuntu live image but if not you should able to install it under live anyway) Myself I keep a blackarch live stick around for these type of things.... and for when a family member calls cause they can't remember their win 10 password or something stupid. (really I think I have used chntpw a few times this year already lol)
chroot /mnt/system

From their you can run your package managers updates ect and see if that solves the issue. You can also manually rebuild your init... go through the logs and figure out what is really wrong ect.
http://manpages.ubuntu.com/manpages/cosmic/man8/live-update-initramfs.8.html

I would think chrooting in and running updates and rebuilding your initramfs would likley take care of your issue. Either way check the logs so you know what happened.
 
Boot live media.

Sudo lsbsk, work out what drive is your root partition and run fsck (Linux equivalent of chkdsk). Also check your /home partition if you run a separate /home partition.

Being a server I recommend running LTS variants of Ubuntu.
 
If the above doesn't highlight any physical hard disk issues you may need to fix broken packages/dependencies.

sudo apt update --fix-missing

sudo apt install -f

Kinda sounds like a hardware failure though as I'd be surprised if the OS just rebooted with no real reason. Even updates don't force reboots without intervention.
 
  • Like
Reactions: ChadD
like this
Good point Mazz.... does sound like it could be hardware. Never hurts to find the logs and track your issue down with the google box.

Live media and mount the drive chroot.

journalctl -b
and
dmesg

Find out what actually happened if you can.
 
Totally agreed re: Log files ChadD. Logging under Ubuntu (most likely Linux in general) is really good, shouldn't be too hard to go through the logs and see exactly where things went wrong.

Weird as it sounds as if part of the OS has simply gone missing?!
 
Thanks a lot guys! Just getting up but I will mess with this for sure. So will

journalctl -b
and
dmesg

Show me log files? I have no idea where to check them on Linux.
 
auntjemima said:
Thanks a lot guys! Just getting up but I will mess with this for sure. So will

journalctl -b
and
dmesg

Show me log files? I have no idea where to check them on Linux.
Click to expand...

Most distros now use systemd so most logs are now kept in systemds non text format. It isn't hard to pull them up and it makes it a bit easier to find what you need imo. You will still find some text logs in /var/log and if you have a non systemd distro I would imagine that is where you would find the old dmsg log files ect.

journalctl is what you want to use to view most boot logs.
journalctl -b
Should give you the boot log from the last boot if your chrooted in that should what you are looking for anyway. If not you can use an offset to view the log from priror boots.... so....
journalctl -b -0
would show you the current boot or last boot.
journalctl -b -1
would show you the boot before the last. (useful if you can say boot to a terminal or something but want to see why your system isn't going to a window manager or something)
From there you can go as deep as required. Might be useful to go journalctl -b -2 to see the last "good" boot... so you can rule out perhaps errors that are normal (generic standard cache or gpu errors that are not issues) from not normal errors.

Other somewhat useful options....
journalctl --since "120 min ago"
or
journalctl --since "2019-01-01 00:00:00"

So if you wanted to view the logs for something like say systemd itself.... you can type
systemctl /usr/lib/systemd/systemd --since "2880 min ago"
That should return the systemd logs for the last 2 days. use the --since or you'll start back at the start of logs likely months ago. ;)

If you suspect a specific issue with a service you can use
journalctl -u upower.service --since "2880 min ago"

If you get things half running again you can check running services with
systemctl list-units --type service
Then you can check logs on any services that are active... or loaded and extied ect.

Hope it helps... systemd logging. Some love it some hate it. lol
 
Thanks for all the help! I noticed a superblock issue on a video I took of the boot up. Found the following on the internet and ran it...

fsck.ext4 -p -b 884736 -B 4096 /dev/sda1

Solved the boot issue.
 
This happened after a random reboot. I still don't know the cause of that but since I tried to boot up and got to initramfs about 45 times, I am sure it won't be easy to find in log files.
 
auntjemima said:
This happened after a random reboot. I still don't know the cause of that but since I tried to boot up and got to initramfs about 45 times, I am sure it won't be easy to find in log files.
Click to expand...

journalctl --list-boots

this will list all the logged boots on your system

the last few of mine for instance....
-4 0cfb11defb394095bf8b93f799babcd3 Wed 2019-07-17 08:32:30 CDT—Thu 2019-07-18 14:21:52 CDT
-3 3f2a8d1e28024e2dbb37070fa631cc34 Thu 2019-07-18 14:22:09 CDT—Thu 2019-07-18 14:25:28 CDT
-2 244764d429c7425faa3ca1344294cffa Thu 2019-07-18 14:25:44 CDT—Thu 2019-07-18 14:33:46 CDT
-1 fab579d7cd434394a0a9be3c266810ae Thu 2019-07-18 16:04:36 CDT—Thu 2019-07-18 16:05:03 CDT
0 f5590ad4e534455087a3561b9ee9e3bc Thu 2019-07-18 16:18:25 CDT—Fri 2019-07-19 00:09:41 CDT
lines 137-159/159 (END)

As you can see on my wifes box 159 boots since I built it for her
-158 4c816927b02b4498a0ee92d1235c160a Tue 2019-02-26 16:01:07 CST—Tue 2019-02-20

With that you can narrow down which boot log you want to see.
Then use
journalctl -b -50
Or whichever boot number corresponds to the time you believe it rebooted itself.
 
Hopefully it was just a power hickup. The bad superblock stuff tends to follow power loss. 884736 is what the cube of 96 or something like that it's one of ext4s superblocks.
 
ChadD said:
Hopefully it was just a power hickup. The bad superblock stuff tends to follow power loss. 884736 is what the cube of 96 or something like that it's one of ext4s superblocks.
Click to expand...

I don't know WHY I chose that superblock. I just followed the suggestion I found. I did look up some other superblock information but I don't understand why I chose that one or which I should be using.

Edit: this did happen after a system reboot. An unintended one.
 
auntjemima said:
I don't know WHY I chose that superblock. I just followed the suggestion I found. I did look up some other superblock information but I don't understand why I chose that one or which I should be using.

Edit: this did happen after a system reboot. An unintended one.
Click to expand...

I'm not a file system expert. But superblocks keep redundant copies of all other superblocks meta data for the file system. So the command you ran if you don't know exactly what it did.
fsck.ext4 -p -b 884736 -B 4096 /dev/sda1
you specified ext4 obviously... -p (automatically repair anything that can be repaired) -b 884736 (-b specify superblock instead of the default location 0 sblock) -B 4096 (byte size 4096... this will tell mke2fs to look for superblocks spaced 4096 apart... that tends to be the standard block size most people format) and drive location.

It's been awhile but I believe you can search superblocks with
mke2fs -n /dev/drive
and restore with
e2fsck -b blocknumber /dev/drive

I am pretty sure what you ran... fsck basically did exactly that. You pointed it at a superblock... gave it a block size to find other blocks and it restored the meta data that was corrupted.

So anyway ya bottom line doesn't matter which superblock you choose they should all have copies of the drives meta stuffs... and fsbk can repair ones that are messed up by pointing it to pretty much any other one, and if that is messed it should be able to use the block size to find the next hopefully non corrupted sblock.
 
You must log in or register to reply here.
Back
Top