Jag vill få liv i en 13 år gammal server.

Bring my (software) raid1 up on Ubuntu after failure
2010-04-10

So, sometimes my raid1 arrays fail. I don’t know why, but it seems to be after software updates or maybe sometimes after hibernate or sleep (or a combination).

To fix this I will boot the Ubuntu 9.10 CD in “try before installing” mode and install mdadm: “sudo apt-get install mdadm” (this will bring postfix with it).

“less /proc/mdstat” will list all raid devices, it will be empty at first

“sfdisk -l” will list all partitions on all hard drives:

Disk /dev/sda: 121601 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System
/dev/sda1 * 0+ 11 12- 96358+ fd Linux raid autodetect
/dev/sda2 12 121600 121589 976663642+ 5 Extended
/dev/sda3 0 - 0 0 0 Empty
/dev/sda4 0 - 0 0 0 Empty
/dev/sda5 12+ 740 729- 5855661 82 Linux swap / Solaris
/dev/sda6 741+ 6819 6079- 48829536 83 Linux
/dev/sda7 6820+ 121600 114781- 921978351 fd Linux raid autodetect
Disk /dev/sdb: 121601 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System
/dev/sdb1 * 0+ 11 12- 96358+ fd Linux raid autodetect
/dev/sdb2 12 121600 121589 976663642+ 5 Extended
/dev/sdb3 0 - 0 0 0 Empty
/dev/sdb4 0 - 0 0 0 Empty
/dev/sdb5 12+ 740 729- 5855661 82 Linux swap / Solaris
/dev/sdb6 741+ 6819 6079- 48829536 83 Linux
/dev/sdb7 6820+ 121600 114781- 921978351 fd Linux raid autodetect

This will show me that /dev/sda1 and /dev/sdb1 is my small Linux raid autodetect , which I know contain /boot. /dev/sda7 and /dev/sdb7 is my large array which contain everything else. The /dev/sdx6 drives are actually unused, even if they are marked with “83/Linux”.

“mdadm –assemble /dev/md0 /dev/sda1 /dev/sdb1” and “mdadm –assemble /dev/md1 /dev/sda7 /dev/sdb7” will bring the arrays online.

Edit: “mdadm –assemble –scan” will also work, even when “sfdisk -l” does not show anything (that just happened…)

A brief look in /proc/mdstat reveals that the first drive of the md1 array is degraded:

Personalities : [raid1]
md1 : active raid1 sdb7[1]
921978240 blocks [2/1] [_U]
md0 : active raid1 sda1[0] sdb1[1]
96256 blocks [2/2] [UU]
unused devices:

Now, this command will also work to find out the state of an array: “mdadm –detail /dev/md1”, the state for me is “clean, degraded”.

To resync the raid array I issue the command: “mdadm –add /dev/md1 /dev/sda7”

Another look in /proc/mdstat reveals that the array is being reconstructed and will by in sync in about 2.5 hours:

Personalities : [raid1]
md1 : active raid1 sda7[2] sdb7[1]
921978240 blocks [2/1] [_U]
[>....................] recovery = 0.1% (1504512/921978240) finish=142.7min speed=107465K/sec
md0 : active raid1 sda1[0] sdb1[1]
96256 blocks [2/2] [UU]

When the recovery has finished, a reboot will start the computer from hard drive again.

To rebuild an array where one disk failed competely, please see this post. This and this post also contain useful examples about how to set up software raid.

I always forget if I used LVM or not when I installed the system (I did). To fiddle with LVM I will:
– Install lvm2: “sudo apt-get install lvm2”

Then “vgscan -v -v” will scan all partitions for LVM volume groups. lvscan and pvscan will do the same for logical and physical volumes.

Recover encrypted Ubuntu home directory (ecryptfs) from a live CD
2010-12-11

So, my Ubuntu 9.10/Karmic installation died. Again. Probably some software update that screwed up my fairly complicated install.

I have a raid1 (mirrored disks). On top of that I run LVM with one volume group, vg0, that has two logical volumes, one for home and one for root (everything else). Additionaly /boot is NOT on LVM but lies besides everything, still raided though.

Booting stopped. My home directory is encrypted so I can’t just copy everything out and reinstall. Bugger. Of course I can copy the encrypted data and hope to get it running in the new install, but that doesn’t feel safe.

Install tools necessary for RAID and LVM:
sudo apt-get install lvm2
sudo apt-get install mdadm

Bring up the RAID array as I always do:
sudo mdadm --assemble --scan
sudo less /proc/mdstat

Check if the volume group and the logical volumes are active.
sudo vgdisplay
sudo lvdisplay

If necessary one could make the volume group active with
sudo vgchange -a y

Create somewhere to mount the logical volumes
sudo mkdir /mnt/homevol
sudo mkdir /mnt/rootvol

Do not stand in any of the newly created directories, or you will not see the mounting getting done:
sudo mount -t auto /dev/vg0/homevol /mnt/homevol/
sudo mount -t auto /dev/vg0/rootvol /mnt/rootvol

Hmm, maybe /dev/mapper/vg0-rootvol and /dev/mapper/vg0-homevol was used here instead, I don’t remember.

Next, we’re about to prepare for chrooting in order to make the file system look like the file system on the machine when it was running without the live CD.

We need to mount some stuff started on the live CD into the directory we’re about to chroot into. These things must be available after chroot.

sudo mount --bind /dev /mnt/rootvol/dev/
sudo mount --bind /dev/shm /mnt/rootvol/dev/shm
sudo mount --bind /proc /mnt/rootvol/proc
sudo mount --bind /sys/ /mnt/rootvol/sys

Also make the home volume available under the to be root.
sudo mount --bind /mnt/homevol/ /mnt/rootvol/home/

Now it is time to change the root.
sudo chroot /mnt/rootvol/

Now the file system looks almost like it did when the machine was up and running. Specifically the user database is in the correct position, which is necessary because ecryptfs will only do its magic on home directories owned by the user.

Let’s become the user that owns the home directory.
su - kad

Time bring the files back:
ecryptfs-mount-private
cd /home/kad
ls

The password that the system asks for is the users (kad’s) password.

Should the decryption provided by /usr/sbin/ecryptfs-mount-private not work, one could always
strace /sbin/mount.ecryptfs-private
to see what is going on.

Thanks to http://blog.dustinkirkland.com/2009/03/mounting-your-encrypte...
for providing many of the necessary hints.

Bring my (software) raid1 up on Ubuntu after failure
2010-04-10

So, sometimes my raid1 arrays fail. I don’t know why, but it seems to be after software updates or maybe sometimes after hibernate or sleep (or a combination).

To fix this I will boot the Ubuntu 9.10 CD in “try before installing” mode and install mdadm: “sudo apt-get install mdadm” (this will bring postfix with it).

“less /proc/mdstat” will list all raid devices, it will be empty at first

“sfdisk -l” will list all partitions on all hard drives:

Disk /dev/sda: 121601 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System
/dev/sda1 * 0+ 11 12- 96358+ fd Linux raid autodetect
/dev/sda2 12 121600 121589 976663642+ 5 Extended
/dev/sda3 0 - 0 0 0 Empty
/dev/sda4 0 - 0 0 0 Empty
/dev/sda5 12+ 740 729- 5855661 82 Linux swap / Solaris
/dev/sda6 741+ 6819 6079- 48829536 83 Linux
/dev/sda7 6820+ 121600 114781- 921978351 fd Linux raid autodetect
Disk /dev/sdb: 121601 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System
/dev/sdb1 * 0+ 11 12- 96358+ fd Linux raid autodetect
/dev/sdb2 12 121600 121589 976663642+ 5 Extended
/dev/sdb3 0 - 0 0 0 Empty
/dev/sdb4 0 - 0 0 0 Empty
/dev/sdb5 12+ 740 729- 5855661 82 Linux swap / Solaris
/dev/sdb6 741+ 6819 6079- 48829536 83 Linux
/dev/sdb7 6820+ 121600 114781- 921978351 fd Linux raid autodetect

This will show me that /dev/sda1 and /dev/sdb1 is my small Linux raid autodetect , which I know contain /boot. /dev/sda7 and /dev/sdb7 is my large array which contain everything else. The /dev/sdx6 drives are actually unused, even if they are marked with “83/Linux”.

“mdadm –assemble /dev/md0 /dev/sda1 /dev/sdb1” and “mdadm –assemble /dev/md1 /dev/sda7 /dev/sdb7” will bring the arrays online.

Edit: “mdadm –assemble –scan” will also work, even when “sfdisk -l” does not show anything (that just happened…)

A brief look in /proc/mdstat reveals that the first drive of the md1 array is degraded:

Personalities : [raid1]
md1 : active raid1 sdb7[1]
921978240 blocks [2/1] [_U]
md0 : active raid1 sda1[0] sdb1[1]
96256 blocks [2/2] [UU]
unused devices:

Now, this command will also work to find out the state of an array: “mdadm –detail /dev/md1”, the state for me is “clean, degraded”.

To resync the raid array I issue the command: “mdadm –add /dev/md1 /dev/sda7”

Another look in /proc/mdstat reveals that the array is being reconstructed and will by in sync in about 2.5 hours:

Personalities : [raid1]
md1 : active raid1 sda7[2] sdb7[1]
921978240 blocks [2/1] [_U]
[>....................] recovery = 0.1% (1504512/921978240) finish=142.7min speed=107465K/sec
md0 : active raid1 sda1[0] sdb1[1]
96256 blocks [2/2] [UU]

When the recovery has finished, a reboot will start the computer from hard drive again.

To rebuild an array where one disk failed competely, please see this post. This and this post also contain useful examples about how to set up software raid.

I always forget if I used LVM or not when I installed the system (I did). To fiddle with LVM I will:
– Install lvm2: “sudo apt-get install lvm2”

Then “vgscan -v -v” will scan all partitions for LVM volume groups. lvscan and pvscan will do the same for logical and physical volumes.

Recover encrypted Ubuntu home directory (ecryptfs) from a live CD
2010-12-11

So, my Ubuntu 9.10/Karmic installation died. Again. Probably some software update that screwed up my fairly complicated install.

I have a raid1 (mirrored disks). On top of that I run LVM with one volume group, vg0, that has two logical volumes, one for home and one for root (everything else). Additionaly /boot is NOT on LVM but lies besides everything, still raided though.

Booting stopped. My home directory is encrypted so I can’t just copy everything out and reinstall. Bugger. Of course I can copy the encrypted data and hope to get it running in the new install, but that doesn’t feel safe.

Install tools necessary for RAID and LVM:
sudo apt-get install lvm2
sudo apt-get install mdadm

Bring up the RAID array as I always do:
sudo mdadm --assemble --scan
sudo less /proc/mdstat

Check if the volume group and the logical volumes are active.
sudo vgdisplay
sudo lvdisplay

If necessary one could make the volume group active with
sudo vgchange -a y

Create somewhere to mount the logical volumes
sudo mkdir /mnt/homevol
sudo mkdir /mnt/rootvol

Do not stand in any of the newly created directories, or you will not see the mounting getting done:
sudo mount -t auto /dev/vg0/homevol /mnt/homevol/
sudo mount -t auto /dev/vg0/rootvol /mnt/rootvol

Hmm, maybe /dev/mapper/vg0-rootvol and /dev/mapper/vg0-homevol was used here instead, I don’t remember.

Next, we’re about to prepare for chrooting in order to make the file system look like the file system on the machine when it was running without the live CD.

We need to mount some stuff started on the live CD into the directory we’re about to chroot into. These things must be available after chroot.

sudo mount --bind /dev /mnt/rootvol/dev/
sudo mount --bind /dev/shm /mnt/rootvol/dev/shm
sudo mount --bind /proc /mnt/rootvol/proc
sudo mount --bind /sys/ /mnt/rootvol/sys

Also make the home volume available under the to be root.
sudo mount --bind /mnt/homevol/ /mnt/rootvol/home/

Now it is time to change the root.
sudo chroot /mnt/rootvol/

Now the file system looks almost like it did when the machine was up and running. Specifically the user database is in the correct position, which is necessary because ecryptfs will only do its magic on home directories owned by the user.

Let’s become the user that owns the home directory.
su - kad

Time bring the files back:
ecryptfs-mount-private
cd /home/kad
ls

The password that the system asks for is the users (kad’s) password.

Should the decryption provided by /usr/sbin/ecryptfs-mount-private not work, one could always
strace /sbin/mount.ecryptfs-private
to see what is going on.

Thanks to http://blog.dustinkirkland.com/2009/03/mounting-your-encrypte...
for providing many of the necessary hints.