Schlagwort-Archive: lvm

Replace harddisk to grow raid and lvm volume

Schreibe eine Antwort

I ran out of disk space on a 2 disk raid mirror. I already replaced one of the harddisks with a bigger 4TB one. The size doesn’t allow for MBR anymore and I needed to switch to GPT. The now smaller drive is also to replaced. Here are some of my notes for the procedure for later use. In the end I didn’t use this guide. I had a good backup and some time on the weekend as nobody needed the server, so I opted for the live migration. Since I already wrote most of the steps down, I will keep it and just add some notes at the end.

Mark the smaller disk as failed and remove it from the array.
mdadm --manage /dev/md0 --fail /dev/sda1
mdadm --manage /dev/md1 --fail /dev/sda2
mdadm --manage /dev/md2 --fail /dev/sda3
cat /proc/mdstat
mdadm --manage /dev/md0 --remove /dev/sda1
mdadm --manage /dev/md1 --remove /dev/sda2
mdadm --manage /dev/md2 --remove /dev/sda3

Shutdown the system, replace harddisk with new one and boot a live system. Install the needed packages.
aptitude install mdadm gdisk
modprobe raid1

Start raid.
mdadm --examine --scan >> /etc/mdadm/mdadm.conf

Clone the GPT partition schema to the new disk.
sgdisk --backup=table /dev/sdb
sgdisk --load-backup=table /dev/sda
sgdisk -G /dev/sda

Add to raid
mdadm --manage /dev/md0 --add /dev/sda1
mdadm --manage /dev/md1 --add /dev/sda2
mdadm --manage /dev/md2 --add /dev/sda3

Synchronisation starts. You can watch the process with
watch cat /proc/mdstat

Expand the raid to the new maximum size.
mdadm --grow /dev/md2 --size=max

Grow the LVM.
pvresize /dev/md2
lvextend -L +1TB /dev/mapper/deb7-home
resize2fs /dev/mapper/deb7-home
grub-install /dev/sdc --recheck

Reboot.

Manual integrity-check of raid.
/usr/share/mdadm/checkarray /dev/md0
/usr/share/mdadm/checkarray /dev/md1
/usr/share/mdadm/checkarray /dev/md2

 

Alternative: Live migration

Live migration is nearly the same, but you don’t have to reboot the system.

Hotplug the new (third) drive to your system. If the Sata-controller is set to AHCI mode, the system should recognize the new drive.

After cloning the partition table with sgdisk, add the drive to the raid.
mdadm /dev/md0 --manage --add /dev/sdc1
mdadm /dev/md1 --manage --add /dev/sdc2
mdadm /dev/md2 --manage --add /dev/sdc5

Grow the raid to 3 devices and let it recover.
mdadm /dev/md0 --grow -n3
mdadm /dev/md1 --grow -n3
mdadm /dev/md2 --grow -n3

Mark the to-be-replaced drive as failed and remove it from the raid array.
mdadm /dev/md0 --manage -f /dev/sda1 -r /dev/sda1
mdadm /dev/md1 --manage -f /dev/sda2 -r /dev/sda2
mdadm /dev/md2 --manage -f /dev/sda3 -r /dev/sda3

Shrink the array again to 2 drives.
mdadm /dev/md0 --grow -n2
mdadm /dev/md1 --grow -n2
mdadm /dev/md2 --grow -n2

Grow the raid and extend pv, lv and filesystem like above.

Fully encrypted gentoo system with LUKS/cryptsetup and LVM

5 Antworten

I have installed this setup some time ago on my system and wrote down the steps. To validate that every step is still correct I used VirtualBox and installed the system again.

At the end of this howto, you will have a full working gentoo system with (nearly) full disk encryption, coded RAM and an easily expandable disk partition schema. Full disk encryption under GNU/Linux isn’t possible like it is with e.g. Truecrypt under Windows. So in most cases you will have to leave your /boot partition unencrypted or use a removable medium like an usb stick to boot your system. In this setup we will leave /boot as it is.

What you need:

  • Backup of your running system since your disks will be erased
  • Linux compatible hardware
  • Internet connection
  • Some time
  • Experience with the GNU/Linux system is highly recommended
  • A GNU/Linux system (Live CD, usb stick, installation on other disk etc.) This howto uses the Gentoo Weekly Minimal Install CD for amd64

Although I will try, explaining every step in detail would be to much for this howto so I have to point you to Google and the Gentoo Handbook.

1. Preparing the installation

Use for example UNetbootin to create a bootable usb stick or just burn the cd. When you’re using GRUB2 on your system, you can boot the iso file directly through grub’s loopback ability. Change your BIOS settings accordingly.

2. Booting and choosing the correct keyboard layout

Boot the installation system and choose the default kernel at prompt. When all kernel modules are loaded you have the opportunity to change the keyboard layout to your needs (for german layout, type de or 10).

3. Configuring your network

Find out how your network adapters are called with ifconfig -a. To obtain an IP for the wired network link on eth0, type dhcpd eth0. To show your IP just type ifconfig eth0 and look for inet addr. To continue the installation from another computer, start the sshd server with /etc/init.d/sshd start, set a password for root with the passwd command and connect from the other computer with ssh root@IP under Linux or use Putty under Windows.

4. Preparing your harddisk(s)

At this point you have more then one possibility on how to create your disk layout, so you should be really sure what you want. If you are not sure or don’t know how to split your disk, check out the handbook again and/or search Google. You have to decide in which order your different layers occur. It is possible to create one big (physical) LVM partition, configure the logical volumes and encrypt these individually. You can also create a standard partition, encrypt the whole partition and configure your logical volumes inside your secure container. Both setups have advantages and disadvantages. The first makes it quite easy to change your LVM layout and you can different passwords for every mount point if you like. The second method has the advantage that other systems will only see one big encrypted partition and nothing else and you only need one password to decrypt the system (although this is also possible with the first setup).
In this setup we will create a small partition for /boot, a separate partition for swap and another one for the physical LVM. Maybe I will add one or two alternatives to this guide in the future, but for now I am going with the first method as I think it is the more difficult one to reproduce. Use fdisk or cfdisk to configure the harddisk(s) to your needs.

The partition table I chose for this howto

livecd ~ # fdisk -l /dev/sda
Disk /dev/sda: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sda1 * 1 61 489951 83 Linux
/dev/sda2 62 183 979965 82 Linux swap / Solaris
/dev/sda3 184 2007 14651280 8e Linux LVM

5. Loading the necessary kernel modules

You need to load the necessary kernel modules for your algorithm and cipher. Use google to learn more about algorithms and ciphers.

livecd ~ # modprobe dm-mod dm-crypt sha256

6. Create and activate the LVM Setup

First we need to create the physical volume on /dev/sda3 and a volume group called vg1 with

livecd ~ # pvcreate /dev/sda3
livecd ~ # vgcreate vg1 /dev/sda3

Next we use lvcreate to create the individual logical volumes. This step is similar to the standard partitioning process, however, since we are creating logical volumes here, it is possible to change their size later again. The chosen layout here is just a demonstration of the general functionality. The name defined with the -n flag (-L is size) is just for you, so the name doesn’t have to match the later mountpoint.

livecd ~ # lvcreate -L7G -nroot vg1
livecd ~ # lvcreate -L3G -ntmp vg1
livecd ~ # lvcreate -L3G -nhome vg1

You can use the vgdisplay or vgs command to see the remaining disk space. Check your result with lvscan.

livecd ~ # lvscan
ACTIVE ‚/dev/vg1/root‘ [7.00 GiB] inherit
ACTIVE ‚/dev/vg1/tmp‘ [3.00 GiB] inherit
ACTIVE ‚/dev/vg1/home‘ [3.00 GiB] inherit

HINT:If you need to reboot, you can reactivate the LVM setup with lvchange -a y vg1 .

7. Encrypting

Next step is to encrypt the freshly generated volumes using this short loop (easily expandable if you want to create more containers).

livecd ~ # for i in root tmp home
> do
> cryptsetup -c aes-cbc-essiv:sha256 -s 256 luksFormat /dev/vg1/$i
> done

Confirm with YES and type in your password twice per volume.
Also encrypt the swap partition during the setup process.

livecd ~ # cryptsetup create -c aes-cbc-essiv:sha256 -s 256 -d /dev/urandom swap /dev/sda2
livecd ~ # mkswap /dev/mapper/swap
livecd ~ # swapon /dev/mapper/swap

8. Open the containers and format with a filesystem

Next we open our containers and create a filesystem. The choice of the filesystem depends on your needs, see google or the gentoo handbook.

livecd ~ # for i in root tmp home
> do
> cryptsetup luksOpen /dev/vg1/$i crypt$i
> done

Put in the password for every volume.

livecd ~ # for i in root tmp home
> do
> mkfs.ext4 /dev/mapper/crypt$i
> done
livecd ~ # mkfs.ext2 /dev/sda1

Don’t forget to format the boot partition as well.

9. Create the mountpoints and mount your volumes

livecd ~ # mount /dev/mapper/cryptroot /mnt/gentoo/
livecd ~ # for i in boot tmp home
> do
> mkdir /mnt/gentoo/$i
> done
livecd ~ # mount /dev/sda1 /mnt/gentoo/boot/
livecd ~ # for i in tmp home
> do
> mount /dev/mapper/crypt$i /mnt/gentoo/$i
> done

10. Preparing the chroot

Before chrooting into the new installation directory we have to prepare a few things, e.g. setting the correct file permissions before changing the directory

livecd ~ # chmod 1777 /mnt/gentoo/tmp/
livecd ~ # cd /mnt/gentoo/

and setting the correct date.

livecd gentoo # date MMDDhhmmYYYY

Next download the needed installation files. Select the best mirror near you and download (press d in links) the latest stage3 and portage files as well as the .CONTENTS .DIGESTS and .md5sum files using links.

livecd gentoo # links http://www.gentoo.org/main/en/mirrors.xml

Latest stage3 file can be found under /releases/amd64/current-stage3/ (select the right folder for your architecture) and the latest can be found using the symlink in /snapshots/portage-latest.tar.bz2.

Verify your downloads with md5sum and extract them when there is no error.

livecd gentoo # md5sum -c stage3-amd64-20100617.tar.bz2.DIGESTS
livecd gentoo # md5sum -c portage-latest.tar.bz2.md5sum

livecd gentoo # tar xvjpf stage3-amd64-20100617.tar.bz2
livecd gentoo # tar xvjf /mnt/gentoo/portage-latest.tar.bz2 -C /mnt/gentoo/usr

Take a look at the wiki entry about CFLAGS and edit the make.conf to your needs. I use the Amd Athlon 64 64bit settings here.

livecd gentoo # nano /mnt/gentoo/etc/make.conf

CHOST=“x86_64-pc-linux-gnu“
CFLAGS=“-march=k8 -O2 -pipe“
CXXFLAGS=“${CFLAGS}“

Select the nearest mirror

livecd gentoo # mirrorselect -i -o >> /mnt/gentoo/etc/make.conf
livecd gentoo # mirrorselect -i -r -o >> /mnt/gentoo/etc/make.conf

Copy the DNS information to your chroot and mount the needed devices

livecd gentoo # cp -L /etc/resolv.conf /mnt/gentoo/etc/
livecd gentoo # mount -t proc none /mnt/gentoo/proc
livecd gentoo # mount -o bind /dev /mnt/gentoo/dev

11. Chroot into your new environment

Change into your new root and sync and update your system.

livecd gentoo # chroot /mnt/gentoo/ /bin/bash

livecd / # env-update && source /etc/profile
livecd / # emerge –sync

12. Select a profile

Select the right profile for you using eselect

livecd / # eselect profile list
Available profile symlink targets:
[1] default/linux/amd64/10.0 *
[2] default/linux/amd64/10.0/desktop
[3] default/linux/amd64/10.0/desktop/gnome
[4] default/linux/amd64/10.0/desktop/kde
[5] default/linux/amd64/10.0/developer
[6] default/linux/amd64/10.0/no-multilib
[7] default/linux/amd64/10.0/server
[8] hardened/linux/amd64/10.0
[9] hardened/linux/amd64/10.0/no-multilib
[10] selinux/2007.0/amd64
[11] selinux/2007.0/amd64/hardened
[12] selinux/v2refpolicy/amd64
[13] selinux/v2refpolicy/amd64/desktop
[14] selinux/v2refpolicy/amd64/developer
[15] selinux/v2refpolicy/amd64/hardened
[16] selinux/v2refpolicy/amd64/server
livecd / # eselect profile set 2

13. Adjust make.conf

Adjust the make.conf file to your needs and set the needed USE flags, languages and devices

livecd / # nano /etc/make.conf
MAKEOPTS=“-j2″

USE=“3dnow 3dnowext a52 aac acl acpi alsa -amd64 apache2 -avahi
avi bluetooth -beagle cairo cdb cddb cdparanoia cdr cli cracklib
crypt css -cups daap dbus dga directfb dri dts dvb dvd dvdr dvdread -eds
-emboss encode -esd -evo exif fam ffmpeg flac ftp gdbm     gif -gnome gstreamer gtk
hal -ipv6 -isdnlog -joystick -kde libnotify lirc lm_sensors mad mmx mmxext mp3
mpeg mysql ncurses nls ogg opengl pdf png -ppds -pppd qt3 qt3support quicktime samba sdl sse sse2
ssl svg tiff transcode unicode usb v4l vcd vorbis wma
X x264 xcomposite xine xml xorg xv xvid xvmc zlib“

LINGUAS=“de en“

INPUT_DEVICES=“keyboard mouse“
VIDEO_CARDS=“vesa“

14. Configure your charsets

Define (uncomment) and generate the needed charsets incl. UTF-8 (german)

livecd / # nano /etc/locale.gen
en_US ISO-8859-1
en_US.UTF-8 UTF-8
de_DE ISO-8859-1
de_DE@euro ISO-8859-15
de_DE.UTF-8 UTF-8

livecd / # locale-gen

15. Setting the timezone

Find the right timezone for you and copy it

livecd / # ls /usr/share/zoneinfo/
livecd / # cp /usr/share/zoneinfo/Europe/Berlin /etc/localtime

16. Installation of kernel sources and other needed pakets

Install the kernel sources and other pakets that are needed for your setup

livecd / # emerge -av gentoo-sources
livecd / # emerge -av genkernel cryptsetup lvm2

17. Compile your own kernel

The next step is one of the most important. Take care that you include everything in your kernel that you need for your hardware to work. Use genkernel to make life a little bit easier.

livecd / # cd /usr/src/linux
livecd linux # genkernel –no-clean –menuconfig –save-config –luks –lvm all

You can use the kernel configuration of the gentoo live cd as a basis for your own configuration. Just copy the config file before running genkernel.

livecd linux # zcat /proc/config.gz > /usr/share/genkernel/arch/x86_64/kernel-config

The most important kernel options for this guide to add. You might need others depending on the chosen cipher etc. Don’t forget to enable Ext4 support if you need it.

Device Drivers  —>
Multi-device support (RAID and LVM)  —>
[*] Multiple devices driver support (RAID and LVM)
< >   RAID support
<*>  Device mapper support
<*>  Crypt target support

File Systems —>
<*>  The Extended 4 (ext4) filesystem

Cryptographic API  —>
<*>  SHA256 digest algorithm
<*>  AES cipher algorithms

I found a few postings on the net saying, that genkernel needs a valid /etc/fstab to build a correct initramfs. Do step 19 first to be 100% safe.

18. Kernel Modules

List the compiled kernel modules with

livecd linux # find /lib/modules/<kernel version>/ -type f -iname ‚*.o‘ -or -iname ‚*.ko‘ | less

and add them to your /etc/modules.autoload.d/kernel-2.6.

19. Edit the /etc/fstab

# <fs>                  <mountpoint>    <type>          <opts>          <dump/pass>
/dev/sda1               /boot           ext2            noauto,noatime  1 2
/dev/mapper/swap        none            swap            sw              0 0
/dev/mapper/cryptroot   /               ext4            noatime         0 0
/dev/mapper/crypthome   /home           ext4            noatime         0 1
/dev/mapper/crypttmp    /tmp            ext4            noatime         0 2
/dev/cdrom              /mnt/cdrom      auto            noauto,ro       0 0
shm                     /dev/shm        tmpfs           nodev,nosuid,noexec     0 0

20. Basic system configuration

Setting a hostname

livecd / # nano /etc/conf.d/hostname

Setting the root password. Important: This password is inside your chroot and for your future system, the password you set at the beginning was just needed for ssh access to the box! If you don’t set it, you won’t be able to login.

livecd / # passwd

System information like default editor or windows manager

livecd / # nano /etc/rc.conf

Change the keymap and timezone

livecd / # nano /etc/conf.d/keymaps
KEYMAP=“de“

Setting the clock

livecd / # nano /etc/conf.d/clock
TIMEZONE=“Europe/Berlin“

Installing some system tools (logger, filesystem tools etc.)

livecd / # emerge -av syslog-ng logrotate pciutils gentoolkit

Start the system logger at boot

livecd / # rc-update add syslog-ng default

Networking: Installing a dhcp client

livecd / # emerge -av dhcp

21. Installing the bootloader GRUB

Another really important part is the installation of a bootloader. Take care that you use the correct devices, partitions and paths – double check! If you want or need the features of GRUB 2 take a look at this page.

livecd / # emerge -av grub

Now you need to find out the exact file names for your kernel and initramfs and edit /boot/grub/grub.conf accordingly.

livecd / # ls /boot/initramfs* /boot/kernel*
-rw-r–r– 1 root root 1814297 Jul 24 00:14 /boot/initramfs-genkernel-x86_64-2.6.34-gentoo-r1
-rw-r–r– 1 root root 4261424 Jul 24 00:13 /boot/kernel-genkernel-x86_64-2.6.34-gentoo-r1

livecd / # nano /boot/grub/grub.conf

title Gentoo Linux 2.6.34-r1
root (hd0,0)
kernel /boot/kernel-genkernel-x86_64-2.6.34-gentoo-r1 root=/dev/ram0 crypt_root=/dev/vg1/root init=/linuxrc splash=silent dolvm
initrd /boot/initramfs-genkernel-x86_64-2.6.34-gentoo-r1

Creating /etc/mtab

livecd / # grep -v rootfs /proc/mounts > /etc/mtab

Installing GRUB

livecd / # grub-install –no-floppy /dev/sda

If this doesn’t work for you, try running grub manually

livecd / # grub
root (hd0,0)
setup (hd0)
quit

22. LVM configuration

The LVM process scans all connected drives for volumes. You can define a filter so that only the needed block devices are scanned.

livecd / # nano /etc/lvm/lvm.conf
filter = [ „a|/dev/sda|“, „r/.*/“ ]

23. LUKS/dm-crypt configuration

Next we edit /etc/conf.d/dmcrypt and define which encrypted containers should be opened and with which name. We will use a keyfile to decrypt the volumes.

livecd / # nano /etc/conf.d/dmcrypt

## swap
swap=swap
source=’/dev/sda2′
options=‘-c aes-cbc-essiv:sha256 -s 256 -d /dev/urandom‘

## /home with keyfile
target=crypthome
source=’/dev/mapper/vg1-home‘
key=’/root/hdpw‘

## /tmp with keyfile
target=crypttmp
source=’/dev/mapper/vg1-tmp‘
key=’/root/hdpw‘

Generating the keyfile. This can take some time depending on the activities on your box.

livecd / # dd if=/dev/random of=/root/hdpw bs=1 count=512
livecd / # chmod 400 /root/hdpw

Adding the generated keyfile to the known and valid keys in the LUKS keyring. After this step you can open your containers either using the password defined in step 7 or this file.

livecd / # for i in home tmp
> do
> cryptsetup luksAddKey /dev/vg1/$i /root/hdpw
> done

24. Finish the installation and clean up

Delete the installation files.

livecd / # rm /stage3-* /portage-*

Exit the chroot and unmount all devices.

livecd / # exit
livecd gentoo # cd ..
livecd mnt # umount /mnt/gentoo/*
livecd mnt # umount /mnt/gentoo

Close the LUKS containers and deactivate your LVM.

livecd mnt # for i in home tmp root
> do
> cryptsetup luksClose crypt$i
> done

livecd mnt # lvchange -a n vg1

Restart and pray! :)

livecd mnt # shutdown -r now

25. Success!

Welcome to your new system. Log in with the username root and password set in step 20.

26. Some more basic configuration

Start the SSH daemon at boot.

gentoo ~ # rc-update add sshd default

Synchronize your portage tree and make an update.

gentoo ~ # emerge –sync
gentoo ~ # emerge -avDuN world
gentoo ~ # env-update && source /etc/profile

Install ntp to sync your systemtime on boot.

gentoo ~ # emerge -av ntp

gentoo ~ # nano /etc/conf.d/ntp-client
NTPCLIENT_OPTS=“-s -b -u
0.de.pool.ntp.org 1.de.pool.ntp.org
2.de.pool.ntp.org 3.de.pool.ntp.org“

gentoo ~ # rc-update add ntp-client default

Clean your system and check for consistency and errors.

gentoo ~ # emerge –clean
gentoo ~ # revdep-rebuild