Extending Local Container Storage

From Proxmox VE
Revision as of 16:11, 25 December 2015 by Apmuthu (talk | contribs) (Introduction)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Introduction

Stitching disks end to end to enable more containers to be stored locally is one use for adding hard disks to a Proxmox VE 1.x box.

Actuals by Apmuthu 15:30, 20 September 2012 (UTC)

Commands to list disks

fdisk -l will not be enough for GPT partitions since PVE uses LVMs. The following will yield the current status of disk devices and their partition details.

cat /proc/partitions
lsblk
parted -l

Insert the new Hard Disk

Make sure that the Proxmox VE box is powered down and the new hard disk added into a higher numbered SATA slot and the machine restarted.

The existing hard disk is sda generally, if the SATA slot it is on is numbered lower than the new one is on. Legacy IDE hard disks may be hda, hdb, etc. Check that the new hard disk is recognized with:

dmesg | grep "] Attached SCSI disk"

sd 1:0:0:0: [sdb] Attached SCSI disk
sd 0:0:0:0: [sda] Attached SCSI disk

or with:

dmesg | grep "[h,s]d[a,b]" | grep " logical blocks: "

sd 0:0:0:0: [sda] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
sd 1:0:0:0: [sdb] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)

Prepare the new Hard Disk

fdisk /dev/sdb : Pass 1 (wipe out all partitions in disk)

fdisk /dev/sdb

The number of cylinders for this disk is set to 13953675.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sdb: 2000.3 GB, 2000398934016 bytes
70 heads, 4 sectors/track, 13953675 cylinders
Units = cylinders of 280 * 512 = 143360 bytes
Disk identifier: 0x00094a99

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           8        3745      523264   83  Linux

Command (m for help): d
Selected partition 1

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

All the existing partitions (Linux - 83) are removed now.

fdisk /dev/sdb : Pass 2

fdisk /dev/sdb

The number of cylinders for this disk is set to 13953675.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sdb: 2000.3 GB, 2000398934016 bytes
70 heads, 4 sectors/track, 13953675 cylinders
Units = cylinders of 280 * 512 = 143360 bytes
Disk identifier: 0x00094a99

   Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-243201, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-243201, default 243201):
Using default value 243201

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): L

 0  Empty           24  NEC DOS         81  Minix / old Lin bf  Solaris
 1  FAT12           27  Hidden NTFS Win 82  Linux swap / So c1  DRDOS/sec (FAT-
 2  XENIX root      39  Plan 9          83  Linux           c4  DRDOS/sec (FAT-
 3  XENIX usr       3c  PartitionMagic  84  OS/2 hidden C:  c6  DRDOS/sec (FAT-
 4  FAT16 <32M      40  Venix 80286     85  Linux extended  c7  Syrinx
 5  Extended        41  PPC PReP Boot   86  NTFS volume set da  Non-FS data
 6  FAT16           42  SFS             87  NTFS volume set db  CP/M / CTOS / .
 7  HPFS/NTFS/exFAT 4d  QNX4.x          88  Linux plaintext de  Dell Utility
 8  AIX             4e  QNX4.x 2nd part 8e  Linux LVM       df  BootIt
 9  AIX bootable    4f  QNX4.x 3rd part 93  Amoeba          e1  DOS access
 a  OS/2 Boot Manag 50  OnTrack DM      94  Amoeba BBT      e3  DOS R/O
 b  W95 FAT32       51  OnTrack DM6 Aux 9f  BSD/OS          e4  SpeedStor
 c  W95 FAT32 (LBA) 52  CP/M            a0  IBM Thinkpad hi eb  BeOS fs
 e  W95 FAT16 (LBA) 53  OnTrack DM6 Aux a5  FreeBSD         ee  GPT
 f  W95 Ext\'d (LBA) 54  OnTrackDM6      a6  OpenBSD         ef  EFI (FAT-12/16/
10  OPUS            55  EZ-Drive        a7  NeXTSTEP        f0  Linux/PA-RISC b
11  Hidden FAT12    56  Golden Bow      a8  Darwin UFS      f1  SpeedStor
12  Compaq diagnost 5c  Priam Edisk     a9  NetBSD          f4  SpeedStor
14  Hidden FAT16 <3 61  SpeedStor       ab  Darwin boot     f2  DOS secondary
16  Hidden FAT16    63  GNU HURD or Sys af  HFS / HFS+      fb  VMware VMFS
17  Hidden HPFS/NTF 64  Novell Netware  b7  BSDI fs         fc  VMware VMKCORE
18  AST SmartSleep  65  Novell Netware  b8  BSDI swap       fd  Linux raid auto
1b  Hidden W95 FAT3 70  DiskSecure Mult bb  Boot Wizard hid fe  LANstep
1c  Hidden W95 FAT3 75  PC/IX           be  Solaris boot    ff  BBT
1e  Hidden W95 FAT1 80  Old Minix

Hex code (type L to list codes): 8e (ee - GPT for > 2TB partitions)
Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help): p

Disk /dev/sdb: 2000.3 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xe05dab56

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1      243201  1953512001   8e  Linux LVM

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

There is only one partition that exists now, viz., Linux LVM (8e).

Check Current Disk Status

The original status of the box can be studied using the following commands:

cat /etc/mtab
cat /etc/fstab
mount
df -k
pvdisplay
vgdisplay
lvdisplay

Their outputs are listed below:

cat /etc/mtab

/dev/mapper/pve-root / ext3 rw,errors=remount-ro 0 0
tmpfs /lib/init/rw tmpfs rw,nosuid,mode=0755 0 0
proc /proc proc rw,noexec,nosuid,nodev 0 0
sysfs /sys sysfs rw,noexec,nosuid,nodev 0 0
procbususb /proc/bus/usb usbfs rw 0 0
udev /dev tmpfs rw,mode=0755 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
/dev/mapper/pve-data /var/lib/vz ext3 rw 0 0
/dev/sda1 /boot ext3 rw 0 0
beancounter /proc/vz/beancounter cgroup rw,name=beancounter 0 0
container /proc/vz/container cgroup rw,name=container 0 0
fairsched /proc/vz/fairsched cgroup rw,name=fairsched 0 0

cat /etc/fstab

# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext3 errors=remount-ro 0 1
/dev/pve/data /var/lib/vz ext3 defaults 0 1
UUID=952a2d3d-f610-4841-af9d-04a09a938ddc /boot ext3 defaults 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0

mount

/dev/mapper/pve-root on / type ext3 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
procbususb on /proc/bus/usb type usbfs (rw)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
/dev/mapper/pve-data on /var/lib/vz type ext3 (rw)
/dev/sda1 on /boot type ext3 (rw)
beancounter on /proc/vz/beancounter type cgroup (rw,name=beancounter)
container on /proc/vz/container type cgroup (rw,name=container)
fairsched on /proc/vz/fairsched type cgroup (rw,name=fairsched)

df -k

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/pve-root  99083868   9516628  84534076  11% /
tmpfs                  8201392         0   8201392   0% /lib/init/rw
udev                     10240       764      9476   8% /dev
tmpfs                  8201392         0   8201392   0% /dev/shm
/dev/mapper/pve-data 1803653668  90571196 1713082472   6% /var/lib/vz
/dev/sda1               516040     46416    443412  10% /boot

pvdisplay

  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               pve
  PV Size               1.82 TB / not usable 2.56 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              476803
  Free PE               1022
  Allocated PE          475781
  PV UUID               I8G01N-slgw-6nBh-tjhy-EL6E-TJBC-p18pYS

vgdisplay

  --- Volume group ---
  VG Name               pve
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               1.82 TB
  PE Size               4.00 MB
  Total PE              476803
  Alloc PE / Size       475781 / 1.81 TB
  Free  PE / Size       1022 / 3.99 GB
  VG UUID               Tb9Ooy-N9vl-6fb0-3Px7-BX6w-yazp-lq5nVi

lvdisplay

Note that there are 3 Logical Volumes. The last one, viz., /dev/pve/data is to be expanded for storing more containers.

  --- Logical volume ---
  LV Name                /dev/pve/swap
  VG Name                pve
  LV UUID                GZDLHY-r2Eh-7vxy-L01p-Nwpp-tTbJ-qd40Qf
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                15.00 GB
  Current LE             3840
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

  --- Logical volume ---
  LV Name                /dev/pve/root
  VG Name                pve
  LV UUID                mEsqrP-nsKi-OtHx-A0Zd-u1kK-O0Eg-orYChA
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                96.00 GB
  Current LE             24576
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

  --- Logical volume ---
  LV Name                /dev/pve/data
  VG Name                pve
  LV UUID                Np5v5n-0UDV-Gzcm-48ZI-5VAJ-mCz2-9jVrFU
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                1.71 TB
  Current LE             447365
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

The Current LE highlighted above is the current number of Logical Extents (LE) available to the Logical Volume /dev/pve/data now, ie., before it's expansion.

Create the Physical Extents

pvcreate /dev/sdb1
  Physical volume "/dev/sdb1" successfully created

Now to check if it is available in the set of Physical Volumes:

pvdisplay

  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               pve
  PV Size               1.82 TB / not usable 2.56 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              476803
  Free PE               1022
  Allocated PE          475781
  PV UUID               I8G01N-slgw-6nBh-tjhy-EL6E-TJBC-p18pYS

  "/dev/sdb1" is a new physical volume of "1.82 TB"
  --- NEW Physical volume ---
  PV Name               /dev/sdb1
  VG Name
  PV Size               1.82 TB
  Allocatable           NO
  PE Size (KByte)       0
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               hDyYtM-yJkZ-GKGa-Cva7-d73A-JKk6-YY4GKv

Note that the Physical Volume /dev/sdb1 does not have any Volume Group Name (VG Name) as yet.

Extend the Volume Group

vgextend "pve" /dev/sdb1
  Volume group "pve" successfully extended

vgdisplay

  --- Volume group ---
  VG Name               pve
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  5
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               3.64 TB
  PE Size               4.00 MB
  Total PE              953734
  Alloc PE / Size       475781 / 1.81 TB
  Free  PE / Size       477953 / 1.82 TB
  VG UUID               Tb9Ooy-N9vl-6fb0-3Px7-BX6w-yazp-lq5nVi

The VG Size is now expanded (first highlight) and the Free PE (Physical Extents) / Size enlarged.

Resize the Logical Volume

Locate the volume of interest to allocate the available storage capacity.

mount

..
..
/dev/mapper/pve-data on /var/lib/vz type ext3 (rw)
..
..

lvdisplay /dev/mapper/pve-data

  --- Logical volume ---
  LV Name                /dev/pve/data
  VG Name                pve
  LV UUID                Np5v5n-0UDV-Gzcm-48ZI-5VAJ-mCz2-9jVrFU
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                1.71 TB
  Current LE             447365
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

pvdisplay /dev/sdb1

  --- Physical volume ---
  PV Name               /dev/sdb1
  VG Name               pve
  PV Size               1.82 TB / not usable 2.56 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              476931
  Free PE               476931
  Allocated PE          0
  PV UUID               hDyYtM-yJkZ-GKGa-Cva7-d73A-JKk6-YY4GKv

The Current LE of /dev/pve/data, viz., 447365 needs to be added to the new capacity (Free PE) of /dev/sdb1, i.e., 476931 to obtain the final size (447365 + 476931 = 924296) of the Logical volume /dev/pve/data to be used in the lvresize command.

Actual Resizing the Logical Volume

lvresize -l 924296 /dev/mapper/pve-data
  Extending logical volume data to 3.53 TB
  Logical volume data successfully resized

Generic Notes on lvresize

  • lvresize command can be used to resize the Logical Volumes
  • lvresize command can be used to increase (expand) or decrease (shrink) the size of the filesystem.
  • Always it is better to unmount the file system before resizeing.
  • Decreasing (shrinking) the logical volume size without unmounting is not supported.
  • Filesystems can be expanded online, without unmounting them:
  • LVM and ext3 support resizing filesystems after they’ve been created:
lvresize -L+10G /dev/sys/scratch
resize2fs /dev/sys/scratch
  • When shrinking a filesystem, it’s safest to copy its data elsewhere before unmounting and recreating it:
rsync -avr /scratch /data/.
umount /scratch
lvresize -L-10G /dev/sys/scratch
mkfs.ext3 /dev/sys/scratch

Update the File System

Only the Logical Volume now shows the new size:

lvdisplay /dev/mapper/pve-data

  --- Logical volume ---
  LV Name                /dev/pve/data
  VG Name                pve
  LV UUID                Np5v5n-0UDV-Gzcm-48ZI-5VAJ-mCz2-9jVrFU
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                3.53 TB
  Current LE             924296
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

But the filesystem does not reflect it:

df -k

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/pve-root  99083868   9516844  84533860  11% /
tmpfs                  8201392         0   8201392   0% /lib/init/rw
udev                     10240       768      9472   8% /dev
tmpfs                  8201392         0   8201392   0% /dev/shm
/dev/mapper/pve-data 1803653668  90571196 1713082472   6% /var/lib/vz
/dev/sda1               516040     46416    443412  10% /boot

Hence we need to update the file system with:

resize2fs /dev/mapper/pve-data
resize2fs 1.41.3 (12-Oct-2008)
Filesystem at /dev/mapper/pve-data is mounted on /var/lib/vz; on-line resizing required
old desc_blocks = 110, new_desc_blocks = 226
Performing an on-line resize of /dev/mapper/pve-data to 946479104 (4k) blocks.

The last line above takes a wuile in coming. When complete after a much longer time, the following will be displayed:

The filesystem on /dev/mapper/pve-data is now 946479104 blocks long.

While waiting for the last line above to appear, on a another session, dmesg command will show the cause of the delay as:

device-mapper: table: 253:2: adding target device (start sect 3673186304 len 3898646528) caused an alignment inconsistency

Stats on completion

df -kh

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/pve-root   95G  9.1G   81G  11% /
tmpfs                 7.9G     0  7.9G   0% /lib/init/rw
udev                   10M  768K  9.3M   8% /dev
tmpfs                 7.9G     0  7.9G   0% /dev/shm
/dev/mapper/pve-data  3.5T   87G  3.4T   3% /var/lib/vz
/dev/sda1             504M   46M  434M  10% /boot
  • Logical Volume File System /dev/mapper/pve-data is now enlarged correctly.
  • /etc/fstab and /etc/mtab remain unchanged

References