Disk Health Email Alerts

Introduction

Disk health can be reported on drives which support S.M.A.R.T. monitoring. An email alert can be sent to warn of potential problems as they unfold so appropriate action can be taken to avert loss of data. As of version 1.6 a default Proxmox installation does not include this functionality, and additional packages must be installed and configured to make use of it.

Installation & Configuration

1. Install smartmontools:

First we update aptitude and install the package:

aptitude update && aptitude install smartmontools 

2. Configure the daemon:

The package smartmontools includes a background process, aka 'daemon' that will periodically check on the disks. We need to tell it which to watch, so first we verify how our disks are named. This is done by issuing the command fdisk -l.

 fdisk -l 

This produces output that includes our logical volumes, which cannot be monitored. It looks similar to this:

# fdisk -l

Disk /dev/sda: 32.0 GB, 32000000000 bytes
255 heads, 63 sectors/track, 3890 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          66      524288   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2              66        3890    30722105   8e  Linux LVM

WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdb: 2998.9 GB, 2998960914432 bytes
255 heads, 63 sectors/track, 364602 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1      267350  2147483647+  ee  EFI GPT

Disk /dev/dm-0: 3892 MB, 3892314112 bytes
255 heads, 63 sectors/track, 473 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/dm-0 doesn't contain a valid partition table

Disk /dev/dm-1: 7784 MB, 7784628224 bytes
255 heads, 63 sectors/track, 946 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/dm-1 doesn't contain a valid partition table

Disk /dev/dm-2: 15.8 GB, 15896412160 bytes
255 heads, 63 sectors/track, 1932 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/dm-2 doesn't contain a valid partition table

Our concern is with the disks upon which the logical volumes reside, these have names like /dev/sda, /dev/sdb, /dev/hda, /dev/hdb- not the logical volumes themselves that have names like /dev/dm-#. These logical volumes belong to our VMs, not PVE.

Next we verify that our disks are manufactured with S.M.A.R.T. support, and that the capability is turned on. This is done by issuing the command smartctl -a /dev/sda, where sda is substituted with however your disk is named.

#smartctl -a /dev/sda

This produces output that looks like this:

#smartctl -a /dev/sda
smartctl version 5.38 [x86_64-unknown-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model:     SSDSA2SH032G1GN INTEL
Serial Number:    CVEM001300D0032HGN
Firmware Version: 045C8860
User Capacity:    32,000,000,000 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   7
ATA Standard is:  ATA/ATAPI-7 T13 1532D revision 1
Local Time is:    Fri Nov 12 06:35:42 2010 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

These are the lines that answer our inquiry:

SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Sometimes the capability is present in the disk, and is not enabled. if this is the case your output will include these lines instead:

SMART support is: Available - device has SMART capability.
SMART support is: Disabled

To change the Disabled to Enabled we issue this command:

#smartctl -s on -a /dev/sda

Again, substituting the sda with however your disk is named.

Running the smartctl -a command again as we have done previously can verify that the functionality has indeed been enabled within the disk.

Now since we know our disks can answer with the requested information we have to tell the daemon to ask them how they are at regular intervals. We do this by editing the file /etc/default/smartmontools.

#nano /etc/default/smartmontools

These are the lines before and after we change them:

This one defines which disks are to be monitored, we uncomment this and enter the names of our disks here, as we determined previously:

#enable_smart="/dev/hda /dev/hdb"
enable_smart="/dev/sda"

This one starts the daemon with the host machine, just uncomment it to make that happen:

#start_smartd=yes
start_smartd=yes

This one defines the interval at which the disks are queried, values are in seconds. Defaults are fine however I prefer less checking than every half hour, so I change mine to every 3.

#smartd_opts="--interval=1800"
smartd_opts="--interval=10800"

Now our disks are able to be checked up on, now we set the daemon so that an alarm condition sends an email to let us know something's amiss. By default it will keep tabs on every disk it can find, which in our case with alot of virtual disks not supporting S.M.A.R.T. monitoring can cause alot of unnecessary work for the daemon, and wasting precious CPU cycles. So here we also explicitly define which disks to look at. We do this by editing the file /etc/smartd.conf.

#nano /etc/smartd.conf

These are the pertinent lines before and after we change them:

This line says to scan for and check everything, I comment this one out to disable it, in favor of other lines later.

DEVICESCAN -d removable -n standby -m root -M exec /usr/share/smartmontools/smartd-runner
#DEVICESCAN -d removable -n standby -m root -M exec /usr/share/smartmontools/smartd-runner

This one tells the daemon to treat SCSI as SATA. I don't quite understand why I had to set this, or if you will have to. Maybe someone can edit this to include better information on this point. Just uncomment to enable.

# /dev/sda -a -d sat
/dev/sda -a -d sat

Uncomment this line that defines which of the SCSI disks would get extended testing and when. Mine's a SATA disk, so I changed the identifier. Here I also change the schedule of the disk check from the default 3rd day (Tuesday) at 18:00 hours (6pm) and 7th day (Saturday) at 01:00 hours (1am), to Monday 9pm. The -m flag is added to specify the email address to which warnings or errors will be sent, which will be root. Proxmox is configured to forward root's email to the address specified during installation.

#/dev/sda -d scsi -s L/../../3/18
#/dev/sdb -d scsi -s L/../../7/01
/dev/sda -d sat -s L/../../2/21 -m root

Then for added peace of mind add a line to have a test email sent at startup.

/dev/sda -m root -M test

Of course there are many more ways to do this, as are detailed both in the config file and the man page. This config is done, so the file is saved.

Now start the daemon & see if it works. This is done by issuing this command:

#/etc/init.d/smartmontools start

There should be an email waiting to be read, and we can look in the log to see details with this command:

#tail -n50 /var/log/syslog

External links with regard to S.M.A.R.T.

• Smartmontools website on Sourceforge
• S.M.A.R.T. on Wikipedia