Monitoring with Icinga

From Proxmox VE
Jump to: navigation, search

Introduction

This wikipage describes the installation of icinga (with pnp for graphical output) on a debian-box.

Icinga is a fork of Nagios - the config is the same. If you need further information search also for nagios.

It's take some time to install icinga but if you follow this guide step by step you (hopefully) will get an working installation.

To get to the goal faster - make a preconfig-vm for proxmox (based on devil-linux), but not yet...

Normally the monitoring-host should be a standalone stable machine, for test it's ok in a vm.

Some definitions

  • Call the debian-box monitoring and the ip in this example is 192.168.200.200
  • The proxmox-host is called proxmox and has the ip 192.168.200.10
  • The VM which should be monitored has the ip 192.168.200.66

Put into practice

Install on monitoring using debian-504-i386-netinst.iso - select only standard

Login as root

We need some more stuff

apt-get update
apt-get install openssh-server sudo lynx apache2 build-essential \
       libgd2-xpm-dev librrd-dev rrdtool librrds-perl librrdp-perl \
       libnet-snmp-perl php5 php5-gd unzip

In this example we use as working-user support (defined during the installation process)

ssh support@192.168.200.200

Enable support for sudo

su
visudo -f /etc/sudoers

uncomment the last line:

%sudo ALL=NOPASSWD: ALL
addgroup support sudo

Create a user for icinga

/usr/sbin/useradd -m icinga
passwd icinga
exit # again support
exit # logoff
ssh support@192.168.200.200 # to activate changes in group
mkdir ~/software
cd ~/software

Build and install icinga

Current Version - 1.5.1 (Circa Sep 2011)

wget http://downloads.sourceforge.net/project/icinga/icinga/1.2.1/icinga-1.2.1.tar.gz
tar xvzf icinga-1.2.1.tar.gz
cd icinga-1.2.1
./configure
make all
sudo make fullinstall
cd /etc/rc2.d/
sudo ln -s ../init.d/icinga S92icinga
cd ../rc0.d
sudo ln -s ../init.d/icinga K10icinga
sudo vi /usr/local/icinga/etc/objects/contacts.cfg # change the email-address of icingaadmin
sudo htpasswd -c /usr/local/icinga/etc/htpasswd.users icingaadmin # this account/password is for the webfrontend
sudo ln -s /usr/local/icinga/etc /etc/icinga # to access the config below /etc
sudo addgroup www-data icinga

Build and install the nagios-plugins

Version is current as of Sep 2011.

cd ~/software
wget http://sourceforge.net/projects/nagiosplug/files/nagiosplug/1.4.15/nagios-plugins-1.4.15.tar.gz
tar xvzf nagios-plugins-1.4.15.tar.gz
cd nagios-plugins-1.4.15
./configure --prefix=/usr/local/icinga --with-nagios-user=icinga --with-cgiurl=/icinga/cgi-bin
make
sudo make install

Install snmp-plugin

cd ~/software
wget http://nagios.manubulon.com/check_snmp_int.pl
chmod +x check_snmp_int.pl
sudo chown icinga check_snmp_int.pl
sudo mv check_snmp_int.pl /usr/local/icinga/libexec/

Build and install pnp

cd ~/software

Download the kohana framework and scp it to support@monitoring:software/ - or with lynx directly on monitoring. Current versions are 3.2.0 and 3.1.4.

cd ~/software 
unzip kohana-3.0.8.zip
sudo mv kohana /var/www/
sudo chown -R www-data:www-data /var/www/kohana
sudo mv /var/www/kohana/install.php /var/www/kohana/install.php.org

PHP4Nagois Current Version is 0.6.15

wget http://downloads.sourceforge.net/project/pnp4nagios/PNP-0.6/pnp4nagios-0.6.7.tar.gz
tar xvzf pnp4nagios-0.6.7.tar.gz
cd pnp4nagios-0.6.7/
USER=icinga
GROUP=icinga
PREFIX=/usr/local/icinga
./configure --with-nagios-user=$USER --with-nagios-group=$GROUP --sysconfdir=$PREFIX/etc/pnp \
           --prefix=$PREFIX --datarootdir=$PREFIX/share/pnp --with-rrdtool=/usr/bin/rrdtool \
           --sysconfdir=$PREFIX/etc/pnp --with-perfdata-dir=$PREFIX/share/perfdata \
           --with-kohana_system=/var/www/kohana/system --exec-prefix=$PREFIX \
           --with-perfdata-logfile=$PREFIX/var/perfdata.log \
           --with-perfdata-spool-dir=$PREFIX/var/spool/perfdata
make all
sudo make install
sudo make install-webconf
sudo make install-config
sudo ln -s /usr/local/icinga/lib/kohana/system /usr/local/icinga/share/pnp/system
sudo vi /usr/local/icinga/share/pnp/index.php
# change "$kohana_system = '/var/www/kohana/system';" to
$kohana_system = '/usr/local/icinga/lib/kohana/system';

Configuration

cd /etc/apache2/mods-enabled
sudo ln -s ../mods-available/rewrite.load .
cd /etc/icinga
su icinga
ssh-keygen # we need for further plugins the ssh-key - not in this example
mv /etc/icinga/pnp/process_perfdata.cfg-sample /etc/icinga/pnp/process_perfdata.cfg
mkdir proxmox
mkdir check_commands
# add two lines to icinga.cfg:
cfg_dir=/usr/local/icinga/etc/proxmox
cfg_dir=/usr/local/icinga/etc/check_commands

change in icinga.cfg "process_performance_data=0" to

process_performance_data=1

and uncomment the following two lines:

host_perfdata_command=process-host-perfdata
service_perfdata_command=process-service-perfdata

Comment out the last two blocks in /etc/icinga/objects/commands.cfg (perfdata), or delete the line (but make a copy before) afterwards append the following configuration with:

cat << EOF >> /etc/icinga/objects/commands.cfg
define command {
      command_name    process-service-perfdata
      command_line    /usr/local/icinga/libexec/process_perfdata.pl
}

define command {
       command_name    process-host-perfdata
       command_line    /usr/local/icinga/libexec/process_perfdata.pl -d HOSTPERFDATA
}
EOF
mv /etc/icinga/pnp/rra.cfg-sample /etc/icinga/pnp/rra.cfg

add the snmp-password in /etc/icinga/resource.cfg:

$USER4$=pvesecret
cat << EOF > /etc/icinga/check_commands/check_snmp_int.cfg
define command {
       command_name    check_snmp_int
       command_line    \$USER1\$/check_snmp_int.pl  -H \$HOSTADDRESS\$ -C \$USER4\$ -n "\$ARG1\$" "\$ARG2\$" "\$ARG3\$" -f
}

define command {
       command_name    check_snmp_int_pve
       command_line    \$USER1\$/check_snmp_int.pl  -H \$ARG1\$ -C \$USER4\$ -n "\$ARG2\$" "\$ARG3\$" "\$ARG4\$" -f
}

EOF
cat << EOF >> /etc/icinga/objects/templates.cfg

# PVE host definition template - This is NOT a real host, just a template!
define host{
       name                    pve            ; The name of this host template
       use                     generic-host   ; This template inherits other values from the generic-host template
       check_period            24x7           ; By default, Linux hosts are checked round the clock
       check_interval          5              ; Actively check the host every 5 minutes
       retry_interval          1              ; Schedule host check retries at 1 minute intervals
       max_check_attempts      5              ; Check each Linux host 10 times (max)
       check_command         check-host-alive ; Default command to check Linux hosts
       notification_period     workhours      ; Linux admins hate to be woken up, so we only notify during the day
                                              ; Note that the notification_period variable is being overridden from
                                              ; the value that is inherited from the generic-host template!
       notification_interval   120            ; Resend notifications every 2 hours
       notification_options    d,u,r          ; Only send notifications for specific host states
       contact_groups          pve-admins     ; Notifications get sent to the admins by default
       register                0              ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL HOST, JUST A TEMPLATE!
       }
EOF
cat << EOF >> /etc/icinga/objects/contacts.cfg

define contactgroup{
       contactgroup_name       pve-admins
       alias                   Proxmox Administrators
       members                 icingaadmin
       }
EOF
exit # to be support
sudo vi /etc/apache2/conf.d/pnp4nagios.conf # to change /usr/local/nagios/etc/htpasswd.users in:
<Directory "/usr/local/icinga/share/pnp">
        AllowOverride None
        Order allow,deny
        Allow from all
        #
        # Use the same value as defined in nagios.conf
        #
        AuthName "Incinga Access"
        AuthType Basic
        AuthUserFile /usr/local/icinga/etc/htpasswd.users

sudo vi /etc/php5/apache2/php.ini

magic_quotes_gpc = Off

Now it's time for a first test!

sudo /etc/init.d/apache2 restart
sudo /etc/init.d/icinga start

connect with a web browser to http://192.168.200.200/icinga
and to http://192.168.200.200/pnp4nagios
Use as account icingaadmin with the selected password
The "PNP4Nagios Environment Tests" should be passed. Ok! Further...

cd /usr/local/icinga/share/pnp/
sudo mv install.php install.php.org

Now, it's time to monitor one client

Some stuff is to be done on the proxmox-host:

ssh root@proxmox
apt-get update
apt-get install snmpd # we monitor the interfaces via snmp

change in /etc/default/snmpd the listen address from 127.0.0.1

SNMPDOPTS='-Lsd -Lf /dev/null -u snmp -I -smux -p /var/run/snmpd.pid 192.168.200.10'
vi /etc/snmp/snmpd.conf
#       sec.name  source          community
#com2sec paranoid  default         public
com2sec readonly  192.168.200.200  pvesecret
#com2sec readwrite default         private

restart snmpd:

/etc/init.d/snmpd restart

Place the script to generate icinga-configs:

mkdir /usr/local/scripts
cd /usr/local/scripts
mkdir pvekvm2icinga.d

create the script pvekvm2icinga.sh with following content:

#!/bin/bash
#
# pvekvm2icinga.sh (c) Udo Lembke 2/2010 v. 0.1 gplv3
#
# this script need the vmid as parameter

LANG=C

icinga_host=192.168.200.200
icinga_user=icinga
proxmox_hostname=`hostname`
proxmox_host=`/sbin/ifconfig vmbr0 | grep "inet addr:" | awk '{print $2}' | tr -d "addr:"`
host_preconf=/usr/local/scripts/pvekvm2icinga.d
vlan[0]=PROD
vlan[1]=INTERNET
vlan[2]=DMZ
vlan[4]=TESTNET

if [[ $# -ne 1 ]] || [[ $1 -le 100 ]]; then
   echo "call \""$0" VMID\" VMID is 101-999"
   exit 1
fi

echo "create "$1".conf for icinga"

if [[ ! -f /etc/qemu-server/$1.conf ]]; then
   echo "VM-config not found";
   exit 1;
fi
if [[ ! -f $host_preconf/$1.conf ]]; then
   echo "$host_preconf/$1.conf not accessible"
   echo "nessasary for VM-ip-address"
   exit 1;
fi

vm_name=`grep "name:" /etc/qemu-server/$1.conf | awk '{print $2}'`
vm_vlans_conf=`grep "vlan[0-9]" /etc/qemu-server/$1.conf | awk -F: '{print $1}'| tr -d "vlan"`
vm_vlans=`/sbin/ifconfig | grep "vmtab"$1 | awk '{print $1}' | awk -Fi '{print $2}'`
if [[ `echo $vm_vlans| wc -w` -ne `echo $vm_vlans_conf | wc -w` ]]; then echo "Attention, number of active interfaces are not equal to the config-file"; fi

echo "define host{" > /tmp/pvekvm2icinga_$1
echo "        use                     pve" >> /tmp/pvekvm2icinga_$1
echo "        host_name               $vm_name" >> /tmp/pvekvm2icinga_$1

alias_line=`grep "alias" $host_preconf/$1.conf`
address_line=`grep "address" $host_preconf/$1.conf`

if [[ -n $alias_line ]]; then
    echo $alias_line | awk '{print "        "$1"                   "$2,$3,$4,$5}'>> /tmp/pvekvm2icinga_$1
else echo "        alias                   $vm_name; VM$1 on $proxmox_hostname" >> /tmp/pvekvm2icinga_$1
fi
if [[ -n $address_line ]]; then
    echo $address_line | awk '{print "        "$1"                 "$2}' >> /tmp/pvekvm2icinga_$1
else echo "        address                 99.99.99.99" >> /tmp/pvekvm2icinga_$1
fi

echo "        }" >> /tmp/pvekvm2icinga_$1
echo "" >> /tmp/pvekvm2icinga_$1
echo "define service{" >> /tmp/pvekvm2icinga_$1
echo "        use                             generic-service" >> /tmp/pvekvm2icinga_$1
echo "        host_name                       $vm_name" >> /tmp/pvekvm2icinga_$1
echo "        service_description             PING" >> /tmp/pvekvm2icinga_$1
echo "        is_volatile                     0" >> /tmp/pvekvm2icinga_$1
echo "        check_period                    24x7" >> /tmp/pvekvm2icinga_$1
echo "        max_check_attempts              4" >> /tmp/pvekvm2icinga_$1
echo "        normal_check_interval           1" >> /tmp/pvekvm2icinga_$1
echo "        retry_check_interval            1" >> /tmp/pvekvm2icinga_$1
echo "        contact_groups                  pve-admins" >> /tmp/pvekvm2icinga_$1
echo "        notification_options            w,u,c,r" >> /tmp/pvekvm2icinga_$1
echo "        notification_interval           120" >> /tmp/pvekvm2icinga_$1
echo "        notification_period             24x7" >> /tmp/pvekvm2icinga_$1
echo "        check_command                   check_ping!200.0,40%!500.0,80%" >> /tmp/pvekvm2icinga_$1
echo "        }" >> /tmp/pvekvm2icinga_$1
echo "" >> /tmp/pvekvm2icinga_$1

for vlanid in $vm_vlans_conf
   do
    echo "define service{" >> /tmp/pvekvm2icinga_$1
    echo "        use                             generic-service" >> /tmp/pvekvm2icinga_$1 
    echo "        host_name                       $vm_name" >> /tmp/pvekvm2icinga_$1
    echo "        service_description             ${vlan[$vlanid]}_TRAFFIC" >> /tmp/pvekvm2icinga_$1
    echo "        is_volatile                     0" >> /tmp/pvekvm2icinga_$1
    echo "        check_period                    24x7" >> /tmp/pvekvm2icinga_$1
    echo "        max_check_attempts              4" >> /tmp/pvekvm2icinga_$1
    echo "        normal_check_interval           1" >> /tmp/pvekvm2icinga_$1
    echo "        retry_check_interval            1" >> /tmp/pvekvm2icinga_$1
    echo "        contact_groups                  pve-admins" >> /tmp/pvekvm2icinga_$1
    echo "        notification_options            w,u,c,r" >> /tmp/pvekvm2icinga_$1
    echo "        notification_interval           120" >> /tmp/pvekvm2icinga_$1
    echo "        notification_period             24x7" >> /tmp/pvekvm2icinga_$1
    echo "        check_command                   check_snmp_int_pve!$proxmox_host!vmtab$1i$vlanid!!" >> /tmp/pvekvm2icinga_$1
    echo "        }" >> /tmp/pvekvm2icinga_$1
    echo "" >> /tmp/pvekvm2icinga_$1
    echo "define serviceextinfo{" >> /tmp/pvekvm2icinga_$1 
    echo "        host_name                       $vm_name" >> /tmp/pvekvm2icinga_$1
    echo "        service_description             ${vlan[$vlanid]}_TRAFFIC" >> /tmp/pvekvm2icinga_$1
    echo "        action_url                      /pnp4nagios/graph?host=$vm_name&srv=${vlan[$vlanid]}_TRAFFIC" >> /tmp/pvekvm2icinga_$1
    echo "        #notes_url                       /wiki/index.php/$vm_name#${vlan[$vlanid]}_TRAFFIC" >> /tmp/pvekvm2icinga_$1
    echo "}" >> /tmp/pvekvm2icinga_$1
    echo "" >> /tmp/pvekvm2icinga_$1

done

scp /tmp/pvekvm2icinga_$1 $icinga_user@$icinga_host:/etc/icinga/proxmox/$1.cfg
ssh $icinga_user@$icinga_host /etc/init.d/icinga reload
# rm /tmp/pvekvm2icinga_$1
chmod +x pvekvm2icinga.sh

Modify the names for the networks for your usage
To make your life easier, copy your public key to monitoring

scp ~/.ssh/id_rsa.pub icinga@monitoring:~/.ssh/authorized_keys

To run the script with an VMID of a kvm-guest, which have bridged network we must first create a configfile in /usr/local/scripts/pvekvm2icinga.d like this:

cat 101.conf
alias Lubuntu test machine
address 192.168.200.66

alias is "can have" address is "must have".

generate the config of VMID 101 (with transfer to monitor)

/usr/local/scripts/pvekvm2icinga.sh 101

Now you should see in icinga->Service Detail the vm with the service ping and one for each network-interface. If you click on the sign near Status, you get the network-chart in an extra browser-tab (or window).

That's the beginning! You can do a lot more with incinga.

If you have trouble with the icinga-config check them with (as user icinga or root):

/usr/local/icinga/bin/icinga -v /etc/icinga/icinga.cfg

References

http://www.icinga.org
http://www.icinga-portal.org
http://www.monitoringexchange.org
http://docs.pnp4nagios.org