Monitoring with Icinga
From Proxmox VE
Introduction
This wikipage describes the installation of icinga (with pnp for graphical output) on a debian-box
Icinga is a fork of Nagios - the config is the same. If you need further information search also for nagios.
It's take some time to install icinga but if you follow this guide step by step you (hopefully) got an running installation.
To get the goal faster - I would make a preconfig-vm for proxmox (based on devil-linux), but not yet...
Normally the monitoring-host should be a standalone stable machine, for test it's ok in a vm.
Some definitions
I call the debian-box monitoring and, if we need the ip, it's in this example 192.168.200.200
The proxmox-host is called proxmox and has the ip 192.168.200.10
The VM which should be monitored has the ip 192.168.200.66
Put into practice
install on monitoring debian-504-i386-netinst.iso - select only standard
login as root
We need some more stuff
apt-get update apt-get install openssh-server sudo lynx apache2 build-essential \ libgd2-xpm-dev librrd-dev rrdtool librrds-perl librrdp-perl \ libnet-snmp-perl php5 php5-gd unzip
At this example we use as working-user support (defined during the installation process)
ssh support@192.168.200.200
Enable support for sudo
su visudo -f /etc/sudoers
uncomment the last line:
%sudo ALL=NOPASSWD: ALL
addgroup support sudo
Create a user for icinga
/usr/sbin/useradd -m icinga passwd icinga exit # again support exit # logoff ssh support@192.168.200.200 # to activate changes in group
mkdir ~/software cd ~/software
Build and install icinga
Current Version - 1.5.1 (Circa Sep 2011)
wget http://downloads.sourceforge.net/project/icinga/icinga/1.2.1/icinga-1.2.1.tar.gz tar xvzf icinga-1.2.1.tar.gz cd icinga-1.2.1 ./configure make all sudo make fullinstall cd /etc/rc2.d/ sudo ln -s ../init.d/icinga S92icinga cd ../rc0.d sudo ln -s ../init.d/icinga K10icinga sudo vi /usr/local/icinga/etc/objects/contacts.cfg # change the email-address of icingaadmin sudo htpasswd -c /usr/local/icinga/etc/htpasswd.users icingaadmin # this account/password is for the webfrontend sudo ln -s /usr/local/icinga/etc /etc/icinga # to access the config below /etc sudo addgroup www-data icinga
Build and install the nagios-plugins
Version is current as of Sep 2011.
cd ~/software wget http://sourceforge.net/projects/nagiosplug/files/nagiosplug/1.4.15/nagios-plugins-1.4.15.tar.gz tar xvzf nagios-plugins-1.4.15.tar.gz cd nagios-plugins-1.4.15 ./configure --prefix=/usr/local/icinga --with-nagios-user=icinga --with-cgiurl=/icinga/cgi-bin make sudo make install
Install snmp-plugin
cd ~/software wget http://nagios.manubulon.com/check_snmp_int.pl chmod +x check_snmp_int.pl sudo chown icinga check_snmp_int.pl sudo mv check_snmp_int.pl /usr/local/icinga/libexec/
Build and install pnp
cd ~/software
Download with an webbrowser kohana http://kohanaphp.com/download and scp it to support@monitoring:software/ - or with lynx directly on monitoring. Current versions are 3.2.0 and 3.1.4.
cd ~/software unzip kohana-3.0.8.zip sudo mv kohana /var/www/ sudo chown -R www-data:www-data /var/www/kohana sudo mv /var/www/kohana/install.php /var/www/kohana/install.php.org
PHP4Nagois Current Version is 0.6.15
wget http://downloads.sourceforge.net/project/pnp4nagios/PNP-0.6/pnp4nagios-0.6.7.tar.gz tar xvzf pnp4nagios-0.6.7.tar.gz cd pnp4nagios-0.6.7/ USER=icinga GROUP=icinga PREFIX=/usr/local/icinga ./configure --with-nagios-user=$USER --with-nagios-group=$GROUP --sysconfdir=$PREFIX/etc/pnp \ --prefix=$PREFIX --datarootdir=$PREFIX/share/pnp --with-rrdtool=/usr/bin/rrdtool \ --sysconfdir=$PREFIX/etc/pnp --with-perfdata-dir=$PREFIX/share/perfdata \ --with-kohana_system=/var/www/kohana/system --exec-prefix=$PREFIX \ --with-perfdata-logfile=$PREFIX/var/perfdata.log \ --with-perfdata-spool-dir=$PREFIX/var/spool/perfdata make all sudo make install sudo make install-webconf sudo make install-config sudo ln -s /usr/local/icinga/lib/kohana/system /usr/local/icinga/share/pnp/system sudo vi /usr/local/icinga/share/pnp/index.php # change "$kohana_system = '/var/www/kohana/system';" to $kohana_system = '/usr/local/icinga/lib/kohana/system';
Configuration
cd /etc/apache2/mods-enabled sudo ln -s ../mods-available/rewrite.load . cd /etc/icinga su icinga ssh-keygen # we need for further plugins the ssh-key - not in this example mv /etc/icinga/pnp/process_perfdata.cfg-sample /etc/icinga/pnp/process_perfdata.cfg mkdir proxmox mkdir check_commands # add two lines to icinga.cfg: cfg_dir=/usr/local/icinga/etc/proxmox cfg_dir=/usr/local/icinga/etc/check_commands
change in icinga.cfg "process_performance_data=0" to
process_performance_data=1
and uncomment the following two lines:
host_perfdata_command=process-host-perfdata service_perfdata_command=process-service-perfdata
comment out the last two blocks in /etc/icinga/objects/commands.cfg (perfdata), or delete the line (but make a copy before)
afterwards append the following configuration with:
cat << EOF >> /etc/icinga/objects/commands.cfg
define command {
command_name process-service-perfdata
command_line /usr/local/icinga/libexec/process_perfdata.pl
}
define command {
command_name process-host-perfdata
command_line /usr/local/icinga/libexec/process_perfdata.pl -d HOSTPERFDATA
}
EOF
mv /etc/icinga/pnp/rra.cfg-sample /etc/icinga/pnp/rra.cfg
add the snmp-password in /etc/icinga/resource.cfg:
$USER4$=pvesecret
cat << EOF > /etc/icinga/check_commands/check_snmp_int.cfg
define command {
command_name check_snmp_int
command_line \$USER1\$/check_snmp_int.pl -H \$HOSTADDRESS\$ -C \$USER4\$ -n "\$ARG1\$" "\$ARG2\$" "\$ARG3\$" -f
}
define command {
command_name check_snmp_int_pve
command_line \$USER1\$/check_snmp_int.pl -H \$ARG1\$ -C \$USER4\$ -n "\$ARG2\$" "\$ARG3\$" "\$ARG4\$" -f
}
EOF
cat << EOF >> /etc/icinga/objects/templates.cfg
# PVE host definition template - This is NOT a real host, just a template!
define host{
name pve ; The name of this host template
use generic-host ; This template inherits other values from the generic-host template
check_period 24x7 ; By default, Linux hosts are checked round the clock
check_interval 5 ; Actively check the host every 5 minutes
retry_interval 1 ; Schedule host check retries at 1 minute intervals
max_check_attempts 5 ; Check each Linux host 10 times (max)
check_command check-host-alive ; Default command to check Linux hosts
notification_period workhours ; Linux admins hate to be woken up, so we only notify during the day
; Note that the notification_period variable is being overridden from
; the value that is inherited from the generic-host template!
notification_interval 120 ; Resend notifications every 2 hours
notification_options d,u,r ; Only send notifications for specific host states
contact_groups pve-admins ; Notifications get sent to the admins by default
register 0 ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL HOST, JUST A TEMPLATE!
}
EOF
cat << EOF >> /etc/icinga/objects/contacts.cfg
define contactgroup{
contactgroup_name pve-admins
alias Proxmox Administrators
members icingaadmin
}
EOF
exit # to be support sudo vi /etc/apache2/conf.d/pnp4nagios.conf # to change /usr/local/nagios/etc/htpasswd.users in:
<Directory "/usr/local/icinga/share/pnp">
AllowOverride None
Order allow,deny
Allow from all
#
# Use the same value as defined in nagios.conf
#
AuthName "Incinga Access"
AuthType Basic
AuthUserFile /usr/local/icinga/etc/htpasswd.users
sudo vi /etc/php5/apache2/php.ini
magic_quotes_gpc = Off
Now it's time for a first test!
sudo /etc/init.d/apache2 restart sudo /etc/init.d/icinga start
connect with a web browser to http://192.168.200.200/icinga
and to http://192.168.200.200/pnp4nagios
Use as account icingaadmin with the selected password
The "PNP4Nagios Environment Tests" should be passed. Ok! Further...
cd /usr/local/icinga/share/pnp/ sudo mv install.php install.php.org
Now, it's time to monitor one client
Some stuff is to be done on the proxmox-host:
ssh root@proxmox apt-get update apt-get install snmpd # we monitor the interfaces via snmp
change in /etc/default/snmpd the listen address from 127.0.0.1
SNMPDOPTS='-Lsd -Lf /dev/null -u snmp -I -smux -p /var/run/snmpd.pid 192.168.200.10'
vi /etc/snmp/snmpd.conf # sec.name source community #com2sec paranoid default public com2sec readonly 192.168.200.200 pvesecret #com2sec readwrite default private
restart snmpd:
/etc/init.d/snmpd restart
Place the script to generate icinga-configs:
mkdir /usr/local/scripts cd /usr/local/scripts mkdir pvekvm2icinga.d
create the script pvekvm2icinga.sh with following content:
#!/bin/bash
#
# pvekvm2icinga.sh (c) Udo Lembke 2/2010 v. 0.1 gplv3
#
# this script need the vmid as parameter
LANG=C
icinga_host=192.168.200.200
icinga_user=icinga
proxmox_hostname=`hostname`
proxmox_host=`/sbin/ifconfig vmbr0 | grep "inet addr:" | awk '{print $2}' | tr -d "addr:"`
host_preconf=/usr/local/scripts/pvekvm2icinga.d
vlan[0]=PROD
vlan[1]=INTERNET
vlan[2]=DMZ
vlan[4]=TESTNET
if [[ $# -ne 1 ]] || [[ $1 -le 100 ]]; then
echo "call \""$0" VMID\" VMID is 101-999"
exit 1
fi
echo "create "$1".conf for icinga"
if [[ ! -f /etc/qemu-server/$1.conf ]]; then
echo "VM-config not found";
exit 1;
fi
if [[ ! -f $host_preconf/$1.conf ]]; then
echo "$host_preconf/$1.conf not accessible"
echo "nessasary for VM-ip-address"
exit 1;
fi
vm_name=`grep "name:" /etc/qemu-server/$1.conf | awk '{print $2}'`
vm_vlans_conf=`grep "vlan[0-9]" /etc/qemu-server/$1.conf | awk -F: '{print $1}'| tr -d "vlan"`
vm_vlans=`/sbin/ifconfig | grep "vmtab"$1 | awk '{print $1}' | awk -Fi '{print $2}'`
if [[ `echo $vm_vlans| wc -w` -ne `echo $vm_vlans_conf | wc -w` ]]; then echo "Attention, number of active interfaces are not equal to the config-file"; fi
echo "define host{" > /tmp/pvekvm2icinga_$1
echo " use pve" >> /tmp/pvekvm2icinga_$1
echo " host_name $vm_name" >> /tmp/pvekvm2icinga_$1
alias_line=`grep "alias" $host_preconf/$1.conf`
address_line=`grep "address" $host_preconf/$1.conf`
if [[ -n $alias_line ]]; then
echo $alias_line | awk '{print " "$1" "$2,$3,$4,$5}'>> /tmp/pvekvm2icinga_$1
else echo " alias $vm_name; VM$1 on $proxmox_hostname" >> /tmp/pvekvm2icinga_$1
fi
if [[ -n $address_line ]]; then
echo $address_line | awk '{print " "$1" "$2}' >> /tmp/pvekvm2icinga_$1
else echo " address 99.99.99.99" >> /tmp/pvekvm2icinga_$1
fi
echo " }" >> /tmp/pvekvm2icinga_$1
echo "" >> /tmp/pvekvm2icinga_$1
echo "define service{" >> /tmp/pvekvm2icinga_$1
echo " use generic-service" >> /tmp/pvekvm2icinga_$1
echo " host_name $vm_name" >> /tmp/pvekvm2icinga_$1
echo " service_description PING" >> /tmp/pvekvm2icinga_$1
echo " is_volatile 0" >> /tmp/pvekvm2icinga_$1
echo " check_period 24x7" >> /tmp/pvekvm2icinga_$1
echo " max_check_attempts 4" >> /tmp/pvekvm2icinga_$1
echo " normal_check_interval 1" >> /tmp/pvekvm2icinga_$1
echo " retry_check_interval 1" >> /tmp/pvekvm2icinga_$1
echo " contact_groups pve-admins" >> /tmp/pvekvm2icinga_$1
echo " notification_options w,u,c,r" >> /tmp/pvekvm2icinga_$1
echo " notification_interval 120" >> /tmp/pvekvm2icinga_$1
echo " notification_period 24x7" >> /tmp/pvekvm2icinga_$1
echo " check_command check_ping!200.0,40%!500.0,80%" >> /tmp/pvekvm2icinga_$1
echo " }" >> /tmp/pvekvm2icinga_$1
echo "" >> /tmp/pvekvm2icinga_$1
for vlanid in $vm_vlans_conf
do
echo "define service{" >> /tmp/pvekvm2icinga_$1
echo " use generic-service" >> /tmp/pvekvm2icinga_$1
echo " host_name $vm_name" >> /tmp/pvekvm2icinga_$1
echo " service_description ${vlan[$vlanid]}_TRAFFIC" >> /tmp/pvekvm2icinga_$1
echo " is_volatile 0" >> /tmp/pvekvm2icinga_$1
echo " check_period 24x7" >> /tmp/pvekvm2icinga_$1
echo " max_check_attempts 4" >> /tmp/pvekvm2icinga_$1
echo " normal_check_interval 1" >> /tmp/pvekvm2icinga_$1
echo " retry_check_interval 1" >> /tmp/pvekvm2icinga_$1
echo " contact_groups pve-admins" >> /tmp/pvekvm2icinga_$1
echo " notification_options w,u,c,r" >> /tmp/pvekvm2icinga_$1
echo " notification_interval 120" >> /tmp/pvekvm2icinga_$1
echo " notification_period 24x7" >> /tmp/pvekvm2icinga_$1
echo " check_command check_snmp_int_pve!$proxmox_host!vmtab$1i$vlanid!!" >> /tmp/pvekvm2icinga_$1
echo " }" >> /tmp/pvekvm2icinga_$1
echo "" >> /tmp/pvekvm2icinga_$1
echo "define serviceextinfo{" >> /tmp/pvekvm2icinga_$1
echo " host_name $vm_name" >> /tmp/pvekvm2icinga_$1
echo " service_description ${vlan[$vlanid]}_TRAFFIC" >> /tmp/pvekvm2icinga_$1
echo " action_url /pnp4nagios/graph?host=$vm_name&srv=${vlan[$vlanid]}_TRAFFIC" >> /tmp/pvekvm2icinga_$1
echo " #notes_url /wiki/index.php/$vm_name#${vlan[$vlanid]}_TRAFFIC" >> /tmp/pvekvm2icinga_$1
echo "}" >> /tmp/pvekvm2icinga_$1
echo "" >> /tmp/pvekvm2icinga_$1
done
scp /tmp/pvekvm2icinga_$1 $icinga_user@$icinga_host:/etc/icinga/proxmox/$1.cfg
ssh $icinga_user@$icinga_host /etc/init.d/icinga reload
# rm /tmp/pvekvm2icinga_$1
chmod +x pvekvm2icinga.sh
Modify the names for the networks for your usage
To make your life easier, copy your public key to monitoring
scp ~/.ssh/id_rsa.pub icinga@monitoring:~/.ssh/authorized_keys
To run the script with an VMID of a kvm-guest, which have bridged network we must first create a configfile in /usr/local/scripts/pvekvm2icinga.d like this:
cat 101.conf alias Lubuntu test machine address 192.168.200.66
alias is "can have" address is "must have".
generate the config of VMID 101 (with transfer to monitor)
/usr/local/scripts/pvekvm2icinga.sh 101
Now you should see in icinga->Service Detail the vm with the service ping and one for each network-interface. If you click on the sign near Status, you get the network-chart in an extra browser-tab (or window).
That's the beginning! You can do a lot more with incinga.
If you have trouble with the icinga-config check them with (as user icinga or root):
/usr/local/icinga/bin/icinga -v /etc/icinga/icinga.cfg
Links
http://www.icinga.org
http://www.icinga-portal.org
http://www.monitoringexchange.org
http://docs.pnp4nagios.org
