Auto deploy and update NRPE


Assumptions / Why things are done some ways

  • We have ssh keys on all servers for the nagios user
  • We are installing nrpe as standalone daemon
  • No where near all of our servers have openssl-devel install or other prerequisites, so precompiled was the way for us.
  • NRPE needs compiled to allow command line argument to allow the version checking to work
  • Working folder is as the nagios user on the nagios server /home/nagios/nrpe
  • We have /home/nagios/newservers/plugins as well, and that is where check_nrpe_ver.sh and restart_nrpe.sh reside and anything else(plugins/scripts) we want to send to servers

Installation

  1. Download latest nrpe(in my example nrpe2.15)
  2. Extract nrpe2.15.tar.gz
  3. Modify nrpe-2.15/sample-config/nrpe.cfg.in to fit your needs.  Make sure to add the include folder of /usr/local/nagios/etc/nrpe so it will see the checks.cfg file.  If compiled to allow arguments, make sure and change dont blame nrpe to 1.
  4. We then proceed to compile on rh4, rh5-32 and either rh5 or 6 64bit.  We then copy the compiled nrpe binary back to our nagios server into ~/nrpe/nrpe/nrpe(rh4)(32)(64) folder
  5. Add nrpe.init to ~nrpe/nrpe folder
#!/bin/bash## chkconfig: 2345 20 80
# nrpe daemon
# description: The NRPE daemon communicates with the nagios daemon \
#              transmitting vital system & hardware information about \
#              different services.
#
# Author: Nick Winn
#
# $Revision: 1.2 $
# Source function library.
. /etc/init.d/functions
# Get config.
. /etc/sysconfig/network
# Check that networking is up.
[ “${NETWORKING}” = “no” ] && exit 0
[ -f /usr/local/nagios/etc/nrpe.cfg ] || exit 0
NRPE=”/usr/local/nagios/bin/nrpe”
PIDFILE=”/var/run/nrpe.pid”
CFG=”/usr/local/nagios/etc/nrpe.cfg”
RETVAL=0
start() {
echo -n $”Starting NRPE: ”
“${NRPE}” -c $CFG -d > /dev/null
RETVAL=$?
echo “OK”
}
stop() {
if [ -f “${PIDFILE}” ]; then
echo -n $”Stopping NRPE: ”
killall ${NRPE} > /dev/null
echo “OK”
fi
}
restart() {
stop
start
}
# See how we were called.
case “$1” in
start)
start
;;
stop)
stop
;;
status)
status nrpe
RETVAL=$?
;;
reload)
restart
;;
restart)
restart
;;
condrestart)
if [ -f “${PIDFILE}” ]; then
restart
fi
;;
*)
echo $”Usage: $0 {start|stop|status|restart|condrestart}”
exit 1
;;
esac
exit $RETVAL
  1. Put checks.cfg in /home/nagios/nrpe/nrpe folder
command[restart_nrpe]=/usr/local/nagios/libexec/restart_nrpe.sh
command[check_nrpe_ver]=/usr/local/nagios/libexec/check_nrpe_ver.sh $ARG1$
  1. Put these two scripts in the working folder
install-nrpe.sh
#!/bin/sh## This is a script to automate the installation of nrpe#
# Written by Jim Clark
#
# Get Hostname
cat <<- EOF
#######################################
#                                     #
# This is the installer script        #
# for installing Nagios NRPE          #
#                                     #
#                                     #
#######################################
EOF
echo “Please enter the FQDN of the new Host”
read hostname
echo “You entered: $hostname”
#
#
RELEASE_VER=`/usr/local/nagios/libexec/check_by_ssh -H $hostname -C “cat /etc/redhat-release”`
MACHINE_TYPE=`/usr/local/nagios/libexec/check_by_ssh -H $hostname -C “uname -m”`
if [[ “$RELEASE_VER” == *”release 4″* ]]; then
echo “release 4”
scp ~/nrpe/nrperh4/* $hostname:/tmp
elif [ ${MACHINE_TYPE} == ‘x86_64’ ]; then
# 64-bit stuff here
echo “64 bit”
scp /home/nagios/nrpe/nrpe64/* $hostname:/tmp
else
# 32-bit stuff here
echo “32 bit”
scp /home/nagios/nrpe/nrpe32/* $hostname:/tmp
fi
# Connect via ssh and run script to install nrpe
scp /home/nagios/nrpe/nrpe/* $hostname:/tmp
scp /home/nagios/newservers/plugins/restart_nrpe.sh $hostname:/usr/local/nagios/libexec
scp /home/nagios/newservers/plugins/check_nrpe_ver.sh $hostname:/usr/local/nagios/libexec
ssh nagios@$hostname ‘bash -s’ — < /home/nagios/nrpe/nrperemote.sh
ssh nagios@$hostname ‘bash -s’ — < /etc/init.d/nrpe start
nrperemote.sh
#!/bin/sh## Get rootsudo su –
cd /tmp
# Copy over nrpe.cfg and nrpe init script
mkdir /usr/local/nagios/etc ; mv nrpe.cfg /usr/local/nagios/etc/nrpe.cfg ; chown -R nagios:nagios /usr/local/nagios/etc
mv nrpe.init /etc/init.d/nrpe
chmod 755 /etc/init.d/nrpe
mkdir /usr/local/nagios/etc/nrpe ; mv checks.cfg /usr/local/nagios/etc/nrpe ; chown -R nagios:nagios /usr/local/nagios/etc/nrpe
# Set nrpe to run at boot
chkconfig nrpe on
  1. Run the install-nrpe.sh script to install.

 

Automating Updating NRPE

Checking nrpe config version

  1. Create a check in Nagios to run a nrpe check that runs this script
#!/bin/bash##Script to compare nrpe config version
#
if [ ! -f /usr/local/nagios/libexec/version.cfg ]; then
echo “1” > “/usr/local/nagios/libexec/version.cfg”
fi
current_ver=$(head -n 1 /usr/local/nagios/libexec/version.cfg)
if [ $current_ver -lt “$1” ];then
echo “Warning – NRPE needs updated!”
exit 1
fi
echo “OK – NRPE does not need updated!”
exit 0
  1. Whenever you want the plugins, nrpe.cfg and checks.cfg updated, just update the command definition($USER1$/check_nrpe -H $HOSTADDRESS$ -c check_nrpe_ver.sh -a 1) and increase the command line argument by 1.  Also modify /usr/local/nagios/libexec/version.cfg to match the new version number.  That should be the only line in the file, just the number.
  2. If the current version number is lower than the commandline argument sent, then event handler event_xx_nrpe.sh is run

Upgrading nrpe config version(eventhandler event_xx_nrpe.sh)

  1. The event handler runs and only executes something if the check above returns warning and check count is at least 10.  This way it eliminates any false positives.  We will run this check every 1440 minutes with a retry of 5 and max tries of 30.
#!/bin/sh## Event handler script for updating nrpe
#
# What state is the NRPE check in?
case “$1” in
OK)
# No update require, so don’t do anything…
;;
WARNING)
# This is where we do our work to update…
# check if at least check #10
if [ $2 -gt 9 ];then
/usr/local/nagios/libexec/cdm-update.sh $3
/usr/local/nagios/libexec/nrpe-update.sh $3
fi
;;
UNKNOWN)
# We don’t know what might be causing an unknown error, so don’t do anything…
;;
CRITICAL)
# it chould never be critical, some other issue is happening, investigate manually
;;
esac
exit 0
  1. Update the plugins with the /usr/local/nagios/libexec/cdm-update.sh
#!/bin/sh
#set -e#set -u
echo “You entered: $1”
# copy from Nagios to new hostRELEASE_VER=`/usr/local/nagios/libexec/check_by_ssh -H $1 -C “cat /etc/redhat-release”`if [[ “$RELEASE_VER” == *”release 4″* ]]; then
echo “release 4”
scp /home/nagios/newservers/pluginsrh4/* $1:/usr/local/nagios/libexec
scp /home/nagios/newservers/plugins/* $1:/usr/local/nagios/libexec
echo “”
echo “Folders created and plugins copied”
exit
fi
MACHINE_TYPE=`/usr/local/nagios/libexec/check_by_ssh -H $1 -C “uname -m”`
if [ ${MACHINE_TYPE} == ‘x86_64’ ]; then
# 64-bit stuff here
echo “64 bit”
scp /home/nagios/newservers/plugins64/* $1:/usr/local/nagios/libexec
else
# 32-bit stuff here
echo “32 bit”
scp /home/nagios/newservers/plugins32/* $1:/usr/local/nagios/libexec
fi
scp /home/nagios/newservers/plugins/* $1:/usr/local/nagios/libexec
echo “”
echo “Folders created and plugins copied”
exit 0
  1. Update nrpe with /usr/local/nagios/libexec/nrpe-update.sh
#!/bin/sh
#set -e#set -u
echo “You entered: $1”
scp /home/nagios/nrpe/nrpe/nrpe.cfg $1:/usr/local/nagios/etc/nrpe.cfgscp /home/nagios/nrpe/nrpe/checks.cfg $1:/usr/local/nagios/etc/nrpe/checks.cfgscp /usr/local/nagios/libexec/version.cfg $1:/usr/local/nagios/libexec
/usr/local/nagios/libexec/check_nrpe -H $1 -c restart_nrpe
echo “NRPE Updated”
exit 0