Ganglia Nagios Integration

Ganglia Nagios Integration

Ganglia Nagios integration is a new feature that is included with Ganglia Web 2.2.0+. It is based on following implementation

[http://vuksan.com/linux/nagios_scripts.html#check_ganglia_metrics]

with the exception that it uses a shell script wrapper which is more efficient since PHP interpreter doesn't need to be spawned each time we check a metric.

There are 4 different Ganglia Checks

• Check heartbeat
• Check single metric on a specific host
• Check multiple metrics on a specific host
• Check multiple metrics on a range of hosts defined with a regular expression

Check Heartbeat

Ganglia uses heartbeat packets to determine if a machine has gone down. It is reset every time a new packet is received. This check avoids you from having to do things like check_ping to make sure machine is alive. To use this check please copy check_heartbeat.sh script from nagios subdirectory in Ganglia Web tarball. Make sure that the Ganglia Web URL inside the script is correct. This is the default

GANGLIA_URL="http://localhost/ganglia2/nagios/check_heartbeat.php"

Define the check command in Nagios. Threshold is the amount of time since last reported heartbeat to raise critical alert.

define command {
  command_name  check_ganglia_heartbeat
  command_line  /bin/sh /var/www/html/ganglia/nagios/check_heartbeat.sh host=$HOSTADDRESS$ threshold=$ARG1$
}

Now for every host you want monitored change check_command to be

  check_command   check_ganglia_heartbeat!50

This will mark any node that reported to Ganglia 50 seconds or more ago as CRITICAL.

Check single metric on a specific host

To use it please copy check_ganglia_metric.sh script from nagios subdirectory in Ganglia Web tarball. Make sure that the Ganglia Web URL inside the script is correct. This is the default

GANGLIA_URL="http://localhost/ganglia2/nagios/check_metric.php"

Nagios configuration consists of defining following command

define command {
  command_name  check_ganglia_metric
  command_line  /bin/sh /var/www/html/ganglia/nagios/check_ganglia_metric.sh host=$HOSTADDRESS$ metric_name=$ARG1$ operator=$ARG2$ critical_value=$ARG3$
}

Now you can use it in a service check. For instance say you want to be alerted if 1-minute load average goes over 5 you would add following directive

        check_command         check_ganglia_metric!load_one!more!5

If you wanted to alert when disk space goes less than 10 GB

        check_command         check_ganglia_metric!disk_free!less!10

Be reminded that operators indicate what should be "critical" state. For instance if you use notequal it means state is critical if the value is NOT equal. etc.

Check multiple metrics on a specific host

Check multiple metrics is a modification of the check single metric script. It will check multiple metrics on the same host e.g. instead of having separate checks for e.g. disk utilization on /, /tmp and /var which may produce three separate alerts you have a single alert any time disk utilization goes below or above a threshold.

To use it please copy check_multiple_metrics.sh script from nagios subdirectory in Ganglia Web tarball. Make sure that the Ganglia Web URL inside the script is correct. This is the default

GANGLIA_URL="http://localhost/ganglia2/nagios/check_multiple_metrics.php"

Then define a check command in Nagios

define command {
  command_name  check_ganglia_multiple_metrics
  command_line  /bin/sh /var/www/html/ganglia/nagios/check_multiple_metrics.sh host=$HOSTADDRESS$ checks='$ARG1$'
}

Then add a list of checks that are delimited with :. Each check consists of

metric_name,operator,critical_value

E.g.

  check_command       check_ganglia_multiple_metrics!disk_free_rootfs,less,10:disk_free_tmp,less,20

WARNING: Drawback of using check multiple metrics is that in certain instances you may not be aware of the scale of a problem. For example say you get an alert for /tmp nearing full. You get this alert over the weekend so you figure it's not THAT critical. After the alert your /var starts rapidly filling up which may be really serious. Unfortunately you will not get another alert (unless obviously you had an aggressive notification interval). Beware.

Check multiple metrics on a range of hosts defined with a regular expression

Use this check to check a single or multiple metrics on a range of hosts defined using a regular expression. This is useful when you want to get a single alert if particular metric is critical across a number of hosts.

To use it please copy check_multiple_metrics.sh script from nagios subdirectory in Ganglia Web tarball. Make sure that the Ganglia Web URL inside the script is correct. This is the default

GANGLIA_URL="http://localhost/ganglia2/nagios/check_host_regex.php"

Then define a check command in Nagios

define command {
  command_name  check_ganglia_host_regex
  command_line  /bin/sh /usr/share/ganglia-web2/nagios/check_host_regex.sh hreg='$ARG1$' checks='$ARG2$'
}

Then add a list of checks that are delimited with :. Each check consists of

metric_name,operator,critical_value

E.g.

For example to check free space on / and /tmp for any machine starting with web-* or app-* you would use something like this

  check_command       check_ganglia_host_regex!^web-|^app-!disk_free_rootfs,less,10:disk_free_tmp,less,10

DOWNSIDES: Downside of this approach similar to check multiple metrics on a single host is that in certain situation the scale of a problem may not be apparent since only a single alert will be generated. Also currently since Nagios and Ganglia are decoupled you may get an alert if machine is under scheduled maintenance and e.g. you start writing to /tmp.

Check value(s) is same on a set of hosts

Use this check to check a single or multiple metrics on a range of hosts have the same value. For example let's say you wanted to make sure SVN revision of the deployed code was the same across all servers. You would send the SVN revision as e.g. a string metric then list it as metric that needs to be same everywhere

To use it please copy check_value_same_everywhere.sh script from nagios subdirectory in Ganglia Web tarball. Make sure that the Ganglia Web URL inside the script is correct. This is the default

GANGLIA_URL="http://localhost/ganglia2/nagios/check_value_same_everywhere.php"

Then define a check command in Nagios

define command {
  command_name  check_ganglia_host_regex
  command_line  /bin/sh /usr/share/ganglia-web2/nagios/check_value_same_everywhere.sh hreg='$ARG1$' checks='$ARG2$'
}

e.g.

  check_command       check_ganglia_host_regex!^web-|^app-!svn_revision,num_config_files