Overview

Out of the box Nagios can actively check hosts and services across the network to ensure that they are online and responding as configured. In situations where remote hosts cannot be directly accessed by the Nagios server passive checks can be setup to allow those hosts and services to push their status to the Nagios server. This blog posting is a tutorial that walks through setting up a passive check that can be securely updated with WGET or CURL.

Background

Passive checks are very useful for replacing notifications from scheduled tasks that send email to report the task completed. Scheduled task email messages are prone to getting missed. A notification email that is has nothing meaningful to report 99% of the time is likely to be the last email looked at because administrators and users are going to handle the more pressing issues first. Email messages like these are also most likely to be forgotten and never missed until something goes wrong.

Transitioning to passive checks:

• Reduces the flow of superfluous information an administrator must review allowing them to easily focus on important issues.
• More permanently documents scheduled tasks and allows other users to more easily see system status from the Nagios administration panel.

Requirements

To follow this blog post you'll need an Ubuntu 14 Nagios 3.5.1 (or later) installation. The instructions should apply to source core installations if you update the appropriate paths.

Setup the Nagios Passive Check 

The following steps will walk you through the setup of a configuration file that contains all of the configuration directives required.

1. Login to the server shell account as a privileged user.
2. Configure nagios to check external commands by setting check_external_commands=1 in /etc/nagios3/nagios.cfg
3. Add the www-data user to the nagios group so it can issue nagios commands:
   usermod -a -G nagios www-data
4. Adjust the directory permissions so the www-data user can post nagios commands to the directory:
   chmod g+x /var/lib/nagios3/rw
5. Create a new htpasswd user for the remote host. This user account is used by the remote host's wget/curl command to update the passive Nagios check.
   htpasswd /etc/nagios3/htpasswd.users elly-9i8up98j
   In this example elly-9i8up98j will be the username for the remote host. You will be prompted to enter a password for this account. Please choose a strong password and note it for use later.
6. Download the example configuration file or copy the following configuration into /etc/nagios3/conf.d/elly.cfg file.

   ; Note: For the sake of brevity these configuration directives are
   ; contained in a single file. You may need to break these directives
   ; into other files depending on your Nagios installation configuration.
   ;
   ; Commands needed for stale host and service checks.
   ; 
   define command {
   ;Command changes a stale host from OK to WARNING
    command_name    stale_check
    command_line    /usr/lib/nagios/plugins/check_dummy 2 "CRITICAL: Stale Result"
   }
   
   define command {
   ;Command always returns true. Useful for keeping host status OK.
    command_name    check_null
    command_line    /bin/true
   }
   
   ;
   ; For elly.cfg
   ; Define contact, contactgroup, host, and service
   ;
   define contact {
    ; Name corresponding to htpasswd username and password
    contact_name					elly-9i8up98j
    alias							Elly
    service_notification_period	24x7
    host_notification_period		24x7
    service_notification_options	w,u,c,r
    host_notification_options		d,r
    service_notification_commands	notify-service-by-email
    host_notification_commands		notify-host-by-email
    email							[email protected]
    }
   
   define contactgroup {
    contactgroup_name	elly-group
    alias				Elly Contacts
    ; Add the contact directive member to contact group
    members			elly-9i8up98j
   }
   
   define host {
    use					generic-host
    ; Name of host passive check will reference when updating the service/host status
    host_name				elly
    alias					Elly
    ; Assume host cannot be reached for active checks so the address can be arbitrary / not reachable
    address				elly.thecongruity.com
    ; Assume host can’t be directly contacted and will report in status. 
    active_checks_enabled	0
    ; Accept passive checks for this host
    passive_checks_enabled	1
    ; This associates which contacts (and htpasswd users) are allowed to update this host and service.
    contact_groups			elly-group
   }
   
   define service{
    use						generic-service
    ; Associate this check with a host 
    host_name					elly
    ; Name of service passive check will reference when updating the status of this service.
    service_description		MyNightlyTask
    ; When service becomes stale this check will be run to change the state to stale.
    check_command				stale_check
    ; A service is considered stale when freshness_threshold (in seconds)
    ; is reached. Set this to 1 to run the stale check as soon as the freshness threshold is reached.
    check_interval				1
    ; Fail service after first active stale check
    max_check_attempts			1
    ; Assume service state is OK
    initial_state				o
    ; Who should receive notifications for this service?
    contact_groups				admins
    ; Disable active checks (will still trigger active check after freshness threshold
    active_checks_enabled		0
    ; Enable passive checks and ensure it is checked for freshness.
    passive_checks_enabled		1
    check_freshness			1
    ; Time since last passive check update before service is marked as stale
    freshness_threshold		100800 ; Result stale after 28 Hours (28 * 60 * 60)
    flap_detection_enabled		0
    check_interval				10
    notification_period		24x7
    ; Only notify once when service is marked as stale
    notification_interval		0
   }
   

7. Restart the nagios service.
   service nagios3 restart
8. If there were no errors with the new configuration file you should be able to login to the nagios web interface as the nagiosadmin user and view the new service. The image below shows an example of the new MyNightlyTask check.

This completes the setup of the MyNightlyTask service. The configuration will mark the task status as stale after 28 hours which will change the status in the Nagios console and send out an email alert. To keep the task in the 'OK' state the task needs to report to the Nagios service at least once every 28 hours. To do this, use the following WGET and CURL examples.

In the commands below change the values in bold with the appropriate values from your configuration.

• elly-9i8up98j - This is the name of the user created when the check was setup. It should correspond to the username of the htpasswd user and the Nagios contact contact_name setting.
• THEHTPASSWD -This value corresponds to the password assigned to the htpasswd user account. It should be kept secret and secure.
• elly - The Nagios host host_name of the host associated with the service to update. 
• MyNightlyTask - The name of the Nagios passive service check that will be updated.
• nagiosdemo.congruityservice.com - The domain name or IP address of the Nagios server.

Updating The Passive Check Using Linux Wget

wget -O - --user=elly-9i8up98j --password=THEHTPASSWD "https://nagiosdemo.congruityservice.com/cgi-bin/nagios3/cmd.cgi?cmd_typ=30&cmd_mod=2&host=elly&service=MyNightlyTask&plugin_state=0&plugin_output=CheckOK&btnSubmit=Commit"

Updating The Passive Check Using Linux Curl

echo 'user="elly-9i8up98j:THEHTPASSWD"' | curl -K - "https://nagiosdemo.congruityservice.com/cgi-bin/nagios3/cmd.cgi?cmd_typ=30&cmd_mod=2&host=elly&service=MyNightlyTask&plugin_state=0&plugin_output=CheckOK&btnSubmit=Commit"

Checking The Results

The command parameter values of plugin_state and plugin_output correspond to the state and status information being reported to Nagios. When plugin_state=0 the service is reported to be in an OK state and plugin_output=CheckOK indicates what should be displayed in the service's status information field. 

After running the command to update the state the service status information shows the status last reported, when the last check time was reported, and also that active checks are disabled.

If the service stops being updated Nagios will fail the service once the freshness_threshold value is exceeded.

An alert from the passive check can be immediately generated if the passive check is updated with plugin_state=2. This indicates the service has failed and requires attention. For other values of plugin_state see the Plugin API documentation.

Epilogue

This article has demonstrated how to configure Nagios for passive checks and update that passive check using CURL or WGET. For additional documentation regarding passive checks and host freshness the Nagios 3.x documentation is recommended. Thanks to the person who wrote this tip which served as a framework.