<\/p>\n\r\n% mc-nrpe -W puppet_enabled=1 check_puppet_run\r\n\r\n * [ ============================================================> ] 47 \/ 47\r\n\r\nFinished processing 47 \/ 47 hosts in 395.51 ms\r\n OK: 47\r\n WARNING: 0\r\n CRITICAL: 0\r\n UNKNOWN: 0\r\n<\/pre>\n<\/code><\/p>\n
Note we’re restricting the run to only enabled hosts.<\/p>\n
Integrating into Nagios<\/h2>\n
The last step is to add this to nagios. I create SSL certs and a specific client configuration for Nagios and put these in it’s home directory.<\/p>\n
The check-mc-nrpe<\/em> plugin works best with Nagios 3 as it will return subsequent lines of output indicating which machines are in what state so you get the details hidden behind the aggregation in alerts. It also outputs performance data for total node, each status and also how long it took to do the check.<\/p>\nThe nagios command would be something like this:<\/p>\n
<\/p>\n\r\ndefine command{\r\n command_name check_mc_nrpe\r\n command_line \/usr\/sbin\/check-mc-nrpe --config \/var\/log\/nagios\/.mcollective\/client.cfg -W $ARG1$ $ARG2$\r\n}\r\n<\/pre>\n<\/code><\/p>\n
And finally we need to make a service:<\/p>\n
<\/p>\n\r\ndefine service{\r\n host_name monitor1\r\n service_description mc_puppet-run\r\n use generic-service\r\n check_command check_mc_nrpe!puppet_enabled=1!check_puppet_run\r\n notification_period awakehours\r\n contact_groups sysadmin\r\n}\r\n<\/pre>\n<\/code><\/p>\n
Here are a few other command examples I use:<\/p>\n
All machines with my Puppet class “pki”, check the age of certs:<\/p>\n
<\/p>\n\r\ncheck_command check_mc_nrpe!pki!check_pki\r\n<\/pre>\n<\/code><\/p>\n
All machines with my Puppet class “bacula::node”, make sure the FD is running:<\/p>\n
<\/p>\n\r\ncheck_command check_mc_nrpe!bacula::node!check_fd\r\n<\/pre>\n<\/code><\/p>\n
…and that they were backed up:<\/p>\n
<\/p>\n\r\ncheck_command check_mc_nrpe!bacula::node!check_bacula_main\r\n<\/pre>\n<\/code><\/p>\n
Using this I removed 100s of checks from my monitoring platform, saving on resources and making sure I can do my critical monitor tasks better.<\/p>\n
Depending on the quality of your monitoring system you might even get a graph showing the details hidden behind the aggregation:<\/p>\n
![](\"http:\/\/www.devco.net\/images\/mcbacula.png\")
<\/center><\/p>\nThe above is a graph showing a series of servers where the backup ran later than usual, I had 2 alerts only, would have had more than 30 before aggregation.<\/p>\n
<\/p>\n
Restrictions for Probes<\/h2>\n
The last remaining requirement I had was to be able to do checks on different probes and restrict them. My Collective is one big one spread all over the world which means sometimes things are a bit slow discovery wise. <\/p>\n
So I have many nagios servers doing local checks. Using MCollective discovery I can now easily restrict checks, for example If I only wanted to check machines in the USA and I had a fact country<\/em> I only have to change my command line in the service declaration:<\/p>\n<\/p>\n\r\ncheck_command check_mc_nrpe!puppet_enabled=1 country=us!check_puppet_run\r\n<\/pre>\n<\/code><\/p>\n
This will then via MCollective discovery just monitor machines in the US.<\/p>\n
What to monitor this way<\/h2>\n
As this style of monitoring is done using Discovery you would need to think carefully about what you monitor this way. It’s totally conceivable that if a node is under high CPU load that it wont respond to discovery commands in time, and so wont get monitored!<\/p>\n
You would then for example not want to monitor things like load averages or really critical services this way, but we all have a lot of peripheral things like zombie process counts and a lot of other places where aggregation makes a lot of sense, in those cases by all means consider this approach.<\/p>\n","protected":false},"excerpt":{"rendered":"
A very typical scenario I come across on many sites is the requirement to monitor something like Puppet across 100s or 1000s of machines. The typical approaches are to add perhaps a central check on your puppet master or to check using NRPE or NSCA on every node. For this example the option exist to […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","footnotes":""},"categories":[7],"tags":[121,85,78,64],"_links":{"self":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/1532"}],"collection":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/comments?post=1532"}],"version-history":[{"count":28,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/1532\/revisions"}],"predecessor-version":[{"id":1694,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/1532\/revisions\/1694"}],"wp:attachment":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/media?parent=1532"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/categories?post=1532"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/tags?post=1532"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}