MCollective 0.4.3 Auditing

I just released version 0.4.3 of mcollective which brings a new auditing capability to SimpleRPC. Using the auditing system you can log to a file on each host every request or build a centralized auditing system for all requests on all nodes.

We ship a simple plugin that logs to the local harddrive but there is also a community plugin that creates a centralized logging system running over MCollective as a transport.

This is the kind of log the centralized logger will produce:

01/24/10 18:24:20 dev1.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: 01/24/10 18:24:20 caller=uid=500@ids1.my.net agent=iptables action=block
01/24/10 18:24:20 dev1.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: {:ipaddr=>"114.255.136.120"}
01/24/10 18:24:20 dev2.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: 01/24/10 18:24:20 caller=uid=500@ids1.my.net agent=iptables action=block
01/24/10 18:24:20 dev2.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: {:ipaddr=>"114.255.136.120"}
01/24/10 18:24:20 dev3.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: 01/24/10 18:24:20 caller=uid=500@ids1.my.net agent=iptables action=block
01/24/10 18:24:20 dev3.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: {:ipaddr=>"114.255.136.120"}

Here we see 3 nodes that got a request to add 114.255.136.120 to their local firewall. The request was sent by UID 500 on the machine ids1.my.net. The request is of course the same everywhere so the request id is the same on every node, the log shows agent and all parameters passed.

MCollective 0.4.2 released

Just a quick blog post for those who follow me here to get notified about new releases of MCollective. I just released version 0.4.2 which brings in big improvements for Debian packages, some tweaks to command line and a bug fix in SimpleRPC.

Read all about it at the Release Notes

MCollective Simple RPC

MCollective is a framework for writing RPC style tools that talk to a cloud of servers, till now doing that has been surprisingly hard for non ruby coders. The reason for this is that I was focussing on getting the framework built and feeling my way around the use cases.

I've now spent 2 days working on simplifying actually writing agents and consumers. This code is not released yet - just in SVN trunk - but here's a taster.

First writing an agent should be simple, here's a simple 'echo' server that takes a message as input and returns it back.

class Rpctest<RPC::Agent
    # Basic echo server
    def echo_action(request, reply)
         raise MissingRPCData, "please supply a :msg" unless request.include?(:msg)
 
         reply.data = request[:msg]
    end
end

This creates an echo action, does a quick check that a message was received and sends it back. I want to create a few more validators so you can check easily if the data passed to you is sane and secure if you're doing anything like system() calls with it.

Here's the client code that calls the echo server 4 times:

#!/usr/bin/ruby
 
require 'mcollective'
 
include MCollective::RPC
 
rpctest = rpcclient("rpctest")
 
puts "Normal echo output, non verbose, shouldn't produce any output:"
printrpc rpctest.echo(:msg => "hello world")
 
puts "Flattened echo output, think combined 'mailq' usecase:"
printrpc rpctest.echo(:msg => "hello world"), :flatten => true
 
puts "Forced verbose output, if you always want to see every result"
printrpc rpctest.echo(:msg => "hello world"), :verbose => true
 
puts "Did not specify needed input:"
printrpc rpctest.echo

This client supports full discovery and all the usual stuff, has pretty --help output and everything else you'd expect in the clients I've supplied with the core mcollective. It caches discovery results so above code will do one discovery only and reuse it for the other calls to the collective.

When running you'll see a twirling status indicator, something like:

  - [5 / 10]

This will give you a nice non scrolling indicator of progress and should work well for 100s of machines without spamming you with noise, at the end of the run you'll get the output.

The printrpc helper function tries its best to print output for you in a way that makes sense on large amounts of machines.

• By default it doesn't print stuff that succeeds, you do get a overall progress indicator though
• If anything does go wrong, useful information gets printed but only for hosts that had problems
• If you ran the client with --verbose, or forced it to verbose mode output you'll get a full bit of info of the result from every server.
• It supports flags to modify the output, you can flatten the output so hostnames etc aren't showed, just a concat of the data.

The script above gives the following output when run in non-verbose mode:

$ rpctest.rb --with-class /devel/
 
Normal echo output, non verbose, shouldn't produce any output:
 
Forced verbose output, if you always want to see every result:
dev1.your.com                          : OK
    "hello world"
 
dev2.your.com                          : OK
    "hello world"
 
dev3.your.com                          : OK
    "hello world"
 
Flattened echo output, think combined 'mailq' usecase:
hello world
hello world
hello world
 
Did not specify needed input:
dev1.your.com                          : please supply a :msg
dev2.your.com                          : please supply a :msg
dev3.your.com                          : please supply a :msg

Still some work to do, specifically stats needs a rethink in a scenario where you are making many calls such as in this script.

This will be in mcollective version 0.4 hopefully out early January 2010