www.devco.net by r.i.pienaar

3Mar/100

Puppet localconfig parser – 20100303

I've had some good feedback on my previous post about the puppet localconfig parser, have implemented the requested features so here's a new version.

First the ability to limit what resources are being printed:

# parselocalconfig.rb --limit package
Classes included on this node:
        fqdn
        common::linux
 
Resources managed by puppet on this node:
        package{redhat-lsb: }
                defined in common/modules/puppet/manifests/init.pp:15

You should only see package resources. You can also disable the classes list using --no-classes and on 0.25.x disable the tags list with --no-tags.

I've improved the detection of where to find the yaml file for 0.25 nodes and added an option --config if your config file is not in the usual place.

You can get the latest version here.

Tagged as: , , No Comments
19Feb/1018

Building files from fragments with Puppet

While building up complex configs with Puppet you often need to build up one file from many fragments. This is useful for files like older sysctl.conf files and maybe named.conf files.

The basic pattern is you want to manage a file, but want the contents to be very different from node to node. A fragment based system lets you register different contents into a file on different nodes. It's exactly like the conf.d directory you'd find in Apache but for daemons that does not support this construct on their own.

I've had an older version of this floating around but had to clean it up for a client today so thought I might as well do a proper job, release it and get some more eye balls on it. This version is specific to Puppet 0.25.x, I will soon make a >= 0.24.8 version too since that is what my client is on.

An example says more than words, so lets create something to manage sysctl.conf:

# Initial setup
class sysctl {
   include concat::setup
 
   exec{"reload-sysctl":
      refreshonly => true,
      command => "/sbin/sysctl -p"
   }
 
   concat{"/etc/sysctl.conf":
      notify => Exec["reload-sysctl"],
   }
}
 
# use this to set values
define sysctl::setting($value) {
   concat::fragment{"sysctl_${name}": 
      target => "/etc/sysctl.conf",
      content => "${name} = ${value}\n",
   }
}

The above sets up a class that will create an empty sysctl.conf and provides an utility for setting individual values. Whenever the sysctl.conf file gets changed the changes will be made live using the refreshonly exec.

Lets see how we might use it:

node "your.example.com" {
   include sysctl
 
   sysctl::setting{"net.ipv4.ip_forward":
      value => 1
   }
}

You can see this looks and feels a lot like a native type but without a lot of the hassle it would take to write one, you can really get a lot of mileage out of this pattern. The concat is clever enough to unregister the setting should you remove lines 4 to 6 in the above node block.

A cleaner approach would be to just make classes like sysctl::ipv4_forward that you can include on the nodes that need it.

You can grab the current code here.

Tagged as: , , 18 Comments
18Feb/101

Custom deployer using MCollective

One of the goals of building the SimpleRPC framework and the overall speed of MCollective is to create interactive tools to manage your infrastructure in a way that it all just seems like a single point of entry with one machine. I've blogged a bit about this before with how I manage Exim clusters.

I've recently built a deployer for a client that does some very specific things with their FastCGI, packages and monitoring in a way that is safe for developers to use. I've made a sanitized demo of it that you can see below. It's sanitized in that the hostnames are replaced with hashes and some monitoring details removed but you'll get the idea.

As usual it's best to just look at the video on youtube in it's HD mode.

17Feb/100

Few Rubyisms

While looking at some bits of other peoples Ruby code I came across a few shortcuts and interesting structures worth mentioning.

Exception handling shortcut

First up a shortcut to catch exceptions thrown by a method:

def say_foo
   puts "foo" if doit
rescue Exception
   puts "#fail"
end

So since we didn't define doit this will raise an exception, which will be handled. Nice shortcut to avoid an extra inner begin / rescue block.

sprintf equivelant

Ruby supports sprintf style string building in a handy little shortcut:

puts "%2.6f\n%d" % [1, 1]

This produces:

$ ruby test.rb
1.000000
1

Get a value from a hash with default for non existing

This is really nice, I've written way too many constructs like this:

foo.include?(:bar) ? bar = foo[:bar] : bar = "unknown"

One option that I was told about was this:

bar = foo[:bar] || "unknown"

But that does not work if you had false in the hash, or maybe even nil.

Turns out there's an awesome shortcut for this:

bar = foo.fetch(:bar, "unknown")

Reloading a class

Sometimes you want to reload a class you previously loaded with require. I have the need in my plugin manager for mcollective. There's a simple fix by simply using Kernel#load to load the .rb file, each time you load it the file will be reloaded from disk.

irb(main):001:0> load "test.rb"
=> true
irb(main):002:0> Foo.doit
foo
irb(main):003:0* load "test.rb"
=> true
irb(main):004:0> Foo.doit
foo foo

In between lines 2 and 3 I edited the file test.rb and just reloaded it, the changes on disk reflected in the current session. The main difference is that you need to supply the full file name and not just test like you would with require.

Tagged as: , , No Comments
24Jan/100

MCollective 0.4.3 Auditing

I just released version 0.4.3 of mcollective which brings a new auditing capability to SimpleRPC. Using the auditing system you can log to a file on each host every request or build a centralized auditing system for all requests on all nodes.

We ship a simple plugin that logs to the local harddrive but there is also a community plugin that creates a centralized logging system running over MCollective as a transport.

This is the kind of log the centralized logger will produce:

01/24/10 18:24:20 dev1.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: 01/24/10 18:24:20 caller=uid=500@ids1.my.net agent=iptables action=block
01/24/10 18:24:20 dev1.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: {:ipaddr=>"114.255.136.120"}
01/24/10 18:24:20 dev2.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: 01/24/10 18:24:20 caller=uid=500@ids1.my.net agent=iptables action=block
01/24/10 18:24:20 dev2.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: {:ipaddr=>"114.255.136.120"}
01/24/10 18:24:20 dev3.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: 01/24/10 18:24:20 caller=uid=500@ids1.my.net agent=iptables action=block
01/24/10 18:24:20 dev3.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: {:ipaddr=>"114.255.136.120"}

Here we see 3 nodes that got a request to add 114.255.136.120 to their local firewall. The request was sent by UID 500 on the machine ids1.my.net. The request is of course the same everywhere so the request id is the same on every node, the log shows agent and all parameters passed.

14Jan/100

Better way to query facts

Facter has some annoying bug where it won't always print all facts when called like facter fact, ones that require dynamic lookups etc just won't print.

This is a long standing bug that doesn't seem to get any love, so I hacked up a little wrapper that works better.

#!/usr/bin/ruby
 
require 'facter'
require 'puppet'
 
Puppet.parse_config
unless $LOAD_PATH.include?(Puppet[:libdir])
    $LOAD_PATH << Puppet[:libdir]
end
 
facts = Facter.to_hash
 
if ARGV.size > 0
    ARGV.each do |f|
        puts "#{f} => #{facts[f]}" if facts.include?(f)
    end
else
    facts.each_pair do |k,v|
        puts("#{k} => #{v}")
    end
end

It behaves by default as if you ran facter -p but you can supply as many fact names as you want on the command line to print just the ones requested.

$ fctr uptime puppetversion processorcount
uptime => 8 days
puppetversion => 0.25.2
processorcount => 1
Tagged as: , , No Comments
13Jan/100

MCollective 0.4.2 released

Just a quick blog post for those who follow me here to get notified about new releases of MCollective. I just released version 0.4.2 which brings in big improvements for Debian packages, some tweaks to command line and a bug fix in SimpleRPC.

Read all about it at the Release Notes

13Jan/102

Backing up Google Code projects

Google Code does not provide it's own usable export methods for projects so we need to make do on our own, there seems no sane way to back up the tickets but for SVN which includes the wiki you can use svnsync.

Here's a little script to automate this, just give it a PREFIX of your choice and a list of projects in the PROJECTS variable, cron it and it will happily keep your repos in sync.

It outputs its actions to STDOUT so you should add some redirect or redirect it from cron.

#!/bin/bash
 
PROJECTS=( your project list )
PREFIX="/path/to/backups"
 
[ -d ${PREFIX} ] || mkdir -p ${PREFIX}
 
cd ${PREFIX}
 
for prj in ${PROJECTS[@]}
do
    if [ ! -d ${PREFIX}/${prj}/conf ]; then
        svnadmin create ${prj}
 
        ln -s /bin/true ${PREFIX}/${prj}/hooks/pre-revprop-change
 
        svnsync init file:///${PREFIX}/${prj} http://${prj}.googlecode.com/svn
    fi
 
    svnsync sync file:///${PREFIX}/${prj}
done

I stongly suggest you backup even your cloud hosted data.

22Dec/090

MCollective Simple RPC

MCollective is a framework for writing RPC style tools that talk to a cloud of servers, till now doing that has been surprisingly hard for non ruby coders. The reason for this is that I was focussing on getting the framework built and feeling my way around the use cases.

I've now spent 2 days working on simplifying actually writing agents and consumers. This code is not released yet - just in SVN trunk - but here's a taster.

First writing an agent should be simple, here's a simple 'echo' server that takes a message as input and returns it back.

class Rpctest<RPC::Agent
    # Basic echo server
    def echo_action(request, reply)
         raise MissingRPCData, "please supply a :msg" unless request.include?(:msg)
 
         reply.data = request[:msg]
    end
end

This creates an echo action, does a quick check that a message was received and sends it back. I want to create a few more validators so you can check easily if the data passed to you is sane and secure if you're doing anything like system() calls with it.

Here's the client code that calls the echo server 4 times:

#!/usr/bin/ruby
 
require 'mcollective'
 
include MCollective::RPC
 
rpctest = rpcclient("rpctest")
 
puts "Normal echo output, non verbose, shouldn't produce any output:"
printrpc rpctest.echo(:msg => "hello world")
 
puts "Flattened echo output, think combined 'mailq' usecase:"
printrpc rpctest.echo(:msg => "hello world"), :flatten => true
 
puts "Forced verbose output, if you always want to see every result"
printrpc rpctest.echo(:msg => "hello world"), :verbose => true
 
puts "Did not specify needed input:"
printrpc rpctest.echo

This client supports full discovery and all the usual stuff, has pretty --help output and everything else you'd expect in the clients I've supplied with the core mcollective. It caches discovery results so above code will do one discovery only and reuse it for the other calls to the collective.

When running you'll see a twirling status indicator, something like:

  - [5 / 10]

This will give you a nice non scrolling indicator of progress and should work well for 100s of machines without spamming you with noise, at the end of the run you'll get the output.

The printrpc helper function tries its best to print output for you in a way that makes sense on large amounts of machines.

  • By default it doesn't print stuff that succeeds, you do get a overall progress indicator though
  • If anything does go wrong, useful information gets printed but only for hosts that had problems
  • If you ran the client with --verbose, or forced it to verbose mode output you'll get a full bit of info of the result from every server.
  • It supports flags to modify the output, you can flatten the output so hostnames etc aren't showed, just a concat of the data.

The script above gives the following output when run in non-verbose mode:

$ rpctest.rb --with-class /devel/
 
Normal echo output, non verbose, shouldn't produce any output:
 
Forced verbose output, if you always want to see every result:
dev1.your.com                          : OK
    "hello world"
 
dev2.your.com                          : OK
    "hello world"
 
dev3.your.com                          : OK
    "hello world"
 
Flattened echo output, think combined 'mailq' usecase:
hello world
hello world
hello world
 
Did not specify needed input:
dev1.your.com                          : please supply a :msg
dev2.your.com                          : please supply a :msg
dev3.your.com                          : please supply a :msg

Still some work to do, specifically stats needs a rethink in a scenario where you are making many calls such as in this script.

This will be in mcollective version 0.4 hopefully out early January 2010

1Dec/090

MCollective Release 0.2.0

I am pleased to announce the the first actual numbered release of The Marionette Collective, you can grab it from the downloads page.

Till now people wanting to test this had to pull out of SVN directly, I put off doing a release till I had most of the major tick boxes in my mind ticked and till I knew I wouldn't be making any major changes to the various plugins and such. This release is 0.2.x since 0.1.x was the release number I used locally for my own testing.

This being the first release I fully anticipate some problems and weirdness, please send any concerns to the mailing list or ticketing system.

This has been a while coming, I've posted lots on this blog about mcollective, what it is and what it does. For those just joining you want to watch the video on this post for some background.

I am keen to get feedback from some testers, specifically keen to hear thoughts around these points:

  • How does the client tools behave on 100s of nodes, I suspect the output format might be useless if it just scrolls and scrolls, I have some ideas about this but need feedback.
  • On large amount of hosts, or when doing lots of requests soon after each other, do you notice any replies going missing.
  • Feed back about the general design principals, especially how you find the plugin system and what else you might want pluggable. I for example want to make it much easier to add new discovery methods.
  • Anything else you can think of

I'll be putting in tickets on the issue system for future features / fixes I am adding so you can track there to get a feel for the milestones toward 0.3.x.

Thanks goes to the countless people who I spoke to in person, on IRC and on Twitter, thanks for all the retweets and general good wishes. Special thanks also to Chris Read who made the debian package code and fixed up the RC script to be LSB compliant.