For background of what this is about please see my earlier post: Building files from fragments with Puppet

New in this release

Paul Elliot sent in most of the patches that enabled this release, lots of thanks Paul!

• 0.24.8 and newer is supported
• You can now prepend warnings to generated files as a shell style comment using the warn property
• You can enable the ability to create empty concat files using the force property
• You can configure the location of your sort binary in setup.pp

The code should auto configure for 0.24.8 use, if it does not work please see setup.pp.

You can grab the code here.

Known issues

As with my earlier attempts at making a concat tool for 0.24.x this version when used on 0.24 will raise some false notifies. Basically the method we use to clear the concat store of unmanaged files has a side effect and on the next run you will get an unneeded notify. Puppets behavior has improved in 0.25 so it works as expected there, for 0.24 though there is no known work around.

You cannot change the owner of a file, I know how to work around this issue and will have something in the next release.

First the ability to limit what resources are being printed:

# parselocalconfig.rb --limit package
Classes included on this node:
        fqdn
        common::linux
 
Resources managed by puppet on this node:
        package{redhat-lsb: }
                defined in common/modules/puppet/manifests/init.pp:15

You should only see package resources. You can also disable the classes list using --no-classes and on 0.25.x disable the tags list with --no-tags.

I've improved the detection of where to find the yaml file for 0.25 nodes and added an option --config if your config file is not in the usual place.

You can get the latest version here.

The yaml cache has some added features in 0.25 so now I can also show the list of tags on a node, output would be:

# parselocalconfig.rb /var/lib/puppet/client_yaml/catalog/fqdn.yaml
Classes included on this node:
        fqdn
        common::linux
        <snip>
 
Tags for this node:
        fqdn
        common::linux
        <snip>
 
Resources managed by puppet on this node:
        yumrepo{centos-base: }
                defined in common/modules/yum/manifests/init.pp:24
 
        file{/root/.ssh: }
                defined in common/modules/users/manifests/root.pp:20
 
        <snip>

You can get the script that supports both 0.24 and 0.25 here.

In the mean time The Guardian has offered us a regular venue at their big auditorium where the London Scale Camp was held. This is an excellent venue setup with projectors and everything. We could get it every month but I figured we'll keep the talks bi-monthly and do drinks night or just something social on the other nights. There are several pubs around the Guardian so we'll do drinks after the 1 or 2 talks a night.

The upcoming dates - excluding the pure pub nights are then:

02/03/2010 7pm – Agile Systems Administration at London Geek Nights – thanks to Thoughtworks
28/04/2010 7pm – Topic TBA – thanks to The Guardian for the venue
30/06/2010 7pm – Topic TBA – thanks to The Guardian for the venue
27/10/2010 6pm – Topic TBA – thanks to The Guardian for the venue
29/12/2010 6pm – Topic TBA – thanks to The Guardian for the venue

Put them in your calendars and again massive thanks to The Guardian for sponsoring us.

The basic pattern is you want to manage a file, but want the contents to be very different from node to node. A fragment based system lets you register different contents into a file on different nodes. It's exactly like the conf.d directory you'd find in Apache but for daemons that does not support this construct on their own.

I've had an older version of this floating around but had to clean it up for a client today so thought I might as well do a proper job, release it and get some more eye balls on it. This version is specific to Puppet 0.25.x, I will soon make a >= 0.24.8 version too since that is what my client is on.

An example says more than words, so lets create something to manage sysctl.conf:

# Initial setup
class sysctl {
   include concat::setup
 
   exec{"reload-sysctl":
      refreshonly => true,
      command => "/sbin/sysctl -p"
   }
 
   concat{"/etc/sysctl.conf":
      notify => Exec["reload-sysctl"],
   }
}
 
# use this to set values
define sysctl::setting($value) {
   concat::fragment{"sysctl_${name}": 
      target => "/etc/sysctl.conf",
      content => "${name} = ${value}\n",
   }
}

The above sets up a class that will create an empty sysctl.conf and provides an utility for setting individual values. Whenever the sysctl.conf file gets changed the changes will be made live using the refreshonly exec.

Lets see how we might use it:

node "your.example.com" {
   include sysctl
 
   sysctl::setting{"net.ipv4.ip_forward":
      value => 1
   }
}

You can see this looks and feels a lot like a native type but without a lot of the hassle it would take to write one, you can really get a lot of mileage out of this pattern. The concat is clever enough to unregister the setting should you remove lines 4 to 6 in the above node block.

A cleaner approach would be to just make classes like sysctl::ipv4_forward that you can include on the nodes that need it.

You can grab the current code here.

I've recently built a deployer for a client that does some very specific things with their FastCGI, packages and monitoring in a way that is safe for developers to use. I've made a sanitized demo of it that you can see below. It's sanitized in that the hostnames are replaced with hashes and some monitoring details removed but you'll get the idea.

As usual it's best to just look at the video on youtube in it's HD mode.

Exception handling shortcut

First up a shortcut to catch exceptions thrown by a method:

def say_foo
   puts "foo" if doit
rescue Exception
   puts "#fail"
end

So since we didn't define doit this will raise an exception, which will be handled. Nice shortcut to avoid an extra inner begin / rescue block.

sprintf equivelant

Ruby supports sprintf style string building in a handy little shortcut:

puts "%2.6f\n%d" % [1, 1]

This produces:

$ ruby test.rb
1.000000
1

Get a value from a hash with default for non existing

This is really nice, I've written way too many constructs like this:

foo.include?(:bar) ? bar = foo[:bar] : bar = "unknown"

One option that I was told about was this:

bar = foo[:bar] || "unknown"

But that does not work if you had false in the hash, or maybe even nil.

Turns out there's an awesome shortcut for this:

bar = foo.fetch(:bar, "unknown")

Reloading a class

Sometimes you want to reload a class you previously loaded with require. I have the need in my plugin manager for mcollective. There's a simple fix by simply using Kernel#load to load the .rb file, each time you load it the file will be reloaded from disk.

irb(main):001:0> load "test.rb"
=> true
irb(main):002:0> Foo.doit
foo
irb(main):003:0* load "test.rb"
=> true
irb(main):004:0> Foo.doit
foo foo

In between lines 2 and 3 I edited the file test.rb and just reloaded it, the changes on disk reflected in the current session. The main difference is that you need to supply the full file name and not just test like you would with require.

The thinking was that we wanted to be sure we will do the meetings roughly monthly rather than start something with a lot of noise and then fizzle out.

We've met 4 months in a row and we figured it's about time to throw open the doors. We are therefore meeting on the 24th of February 2010 at The Priory Bar, we'll aim to be there from 6 or 7pm on wards.

We're trying to create a community of like minded people in London - Sysadmins, Ops People, Developers, Puppet Users, Chef Users we do not discriminate. As long as your interested in Agile Infrastructures or in bridging the gap between the Dev and Ops Silos or just want to hang out with a bunch of Systems guys who aren't stuck in the 1980s you're welcome to join us. The basic format we're aiming for is to meet in a pub/restaurant for Lunch or Dinner one month and every second month to try and arrange some meeting with talks.

Our first actual meeting with talks will be in March at a venue provided by Thought Works so the meeting on the 24th is to discuss what we'll be up to there. We'll also be very glad to hear from any companies who would like to offer us some space on a bi-monthly basis for our talks.

While on the subject I should mention an event that is coming up. In March London hosts Q-Con with a track specifically for DevOps and I will be there doing a talk as well as some of our other regular members.

Please help spread the word by tweeting with the #ldndevops hashtag. We've created a site at http://londondevops.org/ that aggregates some of the blogs of people who we already know about in London who are operating in this space. Please contact me here in comments or on Twitter @ripienaar to get yourself added.

The problem I have is I want a plugin to be able to either make a method using the normal Ruby def foo or via some DSL'ish helpers.

class Foo<Base
   register_action(:name => "do_something", :description => "foo")
 
   def do_something_action
   end
 
   register_action(:name => "do_something_else", :description => "foo") do
      # body of the action here
   end
end

The above code should make me two methods - do_something_action and do_something_else_action - they should be identical to viewers from the outside. Here's the base class that makes this happen correctly:

class Base
   def self.register_input(input, &block)
      name = input[:name]
 
      self.module_eval { define_method("#{name}_action", &block) } if block_given?
   end
end

It's pretty simple, we're just using define_method in the scope of the module and that does the rest.

$ mc-rpc service status service=httpd

This is all fine and well and easy enough, however it requires you to know a lot. You need to know there's a status action and you need to know it expects a service argument, not great.

I'm busy adding the ability for an agent to register its metadata and interface so that 3rd party tools can dynamically generate useful interfaces.

A sample registration for service agent is:

register_meta(:name        => "SimpleRPC Service Agent",
              :description => "Agent to manage services using the Puppet service provider",
              :author      => "R.I.Pienaar",
              :license     => "GPLv2",
              :version     => 1.1,
              :url         => "http://mcollective-plugins.googlecode.com/",
              :timeout     => 60)
 
["start", "stop", "restart", "status"].each do |action|
    register_input(:action      => action,
                   :name        => "service",
                   :prompt      => "Service Name",
                   :description => "The service to #{action}",
                   :type        => :string,
                   :validation  => '^[a-zA-Z\-_\d]+$',
                   :maxlength   => 30):

This includes all the meta data, versions, timeouts, validation of inputs, prompts and help text for every input argument.

Using this we can now generate dynamic UI's, and do something like JavaDoc generated documentation. I've recorded a little video demonstrating a proof of concept Text UI that uses this data to generate a UI dynamically. This is ripe for integration into tools like Foreman and Puppet Dashboard.

Please watch the video here, best viewed full screen.