First the ability to limit what resources are being printed:

# parselocalconfig.rb --limit package
Classes included on this node:
        fqdn
        common::linux
 
Resources managed by puppet on this node:
        package{redhat-lsb: }
                defined in common/modules/puppet/manifests/init.pp:15

You should only see package resources. You can also disable the classes list using --no-classes and on 0.25.x disable the tags list with --no-tags.

I've improved the detection of where to find the yaml file for 0.25 nodes and added an option --config if your config file is not in the usual place.

You can get the latest version here.

The yaml cache has some added features in 0.25 so now I can also show the list of tags on a node, output would be:

# parselocalconfig.rb /var/lib/puppet/client_yaml/catalog/fqdn.yaml
Classes included on this node:
        fqdn
        common::linux
        <snip>
 
Tags for this node:
        fqdn
        common::linux
        <snip>
 
Resources managed by puppet on this node:
        yumrepo{centos-base: }
                defined in common/modules/yum/manifests/init.pp:24
 
        file{/root/.ssh: }
                defined in common/modules/users/manifests/root.pp:20
 
        <snip>

You can get the script that supports both 0.24 and 0.25 here.

The basic pattern is you want to manage a file, but want the contents to be very different from node to node. A fragment based system lets you register different contents into a file on different nodes. It's exactly like the conf.d directory you'd find in Apache but for daemons that does not support this construct on their own.

I've had an older version of this floating around but had to clean it up for a client today so thought I might as well do a proper job, release it and get some more eye balls on it. This version is specific to Puppet 0.25.x, I will soon make a >= 0.24.8 version too since that is what my client is on.

An example says more than words, so lets create something to manage sysctl.conf:

# Initial setup
class sysctl {
   include concat::setup
 
   exec{"reload-sysctl":
      refreshonly => true,
      command => "/sbin/sysctl -p"
   }
 
   concat{"/etc/sysctl.conf":
      notify => Exec["reload-sysctl"],
   }
}
 
# use this to set values
define sysctl::setting($value) {
   concat::fragment{"sysctl_${name}": 
      target => "/etc/sysctl.conf",
      content => "${name} = ${value}\n",
   }
}

The above sets up a class that will create an empty sysctl.conf and provides an utility for setting individual values. Whenever the sysctl.conf file gets changed the changes will be made live using the refreshonly exec.

Lets see how we might use it:

node "your.example.com" {
   include sysctl
 
   sysctl::setting{"net.ipv4.ip_forward":
      value => 1
   }
}

You can see this looks and feels a lot like a native type but without a lot of the hassle it would take to write one, you can really get a lot of mileage out of this pattern. The concat is clever enough to unregister the setting should you remove lines 4 to 6 in the above node block.

A cleaner approach would be to just make classes like sysctl::ipv4_forward that you can include on the nodes that need it.

You can grab the current code here.

This is a long standing bug that doesn't seem to get any love, so I hacked up a little wrapper that works better.

#!/usr/bin/ruby
 
require 'facter'
require 'puppet'
 
Puppet.parse_config
unless $LOAD_PATH.include?(Puppet[:libdir])
    $LOAD_PATH << Puppet[:libdir]
end
 
facts = Facter.to_hash
 
if ARGV.size > 0
    ARGV.each do |f|
        puts "#{f} => #{facts[f]}" if facts.include?(f)
    end
else
    facts.each_pair do |k,v|
        puts("#{k} => #{v}")
    end
end

It behaves by default as if you ran facter -p but you can supply as many fact names as you want on the command line to print just the ones requested.

$ fctr uptime puppetversion processorcount
uptime => 8 days
puppetversion => 0.25.2
processorcount => 1

Using mcollective's discovery features and a suitable agent this is really easy, I've written such an agent and made it available on the mcollective-plugins site.

You can see below a sample session with it. In all of the examples below we're constraining it to hosts with the roles::dev_server puppet class using mcollective discovery. Not shown here is that you can get status as well as use the splay options provided by puppet, see the wiki page for details on that.

First we'll make sure it's enabled.

$ mc-puppetd --with-class roles::dev_server enable
Determining the amount of hosts matching filter for 2 seconds .... 1
 
.
 
Finished processing 1 / 1 hosts in 9.81 ms

Now we'll disable it

$ mc-puppetd --with-class roles::dev_server disable
Determining the amount of hosts matching filter for 2 seconds .... 1
 
.
 
Finished processing 1 / 1 hosts in 3252.13 ms

We'll attempt a runonce, this should fail because we just disabled the agent.

$ mc-puppetd --with-class roles::dev_server runonce -v
Determining the amount of hosts matching filter for 2 seconds .... 1
 
dev1.your.net                      status=false
    Lock file exists                        
 
 
---- puppetd agent stats ----
           Nodes: 1 / 1
      Start Time: Sun Nov 29 23:02:30 +0000 2009
  Discovery Time: 2006.38ms
      Agent Time: 47.62ms
      Total Time: 2054.00ms

Let's enable it and then try to run again.

$ mc-puppetd --with-class roles::dev_server enable
Determining the amount of hosts matching filter for 2 seconds .... 1
 
.
 
Finished processing 1 / 1 hosts in 9.81 ms
 
$ mc-puppetd --with-class roles::dev_server runonce
Determining the amount of hosts matching filter for 2 seconds .... 1
 
.
 
Finished processing 1 / 1 hosts in 2801.82 ms

I think this is a good way to orchestrate these type of maintenance window and I hope someone finds it useful.

require 'find'
 
if File.exists?("/var/spool/ec2/meta-data")
    Find.find("/var/spool/ec2/meta-data") do |path|
        filename = File.basename(path)
        factname = "ec2_#{filename}"
 
        factname.gsub!(/-/, "_")
 
        if File.file?(path)
            lines = File.readlines(path)
 
            if lines.size == 1
                Facter.add(factname) do
                    setcode { lines.first.chomp.to_s }
                end
            else
                lines.each_with_index do |line, i|
                    Facter.add("#{factname}_#{i}") do
                        setcode { lines[i].chomp }
                    end
                end
            end
        end
    end
end
 
if File.exists?("/var/spool/ec2/user-data.raw")
        lines = File.readlines("/var/spool/ec2/user-data.raw")
 
        lines.each do |l|
                if l.chomp =~ /(.+)=(.+)/
                    f = $1; v = $2
 
                    Facter.add(f) do
                        setcode { v }
                    end
                end
        end
end

If you arrange to run /opt/rightscale/bin/ec2.sh in rc.local and pop this fact above into your factdir you should be able to access all the meta data from facter.

# facter -p
ec2_ami_id => ami-73270c07
ec2_ami_launch_index => 0
ec2_ami_manifest_path => pinetecltd-centos-clustera/cluster-webserver-1257783713.manifest.xml
ec2_ancestor_ami_ids_0 => ami-31c72258
ec2_ancestor_ami_ids_1 => ami-ef01e486
ec2_ancestor_ami_ids_2 => ami-0916f360
ec2_ancestor_ami_ids_3 => ami-c8ac48a1
ec2_ancestor_ami_ids_4 => ami-cd52b6a4
ec2_ancestor_ami_ids_5 => ami-19be966d
ec2_ancestor_ami_ids_6 => ami-65200b11
ec2_ancestor_ami_ids_7 => ami-3d200b49
ec2_ancestor_ami_ids_8 => ami-91200be5
ec2_ancestor_ami_ids_9 => ami-81200bf5
ec2_block_device_mapping_ami => sda1
ec2_block_device_mapping_ephemeral0 => sdb
ec2_block_device_mapping_ephemeral1 => sdc
ec2_block_device_mapping_ephemeral2 => sdd
ec2_block_device_mapping_ephemeral3 => sde
ec2_block_device_mapping_root => /dev/sda1
ec2_block_device_mapping_swap => sda3
ec2_hostname => ip-10-227-43-134.eu-west-1.compute.internal
ec2_instance_action => none
ec2_instance_id => i-9411e7e3
ec2_instance_type => m1.small
ec2_kernel_id => aki-7e0d250a
ec2_local_hostname => ip-10-227-43-134.eu-west-1.compute.internal
ec2_local_ipv4 => 10.227.43.134
ec2_placement_availability_zone => eu-west-1b
ec2_public_hostname => ec2-79-125-33-224.eu-west-1.compute.amazonaws.com
ec2_public_ipv4 => 79.125.33.224
ec2_public_keys_0_openssh_key => ssh-rsa AAA
ec2_ramdisk_id => ari-7d0d2509
ec2_reservation_id => r-c655bab1
ec2_security_groups_0 => rip
ec2_security_groups_1 => defaultcluster => a

In addition if you just pass nice key=val pairs in as user data it will add those as facts too, the last above is from that.

The cucumber exploring is of course a result of @auxesis's brilliant cucumber talk at Devops Days recently.

Note: I've updated this from the initial posting, showing how I do filtering with mcollective discovery and put it all into one scenario.

Feature: Update the production systems
 
    Background:
        Given the load balancer has ip address 192.168.1.1
        And I want to update hosts with class roles::dev_server
        And I want to update hosts with fact country=de 
        And I want to pre-discover how many hosts to update
 
    Scenario: Update the website
        When I block the load balancer
        Then traffic from the load balancer should be blocked
 
        When I update the package mywebapp
        Then the package version for mywebapp should be 4.2.6-3.el5
 
        When I unblock the load balancer
        Then traffic from the load balancer should be unblocked

This is completely like any other test driven scenario based system, if it fails to block the firewall deploy will bail out. If it fails to update the package it will bail and finally only if those worked will it unblock the firewall.

Thanks to mcollective this is distributed and parallel over large numbers of machines. I can also apply filters to update just certain clusters using mcollective's discovery features.

Everything's outcome is tested and cucumber will only show the all clear when everything worked on all machines in a consistent way.

This is made possible in part because the mcollective plugins use the Puppet providers underneath the hood, so package and service actions are complete idempotent and repeatable, I can rerun this script 100 times and it will do the same thing.

I have other steps not included here to keep things simple but in a real world I would restart the webserver after the update and I would then call NRPE plugins on all the nodes to make sure their load average is in acceptable ranges before the firewall gets opened letting the load balancer in.

This opens up a whole lot of interesting ideas, kudos to @auxesis and his great talk at devopsdays!

This is of course a problem that Puppet solves with it's providers system, after some chatting with Luke on IRC I now have a pretty nice solution.

Assuming you don't mind shell scripting in Ruby here's a pretty decent bit of Ruby to manage a service on many different environments.

require 'puppet'
 
service = ARGV.shift
action = ARGV.shift
 
ARGV.length > 0 ? hasstatus = true : hasstatus = false
 
begin
    svc = Puppet::Type.type(:service).new(:name => service, 
        :hasstatus => hasstatus).provider
 
    svc.send action
 
    puts("#{service} #{action} status: #{svc.status}")
rescue Exception => e
    puts("Could not #{action} service #{service}: #{e}")
end

# service.rb httpd stop true
httpd stop status: stopped
 
# service.rb httpd start true
httpd start status: running

You'd probably want to put in support for the pattern parameter to keep certain broken Operating Systems happy, but this is pretty nice and platform independent.

The concept is that I use publish / subscribe middleware - ActiveMQ with Stomp in my case - to do one-off administration. Unlike using Capistrano or some of those tools I do not need lists of machines or visit each machine with a request because the network supports discovery and a single message to the middleware results in 10s or 100s or 1000s of machines getting the message.

This means any tasks I ask to be done happens in parallel on any number of machines typically I see ~100 machines finish the task in the same time as 1 machine would and no need for SSH or anything like that.

The app server and client libs I wrote take away all the complexities of the middleware and takes care of crypto signing requests, only responding to requests that has been signed properly etc, serializing and deserialization of data etc.

Discovery is built in and it supports puppet classes and facts and a few other criteria I use for my own systems so there is no need to build any kind of system that keeps track of what machines I have with what version of operating system etc. As long as is on the middleware I can find it.

The bulk - timeout handling and so forth removed - of the ping app you see in the demo can be seen here, client:

client = Stomphost::Client.new(config)
client.sendreq(Time.now.to_f, "echo")
 
loop do
    resp = client.receive 
    elapsed = (Time.now.to_f - resp[:body]) * 1000
end

And the agent is just this:

module Stomphost
    module Agent
        class Echo
            def handlemsg(msg)
                msg[:body]
            end
        end
    end
end

You can see that even data types like the float will flow cleanly through end to end.

Watch the video, I mention my uses cases but it includes distributed Exim administration, package updates, services restarts, iptables management and much more.

UPDATE: This code has now been released as an Open Source Apache 2 licenced project at mcollective.googlecode.com

I'll present here a simple layout that is both a powerful tool for isolating code changes while developing combined with minimizing the headaches with maintaining multiple branches of the same module for an extended period of time.

As with the other posts I did on puppet recently the focus here is simplicity, this will work for small sites and will hopefully give you enough inspiration to successfully model larger sites using these techniques.

To really use environments well I strongly suggest you use modules, I've blogged about them before, if you read this post and think "hey! that's great" and don't yet use modules, go fix that first.

The puppetmaster tries to find modules using it's modulepath setting, typically something like /etc/puppet/modules.  You usually just set this value once in your puppet.conf and that's it, all done. Environments expand on this and give you the ability to set different settings for different environments, below a simple example.

[puppetmasterd]
   modulepath = /etc/puppet/manifests/modules
 
[development]
   modulepath = /etc/puppet/manifests/development/modules

This is your basic multi environment setup, if you run a client with the development environment set modules will be served from an alternative path.

This is already a huge win, if you use something like below:

class ntp {
   file{"/etc/ntpd.conf":
      source => ["puppet:///ntp/ntpd.conf.${environment}", "puppet:///ntp/ntpd.conf"]
   }
}
 
node dev1 {
   include ntp
}

If you put the ntp module in both production and development, the right one will be chosen depending on the nodes environment setting. The file ntp.conf will also be served from the right modules files directory. You could now safely make changes to the development one without affecting the rest of the machines.

The major drawback here is that if you have say 30 modules, you need to duplicate the lot into development if you wish to use them there, this is not great at all as it quickly becomes a maintenance nightmare. If you made a change to ntp module in development, you need to remember to go and do the same change in all your other environments. In cases where you give each sysadmin or developer their own environment this rapidly becomes unmaintainable.

The modulepath option has a trick up it's sleeve though, you can specify several search paths just like the normal Unix PATH environment variable, we can rework our config like this.

[puppetmasterd]
   modulepath = /etc/puppet/manifests/common/modules
 
[development]
   modulepath = /etc/.../development/modules:/etc/.../common/modules
 
[production]
   modulepath = /etc/.../production/modules:/etc/.../common/modules
 
[johndoe]
   modulepath = /etc/.../johndoe/modules:/etc/.../common/modules

Here we've added a few more environments, even one for a developer called John Doe. The big thing we did though is create /etc/puppet/manifests/common/modules as a 2nd search path.

production
`-- modules
development
`-- modules
common
`-- modules
    `-- ntp
        `-- manifests
            `-- init.pp

The idea here is that you'd strive to keep all your modules in the common directory as their stable location but if you wished to do development on your ntp module that you wish to test, you just copy the ntp module into either development or johndoe directories, with that done only machines in those environments will get the new code.

This works especially well if you use version control, I use SVN and treat common as my trunk, whenever I need to work on a module I simply branch it into development, do my work there and when ready to release I merge my changes back to common and get rid of the branched copy so all environments gets the same version.

With this you can easily come up with a simple release testing strategy, do the weeks code in development and test locally, then put the changes you wish to release to production into your QA environment and do tests. Finally when you're ready merge all the changes into common and just get rid of the development and QA copies, all environments are now running the same code ready for the next cycle.

This gives you the ability to create branches, do testing, isolate developers from each other - just put their desktops in individual environments - and lots of other nifty things but in the end you have only one copy of the code for your stable released versions.

I hope you find this useful, I'll reiterate that this is a simple setup, you should use this information and come up with a solution that will work well for your use case and teams.