www.devco.net by r.i.pienaar

12Mar/100

Puppet Concat 20100312

I am pleased to announce the next version of my Puppet Concat script, we now have 0.24.8 and newer support and a few smaller bits mentioned below.

For background of what this is about please see my earlier post: Building files from fragments with Puppet

New in this release

Paul Elliot sent in most of the patches that enabled this release, lots of thanks Paul!

  • 0.24.8 and newer is supported
  • You can now prepend warnings to generated files as a shell style comment using the warn property
  • You can enable the ability to create empty concat files using the force property
  • You can configure the location of your sort binary in setup.pp

The code should auto configure for 0.24.8 use, if it does not work please see setup.pp.

You can grab the code here.

Known issues

As with my earlier attempts at making a concat tool for 0.24.x this version when used on 0.24 will raise some false notifies. Basically the method we use to clear the concat store of unmanaged files has a side effect and on the next run you will get an unneeded notify. Puppets behavior has improved in 0.25 so it works as expected there, for 0.24 though there is no known work around.

You cannot change the owner of a file, I know how to work around this issue and will have something in the next release.

Tagged as: , No Comments
3Mar/100

Puppet localconfig parser – 20100303

I've had some good feedback on my previous post about the puppet localconfig parser, have implemented the requested features so here's a new version.

First the ability to limit what resources are being printed:

# parselocalconfig.rb --limit package
Classes included on this node:
        fqdn
        common::linux
 
Resources managed by puppet on this node:
        package{redhat-lsb: }
                defined in common/modules/puppet/manifests/init.pp:15

You should only see package resources. You can also disable the classes list using --no-classes and on 0.25.x disable the tags list with --no-tags.

I've improved the detection of where to find the yaml file for 0.25 nodes and added an option --config if your config file is not in the usual place.

You can get the latest version here.

Tagged as: , , No Comments
26Feb/104

What does Puppet manage on a node?

Last year I wrote a tool to parse the localconfig.yaml from Puppet 0.24 and display a list of resources and classes. This script failed when 0.25 came out, I've updated it for 0.25 support.

The yaml cache has some added features in 0.25 so now I can also show the list of tags on a node, output would be:

# parselocalconfig.rb /var/lib/puppet/client_yaml/catalog/fqdn.yaml
Classes included on this node:
        fqdn
        common::linux
        <snip>
 
Tags for this node:
        fqdn
        common::linux
        <snip>
 
Resources managed by puppet on this node:
        yumrepo{centos-base: }
                defined in common/modules/yum/manifests/init.pp:24
 
        file{/root/.ssh: }
                defined in common/modules/users/manifests/root.pp:20
 
        <snip>

You can get the script that supports both 0.24 and 0.25 here.

Tagged as: , , 4 Comments
19Feb/1018

Building files from fragments with Puppet

While building up complex configs with Puppet you often need to build up one file from many fragments. This is useful for files like older sysctl.conf files and maybe named.conf files.

The basic pattern is you want to manage a file, but want the contents to be very different from node to node. A fragment based system lets you register different contents into a file on different nodes. It's exactly like the conf.d directory you'd find in Apache but for daemons that does not support this construct on their own.

I've had an older version of this floating around but had to clean it up for a client today so thought I might as well do a proper job, release it and get some more eye balls on it. This version is specific to Puppet 0.25.x, I will soon make a >= 0.24.8 version too since that is what my client is on.

An example says more than words, so lets create something to manage sysctl.conf:

# Initial setup
class sysctl {
   include concat::setup
 
   exec{"reload-sysctl":
      refreshonly => true,
      command => "/sbin/sysctl -p"
   }
 
   concat{"/etc/sysctl.conf":
      notify => Exec["reload-sysctl"],
   }
}
 
# use this to set values
define sysctl::setting($value) {
   concat::fragment{"sysctl_${name}": 
      target => "/etc/sysctl.conf",
      content => "${name} = ${value}\n",
   }
}

The above sets up a class that will create an empty sysctl.conf and provides an utility for setting individual values. Whenever the sysctl.conf file gets changed the changes will be made live using the refreshonly exec.

Lets see how we might use it:

node "your.example.com" {
   include sysctl
 
   sysctl::setting{"net.ipv4.ip_forward":
      value => 1
   }
}

You can see this looks and feels a lot like a native type but without a lot of the hassle it would take to write one, you can really get a lot of mileage out of this pattern. The concat is clever enough to unregister the setting should you remove lines 4 to 6 in the above node block.

A cleaner approach would be to just make classes like sysctl::ipv4_forward that you can include on the nodes that need it.

You can grab the current code here.

Tagged as: , , 18 Comments
14Jan/100

Better way to query facts

Facter has some annoying bug where it won't always print all facts when called like facter fact, ones that require dynamic lookups etc just won't print.

This is a long standing bug that doesn't seem to get any love, so I hacked up a little wrapper that works better.

#!/usr/bin/ruby
 
require 'facter'
require 'puppet'
 
Puppet.parse_config
unless $LOAD_PATH.include?(Puppet[:libdir])
    $LOAD_PATH << Puppet[:libdir]
end
 
facts = Facter.to_hash
 
if ARGV.size > 0
    ARGV.each do |f|
        puts "#{f} => #{facts[f]}" if facts.include?(f)
    end
else
    facts.each_pair do |k,v|
        puts("#{k} => #{v}")
    end
end

It behaves by default as if you ran facter -p but you can supply as many fact names as you want on the command line to print just the ones requested.

$ fctr uptime puppetversion processorcount
uptime => 8 days
puppetversion => 0.25.2
processorcount => 1
Tagged as: , , No Comments
30Nov/090

Managing puppetd with mcollective

It's typical during maintenance windows that you would want to disable puppet, do your work, enable again and do a run. Or perhaps you don't run puppet all the time, you just want to kick it off during your maintenance window. Doing this with ssh for loops is slow and annoying, here's a way to target large sums of machines for these actions using mcollective.

Using mcollective's discovery features and a suitable agent this is really easy, I've written such an agent and made it available on the mcollective-plugins site.

You can see below a sample session with it. In all of the examples below we're constraining it to hosts with the roles::dev_server puppet class using mcollective discovery. Not shown here is that you can get status as well as use the splay options provided by puppet, see the wiki page for details on that.

First we'll make sure it's enabled.

$ mc-puppetd --with-class roles::dev_server enable
Determining the amount of hosts matching filter for 2 seconds .... 1
 
.
 
Finished processing 1 / 1 hosts in 9.81 ms

Now we'll disable it

$ mc-puppetd --with-class roles::dev_server disable
Determining the amount of hosts matching filter for 2 seconds .... 1
 
.
 
Finished processing 1 / 1 hosts in 3252.13 ms

We'll attempt a runonce, this should fail because we just disabled the agent.

$ mc-puppetd --with-class roles::dev_server runonce -v
Determining the amount of hosts matching filter for 2 seconds .... 1
 
dev1.your.net                      status=false
    Lock file exists                        
 
 
---- puppetd agent stats ----
           Nodes: 1 / 1
      Start Time: Sun Nov 29 23:02:30 +0000 2009
  Discovery Time: 2006.38ms
      Agent Time: 47.62ms
      Total Time: 2054.00ms

Let's enable it and then try to run again.

$ mc-puppetd --with-class roles::dev_server enable
Determining the amount of hosts matching filter for 2 seconds .... 1
 
.
 
Finished processing 1 / 1 hosts in 9.81 ms
 
$ mc-puppetd --with-class roles::dev_server runonce
Determining the amount of hosts matching filter for 2 seconds .... 1
 
.
 
Finished processing 1 / 1 hosts in 2801.82 ms

I think this is a good way to orchestrate these type of maintenance window and I hope someone finds it useful.

9Nov/097

RightScale facts

I'm trying to build up a nice demo of mcollective and trying to save some effort by using the RightScale CentOS AMI's.  I noticed they came with a nice script to pull down the user data and meta data so figured I might as well make some facts.

require 'find'
 
if File.exists?("/var/spool/ec2/meta-data")
    Find.find("/var/spool/ec2/meta-data") do |path|
        filename = File.basename(path)
        factname = "ec2_#{filename}"
 
        factname.gsub!(/-/, "_")
 
        if File.file?(path)
            lines = File.readlines(path)
 
            if lines.size == 1
                Facter.add(factname) do
                    setcode { lines.first.chomp.to_s }
                end
            else
                lines.each_with_index do |line, i|
                    Facter.add("#{factname}_#{i}") do
                        setcode { lines[i].chomp }
                    end
                end
            end
        end
    end
end
 
if File.exists?("/var/spool/ec2/user-data.raw")
        lines = File.readlines("/var/spool/ec2/user-data.raw")
 
        lines.each do |l|
                if l.chomp =~ /(.+)=(.+)/
                    f = $1; v = $2
 
                    Facter.add(f) do
                        setcode { v }
                    end
                end
        end
end

If you arrange to run /opt/rightscale/bin/ec2.sh in rc.local and pop this fact above into your factdir you should be able to access all the meta data from facter.

# facter -p
ec2_ami_id => ami-73270c07
ec2_ami_launch_index => 0
ec2_ami_manifest_path => pinetecltd-centos-clustera/cluster-webserver-1257783713.manifest.xml
ec2_ancestor_ami_ids_0 => ami-31c72258
ec2_ancestor_ami_ids_1 => ami-ef01e486
ec2_ancestor_ami_ids_2 => ami-0916f360
ec2_ancestor_ami_ids_3 => ami-c8ac48a1
ec2_ancestor_ami_ids_4 => ami-cd52b6a4
ec2_ancestor_ami_ids_5 => ami-19be966d
ec2_ancestor_ami_ids_6 => ami-65200b11
ec2_ancestor_ami_ids_7 => ami-3d200b49
ec2_ancestor_ami_ids_8 => ami-91200be5
ec2_ancestor_ami_ids_9 => ami-81200bf5
ec2_block_device_mapping_ami => sda1
ec2_block_device_mapping_ephemeral0 => sdb
ec2_block_device_mapping_ephemeral1 => sdc
ec2_block_device_mapping_ephemeral2 => sdd
ec2_block_device_mapping_ephemeral3 => sde
ec2_block_device_mapping_root => /dev/sda1
ec2_block_device_mapping_swap => sda3
ec2_hostname => ip-10-227-43-134.eu-west-1.compute.internal
ec2_instance_action => none
ec2_instance_id => i-9411e7e3
ec2_instance_type => m1.small
ec2_kernel_id => aki-7e0d250a
ec2_local_hostname => ip-10-227-43-134.eu-west-1.compute.internal
ec2_local_ipv4 => 10.227.43.134
ec2_placement_availability_zone => eu-west-1b
ec2_public_hostname => ec2-79-125-33-224.eu-west-1.compute.amazonaws.com
ec2_public_ipv4 => 79.125.33.224
ec2_public_keys_0_openssh_key => ssh-rsa AAA
ec2_ramdisk_id => ari-7d0d2509
ec2_reservation_id => r-c655bab1
ec2_security_groups_0 => rip
ec2_security_groups_1 => defaultcluster => a

In addition if you just pass nice key=val pairs in as user data it will add those as facts too, the last above is from that.

Tagged as: , , , 7 Comments
6Nov/093

Test Driven Deployment – mcollective, puppet, cucumber

With the release of mcollective recently I've been able to work a bit on a deploy problem I've had at a client, I was able to build up the following by combining mcollective, cucumber and the open source mcollective plugins.

The cucumber exploring is of course a result of @auxesis's brilliant cucumber talk at Devops Days recently.

Note: I've updated this from the initial posting, showing how I do filtering with mcollective discovery and put it all into one scenario.

Feature: Update the production systems
 
    Background:
        Given the load balancer has ip address 192.168.1.1
        And I want to update hosts with class roles::dev_server
        And I want to update hosts with fact country=de 
        And I want to pre-discover how many hosts to update
 
    Scenario: Update the website
        When I block the load balancer
        Then traffic from the load balancer should be blocked
 
        When I update the package mywebapp
        Then the package version for mywebapp should be 4.2.6-3.el5
 
        When I unblock the load balancer
        Then traffic from the load balancer should be unblocked

This is completely like any other test driven scenario based system, if it fails to block the firewall deploy will bail out. If it fails to update the package it will bail and finally only if those worked will it unblock the firewall.

Thanks to mcollective this is distributed and parallel over large numbers of machines. I can also apply filters to update just certain clusters using mcollective's discovery features.

Everything's outcome is tested and cucumber will only show the all clear when everything worked on all machines in a consistent way.

This is made possible in part because the mcollective plugins use the Puppet providers underneath the hood, so package and service actions are complete idempotent and repeatable, I can rerun this script 100 times and it will do the same thing.

I have other steps not included here to keep things simple but in a real world I would restart the webserver after the update and I would then call NRPE plugins on all the nodes to make sure their load average is in acceptable ranges before the firewall gets opened letting the load balancer in.

This opens up a whole lot of interesting ideas, kudos to @auxesis and his great talk at devopsdays!

19Oct/096

Reusing Puppet Providers

Last night I was thinking about writing scripts to manage services, packages etc in a heterogeneous environment. This is hard because the operating systems all behave differently.

This is of course a problem that Puppet solves with it's providers system, after some chatting with Luke on IRC I now have a pretty nice solution.

Assuming you don't mind shell scripting in Ruby here's a pretty decent bit of Ruby to manage a service on many different environments.

require 'puppet'
 
service = ARGV.shift
action = ARGV.shift
 
ARGV.length > 0 ? hasstatus = true : hasstatus = false
 
begin
    svc = Puppet::Type.type(:service).new(:name => service, 
        :hasstatus => hasstatus).provider
 
    svc.send action
 
    puts("#{service} #{action} status: #{svc.status}")
rescue Exception => e
    puts("Could not #{action} service #{service}: #{e}")
end
# service.rb httpd stop true
httpd stop status: stopped
 
# service.rb httpd start true
httpd start status: running

You'd probably want to put in support for the pattern parameter to keep certain broken Operating Systems happy, but this is pretty nice and platform independent.

Tagged as: , , 6 Comments
18Oct/093

Middleware for Systems Administration

I spoke a bit on the puppet IRC channel about my middleware based systems administration tool, I made a video to demo it below.

The concept is that I use publish / subscribe middleware - ActiveMQ with Stomp in my case - to do one-off administration. Unlike using Capistrano or some of those tools I do not need lists of machines or visit each machine with a request because the network supports discovery and a single message to the middleware results in 10s or 100s or 1000s of machines getting the message.

This means any tasks I ask to be done happens in parallel on any number of machines typically I see ~100 machines finish the task in the same time as 1 machine would and no need for SSH or anything like that.

The app server and client libs I wrote take away all the complexities of the middleware and takes care of crypto signing requests, only responding to requests that has been signed properly etc, serializing and deserialization of data etc.

Discovery is built in and it supports puppet classes and facts and a few other criteria I use for my own systems so there is no need to build any kind of system that keeps track of what machines I have with what version of operating system etc. As long as is on the middleware I can find it.

The bulk - timeout handling and so forth removed - of the ping app you see in the demo can be seen here, client:

client = Stomphost::Client.new(config)
client.sendreq(Time.now.to_f, "echo")
 
loop do
    resp = client.receive 
    elapsed = (Time.now.to_f - resp[:body]) * 1000
end

And the agent is just this:

module Stomphost
    module Agent
        class Echo
            def handlemsg(msg)
                msg[:body]
            end
        end
    end
end

You can see that even data types like the float will flow cleanly through end to end.

Watch the video, I mention my uses cases but it includes distributed Exim administration, package updates, services restarts, iptables management and much more.

UPDATE: This code has now been released as an Open Source Apache 2 licenced project at mcollective.googlecode.com