www.devco.net by r.i.pienaar

12Mar/100

Puppet Concat 20100312

I am pleased to announce the next version of my Puppet Concat script, we now have 0.24.8 and newer support and a few smaller bits mentioned below.

For background of what this is about please see my earlier post: Building files from fragments with Puppet

New in this release

Paul Elliot sent in most of the patches that enabled this release, lots of thanks Paul!

  • 0.24.8 and newer is supported
  • You can now prepend warnings to generated files as a shell style comment using the warn property
  • You can enable the ability to create empty concat files using the force property
  • You can configure the location of your sort binary in setup.pp

The code should auto configure for 0.24.8 use, if it does not work please see setup.pp.

You can grab the code here.

Known issues

As with my earlier attempts at making a concat tool for 0.24.x this version when used on 0.24 will raise some false notifies. Basically the method we use to clear the concat store of unmanaged files has a side effect and on the next run you will get an unneeded notify. Puppets behavior has improved in 0.25 so it works as expected there, for 0.24 though there is no known work around.

You cannot change the owner of a file, I know how to work around this issue and will have something in the next release.

Tagged as: , No Comments
3Mar/100

Puppet localconfig parser – 20100303

I've had some good feedback on my previous post about the puppet localconfig parser, have implemented the requested features so here's a new version.

First the ability to limit what resources are being printed:

# parselocalconfig.rb --limit package
Classes included on this node:
        fqdn
        common::linux
 
Resources managed by puppet on this node:
        package{redhat-lsb: }
                defined in common/modules/puppet/manifests/init.pp:15

You should only see package resources. You can also disable the classes list using --no-classes and on 0.25.x disable the tags list with --no-tags.

I've improved the detection of where to find the yaml file for 0.25 nodes and added an option --config if your config file is not in the usual place.

You can get the latest version here.

Tagged as: , , No Comments
26Feb/104

What does Puppet manage on a node?

Last year I wrote a tool to parse the localconfig.yaml from Puppet 0.24 and display a list of resources and classes. This script failed when 0.25 came out, I've updated it for 0.25 support.

The yaml cache has some added features in 0.25 so now I can also show the list of tags on a node, output would be:

# parselocalconfig.rb /var/lib/puppet/client_yaml/catalog/fqdn.yaml
Classes included on this node:
        fqdn
        common::linux
        <snip>
 
Tags for this node:
        fqdn
        common::linux
        <snip>
 
Resources managed by puppet on this node:
        yumrepo{centos-base: }
                defined in common/modules/yum/manifests/init.pp:24
 
        file{/root/.ssh: }
                defined in common/modules/users/manifests/root.pp:20
 
        <snip>

You can get the script that supports both 0.24 and 0.25 here.

Tagged as: , , 4 Comments
19Feb/1018

Building files from fragments with Puppet

While building up complex configs with Puppet you often need to build up one file from many fragments. This is useful for files like older sysctl.conf files and maybe named.conf files.

The basic pattern is you want to manage a file, but want the contents to be very different from node to node. A fragment based system lets you register different contents into a file on different nodes. It's exactly like the conf.d directory you'd find in Apache but for daemons that does not support this construct on their own.

I've had an older version of this floating around but had to clean it up for a client today so thought I might as well do a proper job, release it and get some more eye balls on it. This version is specific to Puppet 0.25.x, I will soon make a >= 0.24.8 version too since that is what my client is on.

An example says more than words, so lets create something to manage sysctl.conf:

# Initial setup
class sysctl {
   include concat::setup
 
   exec{"reload-sysctl":
      refreshonly => true,
      command => "/sbin/sysctl -p"
   }
 
   concat{"/etc/sysctl.conf":
      notify => Exec["reload-sysctl"],
   }
}
 
# use this to set values
define sysctl::setting($value) {
   concat::fragment{"sysctl_${name}": 
      target => "/etc/sysctl.conf",
      content => "${name} = ${value}\n",
   }
}

The above sets up a class that will create an empty sysctl.conf and provides an utility for setting individual values. Whenever the sysctl.conf file gets changed the changes will be made live using the refreshonly exec.

Lets see how we might use it:

node "your.example.com" {
   include sysctl
 
   sysctl::setting{"net.ipv4.ip_forward":
      value => 1
   }
}

You can see this looks and feels a lot like a native type but without a lot of the hassle it would take to write one, you can really get a lot of mileage out of this pattern. The concat is clever enough to unregister the setting should you remove lines 4 to 6 in the above node block.

A cleaner approach would be to just make classes like sysctl::ipv4_forward that you can include on the nodes that need it.

You can grab the current code here.

Tagged as: , , 18 Comments
18Feb/101

Custom deployer using MCollective

One of the goals of building the SimpleRPC framework and the overall speed of MCollective is to create interactive tools to manage your infrastructure in a way that it all just seems like a single point of entry with one machine. I've blogged a bit about this before with how I manage Exim clusters.

I've recently built a deployer for a client that does some very specific things with their FastCGI, packages and monitoring in a way that is safe for developers to use. I've made a sanitized demo of it that you can see below. It's sanitized in that the hostnames are replaced with hashes and some monitoring details removed but you'll get the idea.

As usual it's best to just look at the video on youtube in it's HD mode.

17Feb/100

Few Rubyisms

While looking at some bits of other peoples Ruby code I came across a few shortcuts and interesting structures worth mentioning.

Exception handling shortcut

First up a shortcut to catch exceptions thrown by a method:

def say_foo
   puts "foo" if doit
rescue Exception
   puts "#fail"
end

So since we didn't define doit this will raise an exception, which will be handled. Nice shortcut to avoid an extra inner begin / rescue block.

sprintf equivelant

Ruby supports sprintf style string building in a handy little shortcut:

puts "%2.6f\n%d" % [1, 1]

This produces:

$ ruby test.rb
1.000000
1

Get a value from a hash with default for non existing

This is really nice, I've written way too many constructs like this:

foo.include?(:bar) ? bar = foo[:bar] : bar = "unknown"

One option that I was told about was this:

bar = foo[:bar] || "unknown"

But that does not work if you had false in the hash, or maybe even nil.

Turns out there's an awesome shortcut for this:

bar = foo.fetch(:bar, "unknown")

Reloading a class

Sometimes you want to reload a class you previously loaded with require. I have the need in my plugin manager for mcollective. There's a simple fix by simply using Kernel#load to load the .rb file, each time you load it the file will be reloaded from disk.

irb(main):001:0> load "test.rb"
=> true
irb(main):002:0> Foo.doit
foo
irb(main):003:0* load "test.rb"
=> true
irb(main):004:0> Foo.doit
foo foo

In between lines 2 and 3 I edited the file test.rb and just reloaded it, the changes on disk reflected in the current session. The main difference is that you need to supply the full file name and not just test like you would with require.

Tagged as: , , No Comments
5Feb/102

Adding methods to a ruby class

I'm just blogging this because it took me ages to figure out, it seems so simple now but I guess that's how it usually goes.

The problem I have is I want a plugin to be able to either make a method using the normal Ruby def foo or via some DSL'ish helpers.

class Foo<Base
   register_action(:name => "do_something", :description => "foo")
 
   def do_something_action
   end
 
   register_action(:name => "do_something_else", :description => "foo") do
      # body of the action here
   end
end

The above code should make me two methods - do_something_action and do_something_else_action - they should be identical to viewers from the outside. Here's the base class that makes this happen correctly:

class Base
   def self.register_input(input, &block)
      name = input[:name]
 
      self.module_eval { define_method("#{name}_action", &block) } if block_given?
   end
end

It's pretty simple, we're just using define_method in the scope of the module and that does the rest.

Tagged as: , 2 Comments
3Feb/100

MCollective Agent Introspection

With the new SimpleRPC system in MCollective we have a simple interface to creating agents. The way to call an agent would be:

$ mc-rpc service status service=httpd

This is all fine and well and easy enough, however it requires you to know a lot. You need to know there's a status action and you need to know it expects a service argument, not great.

I'm busy adding the ability for an agent to register its metadata and interface so that 3rd party tools can dynamically generate useful interfaces.

A sample registration for service agent is:

register_meta(:name        => "SimpleRPC Service Agent",
              :description => "Agent to manage services using the Puppet service provider",
              :author      => "R.I.Pienaar",
              :license     => "GPLv2",
              :version     => 1.1,
              :url         => "http://mcollective-plugins.googlecode.com/",
              :timeout     => 60)
 
["start", "stop", "restart", "status"].each do |action|
    register_input(:action      => action,
                   :name        => "service",
                   :prompt      => "Service Name",
                   :description => "The service to #{action}",
                   :type        => :string,
                   :validation  => '^[a-zA-Z\-_\d]+$',
                   :maxlength   => 30):

This includes all the meta data, versions, timeouts, validation of inputs, prompts and help text for every input argument.

Using this we can now generate dynamic UI's, and do something like JavaDoc generated documentation. I've recorded a little video demonstrating a proof of concept Text UI that uses this data to generate a UI dynamically. This is ripe for integration into tools like Foreman and Puppet Dashboard.

Please watch the video here, best viewed full screen.

24Jan/100

MCollective 0.4.3 Auditing

I just released version 0.4.3 of mcollective which brings a new auditing capability to SimpleRPC. Using the auditing system you can log to a file on each host every request or build a centralized auditing system for all requests on all nodes.

We ship a simple plugin that logs to the local harddrive but there is also a community plugin that creates a centralized logging system running over MCollective as a transport.

This is the kind of log the centralized logger will produce:

01/24/10 18:24:20 dev1.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: 01/24/10 18:24:20 caller=uid=500@ids1.my.net agent=iptables action=block
01/24/10 18:24:20 dev1.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: {:ipaddr=>"114.255.136.120"}
01/24/10 18:24:20 dev2.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: 01/24/10 18:24:20 caller=uid=500@ids1.my.net agent=iptables action=block
01/24/10 18:24:20 dev2.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: {:ipaddr=>"114.255.136.120"}
01/24/10 18:24:20 dev3.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: 01/24/10 18:24:20 caller=uid=500@ids1.my.net agent=iptables action=block
01/24/10 18:24:20 dev3.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: {:ipaddr=>"114.255.136.120"}

Here we see 3 nodes that got a request to add 114.255.136.120 to their local firewall. The request was sent by UID 500 on the machine ids1.my.net. The request is of course the same everywhere so the request id is the same on every node, the log shows agent and all parameters passed.

14Jan/100

Better way to query facts

Facter has some annoying bug where it won't always print all facts when called like facter fact, ones that require dynamic lookups etc just won't print.

This is a long standing bug that doesn't seem to get any love, so I hacked up a little wrapper that works better.

#!/usr/bin/ruby
 
require 'facter'
require 'puppet'
 
Puppet.parse_config
unless $LOAD_PATH.include?(Puppet[:libdir])
    $LOAD_PATH << Puppet[:libdir]
end
 
facts = Facter.to_hash
 
if ARGV.size > 0
    ARGV.each do |f|
        puts "#{f} => #{facts[f]}" if facts.include?(f)
    end
else
    facts.each_pair do |k,v|
        puts("#{k} => #{v}")
    end
end

It behaves by default as if you ran facter -p but you can supply as many fact names as you want on the command line to print just the ones requested.

$ fctr uptime puppetversion processorcount
uptime => 8 days
puppetversion => 0.25.2
processorcount => 1
Tagged as: , , No Comments