www.devco.net by r.i.pienaar

3Mar/100

Puppet localconfig parser – 20100303

I've had some good feedback on my previous post about the puppet localconfig parser, have implemented the requested features so here's a new version.

First the ability to limit what resources are being printed:

# parselocalconfig.rb --limit package
Classes included on this node:
        fqdn
        common::linux
 
Resources managed by puppet on this node:
        package{redhat-lsb: }
                defined in common/modules/puppet/manifests/init.pp:15

You should only see package resources. You can also disable the classes list using --no-classes and on 0.25.x disable the tags list with --no-tags.

I've improved the detection of where to find the yaml file for 0.25 nodes and added an option --config if your config file is not in the usual place.

You can get the latest version here.

Tagged as: , , No Comments
26Feb/104

What does Puppet manage on a node?

Last year I wrote a tool to parse the localconfig.yaml from Puppet 0.24 and display a list of resources and classes. This script failed when 0.25 came out, I've updated it for 0.25 support.

The yaml cache has some added features in 0.25 so now I can also show the list of tags on a node, output would be:

# parselocalconfig.rb /var/lib/puppet/client_yaml/catalog/fqdn.yaml
Classes included on this node:
        fqdn
        common::linux
        <snip>
 
Tags for this node:
        fqdn
        common::linux
        <snip>
 
Resources managed by puppet on this node:
        yumrepo{centos-base: }
                defined in common/modules/yum/manifests/init.pp:24
 
        file{/root/.ssh: }
                defined in common/modules/users/manifests/root.pp:20
 
        <snip>

You can get the script that supports both 0.24 and 0.25 here.

Tagged as: , , 4 Comments
23Feb/100

London Devops Night Venue Sponsor

Tomorrow is our first public London DevOps meetup, it's looking like there will be a lot of people given the activity on Twitter, hopefully it will be a great night.

In the mean time The Guardian has offered us a regular venue at their big auditorium where the London Scale Camp was held. This is an excellent venue setup with projectors and everything. We could get it every month but I figured we'll keep the talks bi-monthly and do drinks night or just something social on the other nights. There are several pubs around the Guardian so we'll do drinks after the 1 or 2 talks a night.

The upcoming dates - excluding the pure pub nights are then:

02/03/2010 7pm – Agile Systems Administration at London Geek Nights – thanks to Thoughtworks
28/04/2010 7pm – Topic TBA – thanks to The Guardian for the venue
30/06/2010 7pm – Topic TBA – thanks to The Guardian for the venue
27/10/2010 6pm – Topic TBA – thanks to The Guardian for the venue
29/12/2010 6pm – Topic TBA – thanks to The Guardian for the venue

Put them in your calendars and again massive thanks to The Guardian for sponsoring us.

Tagged as: , No Comments
19Feb/1013

Building files from fragments with Puppet

While building up complex configs with Puppet you often need to build up one file from many fragments. This is useful for files like older sysctl.conf files and maybe named.conf files.

The basic pattern is you want to manage a file, but want the contents to be very different from node to node. A fragment based system lets you register different contents into a file on different nodes. It's exactly like the conf.d directory you'd find in Apache but for daemons that does not support this construct on their own.

I've had an older version of this floating around but had to clean it up for a client today so thought I might as well do a proper job, release it and get some more eye balls on it. This version is specific to Puppet 0.25.x, I will soon make a >= 0.24.8 version too since that is what my client is on.

An example says more than words, so lets create something to manage sysctl.conf:

# Initial setup
class sysctl {
   include concat::setup
 
   exec{"reload-sysctl":
      refreshonly => true,
      command => "/sbin/sysctl -p"
   }
 
   concat{"/etc/sysctl.conf":
      notify => Exec["reload-sysctl"],
   }
}
 
# use this to set values
define sysctl::setting($value) {
   concat::fragment{"sysctl_${name}": 
      target => "/etc/sysctl.conf",
      content => "${name} = ${value}\n",
   }
}

The above sets up a class that will create an empty sysctl.conf and provides an utility for setting individual values. Whenever the sysctl.conf file gets changed the changes will be made live using the refreshonly exec.

Lets see how we might use it:

node "your.example.com" {
   include sysctl
 
   sysctl::setting{"net.ipv4.ip_forward":
      value => 1
   }
}

You can see this looks and feels a lot like a native type but without a lot of the hassle it would take to write one, you can really get a lot of mileage out of this pattern. The concat is clever enough to unregister the setting should you remove lines 4 to 6 in the above node block.

A cleaner approach would be to just make classes like sysctl::ipv4_forward that you can include on the nodes that need it.

You can grab the current code here.

Tagged as: , , 13 Comments
18Feb/100

Custom deployer using MCollective

One of the goals of building the SimpleRPC framework and the overall speed of MCollective is to create interactive tools to manage your infrastructure in a way that it all just seems like a single point of entry with one machine. I've blogged a bit about this before with how I manage Exim clusters.

I've recently built a deployer for a client that does some very specific things with their FastCGI, packages and monitoring in a way that is safe for developers to use. I've made a sanitized demo of it that you can see below. It's sanitized in that the hostnames are replaced with hashes and some monitoring details removed but you'll get the idea.

As usual it's best to just look at the video on youtube in it's HD mode.

Tagged as: , , , No Comments
17Feb/100

Few Rubyisms

While looking at some bits of other peoples Ruby code I came across a few shortcuts and interesting structures worth mentioning.

Exception handling shortcut

First up a shortcut to catch exceptions thrown by a method:

def say_foo
   puts "foo" if doit
rescue Exception
   puts "#fail"
end

So since we didn't define doit this will raise an exception, which will be handled. Nice shortcut to avoid an extra inner begin / rescue block.

sprintf equivelant

Ruby supports sprintf style string building in a handy little shortcut:

puts "%2.6f\n%d" % [1, 1]

This produces:

$ ruby test.rb
1.000000
1

Get a value from a hash with default for non existing

This is really nice, I've written way too many constructs like this:

foo.include?(:bar) ? bar = foo[:bar] : bar = "unknown"

One option that I was told about was this:

bar = foo[:bar] || "unknown"

But that does not work if you had false in the hash, or maybe even nil.

Turns out there's an awesome shortcut for this:

bar = foo.fetch(:bar, "unknown")

Reloading a class

Sometimes you want to reload a class you previously loaded with require. I have the need in my plugin manager for mcollective. There's a simple fix by simply using Kernel#load to load the .rb file, each time you load it the file will be reloaded from disk.

irb(main):001:0> load "test.rb"
=> true
irb(main):002:0> Foo.doit
foo
irb(main):003:0* load "test.rb"
=> true
irb(main):004:0> Foo.doit
foo foo

In between lines 2 and 3 I edited the file test.rb and just reloaded it, the changes on disk reflected in the current session. The main difference is that you need to supply the full file name and not just test like you would with require.

Tagged as: , , No Comments
11Feb/100

London DevOps

Since DevOps Days last year a number of us have been meeting monthly, it was all a bit under the radar and not announced to the wider public.

The thinking was that we wanted to be sure we will do the meetings roughly monthly rather than start something with a lot of noise and then fizzle out.

We've met 4 months in a row and we figured it's about time to throw open the doors. We are therefore meeting on the 24th of February 2010 at The Priory Bar, we'll aim to be there from 6 or 7pm on wards.

We're trying to create a community of like minded people in London - Sysadmins, Ops People, Developers, Puppet Users, Chef Users we do not discriminate. As long as your interested in Agile Infrastructures or in bridging the gap between the Dev and Ops Silos or just want to hang out with a bunch of Systems guys who aren't stuck in the 1980s you're welcome to join us. The basic format we're aiming for is to meet in a pub/restaurant for Lunch or Dinner one month and every second month to try and arrange some meeting with talks.

Our first actual meeting with talks will be in March at a venue provided by Thought Works so the meeting on the 24th is to discuss what we'll be up to there. We'll also be very glad to hear from any companies who would like to offer us some space on a bi-monthly basis for our talks.

While on the subject I should mention an event that is coming up. In March London hosts Q-Con with a track specifically for DevOps and I will be there doing a talk as well as some of our other regular members.

Please help spread the word by tweeting with the #ldndevops hashtag. We've created a site at http://londondevops.org/ that aggregates some of the blogs of people who we already know about in London who are operating in this space. Please contact me here in comments or on Twitter @ripienaar to get yourself added.

Tagged as: , No Comments
5Feb/102

Adding methods to a ruby class

I'm just blogging this because it took me ages to figure out, it seems so simple now but I guess that's how it usually goes.

The problem I have is I want a plugin to be able to either make a method using the normal Ruby def foo or via some DSL'ish helpers.

class Foo<Base
   register_action(:name => "do_something", :description => "foo")
 
   def do_something_action
   end
 
   register_action(:name => "do_something_else", :description => "foo") do
      # body of the action here
   end
end

The above code should make me two methods - do_something_action and do_something_else_action - they should be identical to viewers from the outside. Here's the base class that makes this happen correctly:

class Base
   def self.register_input(input, &block)
      name = input[:name]
 
      self.module_eval { define_method("#{name}_action", &block) } if block_given?
   end
end

It's pretty simple, we're just using define_method in the scope of the module and that does the rest.

Tagged as: , 2 Comments
3Feb/100

MCollective Agent Introspection

With the new SimpleRPC system in MCollective we have a simple interface to creating agents. The way to call an agent would be:

$ mc-rpc service status service=httpd

This is all fine and well and easy enough, however it requires you to know a lot. You need to know there's a status action and you need to know it expects a service argument, not great.

I'm busy adding the ability for an agent to register its metadata and interface so that 3rd party tools can dynamically generate useful interfaces.

A sample registration for service agent is:

register_meta(:name        => "SimpleRPC Service Agent",
              :description => "Agent to manage services using the Puppet service provider",
              :author      => "R.I.Pienaar",
              :license     => "GPLv2",
              :version     => 1.1,
              :url         => "http://mcollective-plugins.googlecode.com/",
              :timeout     => 60)
 
["start", "stop", "restart", "status"].each do |action|
    register_input(:action      => action,
                   :name        => "service",
                   :prompt      => "Service Name",
                   :description => "The service to #{action}",
                   :type        => :string,
                   :validation  => '^[a-zA-Z\-_\d]+$',
                   :maxlength   => 30):

This includes all the meta data, versions, timeouts, validation of inputs, prompts and help text for every input argument.

Using this we can now generate dynamic UI's, and do something like JavaDoc generated documentation. I've recorded a little video demonstrating a proof of concept Text UI that uses this data to generate a UI dynamically. This is ripe for integration into tools like Foreman and Puppet Dashboard.

Please watch the video here, best viewed full screen.

Tagged as: , , No Comments
24Jan/100

MCollective 0.4.3 Auditing

I just released version 0.4.3 of mcollective which brings a new auditing capability to SimpleRPC. Using the auditing system you can log to a file on each host every request or build a centralized auditing system for all requests on all nodes.

We ship a simple plugin that logs to the local harddrive but there is also a community plugin that creates a centralized logging system running over MCollective as a transport.

This is the kind of log the centralized logger will produce:

01/24/10 18:24:20 dev1.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: 01/24/10 18:24:20 caller=uid=500@ids1.my.net agent=iptables action=block
01/24/10 18:24:20 dev1.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: {:ipaddr=>"114.255.136.120"}
01/24/10 18:24:20 dev2.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: 01/24/10 18:24:20 caller=uid=500@ids1.my.net agent=iptables action=block
01/24/10 18:24:20 dev2.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: {:ipaddr=>"114.255.136.120"}
01/24/10 18:24:20 dev3.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: 01/24/10 18:24:20 caller=uid=500@ids1.my.net agent=iptables action=block
01/24/10 18:24:20 dev3.my.net> d53a8306f20e9b3a0f7946adccd6eb5e: {:ipaddr=>"114.255.136.120"}

Here we see 3 nodes that got a request to add 114.255.136.120 to their local firewall. The request was sent by UID 500 on the machine ids1.my.net. The request is of course the same everywhere so the request id is the same on every node, the log shows agent and all parameters passed.

Tagged as: , , No Comments
   Older Entries »

Links

Tweets

Reading

Planned books:

Current books:

  • Ariel: A Book of the Change

    Ariel: A Book of the Change by Steven R. Boyett

Recent books:

View full Library

Categories

Archives

Licence

Blog under the Creative Commons Attribution-NonCommercial 3.0 License
Creative Commons License