Archive | Front Page RSS feed for this section

Tips and Tricks for puppet debugging

We often have people asking ‘will this work…’ or ‘how do I…’ type questions on IRC, usually because it seems like such a big deal to upload a bit of code to your master just to test.

Here are a quick few tips and tricks for testing out bits of puppet code to get the feel for things, I’ll show you how to test code without using your puppetmaster or needing root, so it’s ideal for just playing around on your shell, exploring the language structure and syntax..

Getting more info
Most people know this one, but just running puppetd –test will highlight all the various steps puppet is taking, any actions its
performing etc in a nice colored display, handy to just do one-offs and see what is happening.

Testing small bits of code:
Often you’re not sure if you’ve got the right syntax, especially for case statements, selectors and such, or you just want to test out some scenarios, you can’t just dump the stuff into your master because it might not even compile.

Puppet comes with a executable called ‘puppet’ that’s perfect for this task, simply puppet some manifest into a file called test.pp and run it:

puppet --debug --verbose test.pp

This will run through your code in test.pp and execute it.  You should be aware that you couldn’t fetch files from the master in this case since it’s purely local but see the File types reference for a explanation of how the behavior changes – you can still copy files, just not from the master.

You can do anything that puppet code can do, make classes, do defines, make sub classes, install packages, this is great for testing out small concepts in a safe way.  Everything you see in this article was done in my shell like this.

What is the value of a variable?
You’ve set a variable in some class but you’re not sure if it’s set to what you’re expecting, maybe you don’t know the scoping rules just yet or you just want to log some state back to the node or master log.

In a usual scripting language you’d add some debug prints, puppet is the same.  You can print simple values in the master log by doing this in your manifest:

notice("The value is: ${yourvar}")

Now when you run your node you should see this being printed to the syslog (by default) on your master.

To log something to your client, do this:

notify{"The value is: ${yourvar}": }

Now your puppet logs – syslog usually – will show lines on the client or you could just do puppetd –test on the client to see it run and see your debug bits.

What is in an array?
You’ve made an array, maybe from some external function, you want to know what is in it?  This really is an extension to the above hint that would print garbage when passed arrays.

Building on the example above and the fact that puppet loops using resources, lets make a simple defined type that prints each member of an array out to either the master or the client (use the technique above to choose)

$arr = [1, 2, 3]
 
define print() {
   notice("The value is: '${name}'")
}
 
print{$arr: }

This will print one line for each member in the array – or just one if $arr isn’t an array at all.

$ puppet test.pp
notice: Scope(Print[1]): The value is: '1'
notice: Scope(Print[3]): The value is: '3'
notice: Scope(Print[2]): The value is: '2'

Writing shell scripts with puppet?
Puppets a great little language, you might even want to replace some general shell scripts with puppet manifest code, here’s a simple hello world:

#!/usr/bin/puppet
 
notice("Hello world from puppet!")
notice("This is the host ${fqdn}")

If we run this we get the predictable output:

$ ./test.pp
notice: Scope(Class[main]): Hello world from puppet!
notice: Scope(Class[main]): This is the host your.box.com

And note that you can access facts and everything from within this shell script language, really nifty!

Did I get the syntax right?
If I introduce a deliberate error in the code above – remove the last ” – it would blow up, you can test puppet syntax using puppet itself:

$ puppet --parseonly test.pp
err: Could not parse for environment production: Unclosed quote after '' in 'Hello world from puppet!)
' at /home/rip/test.pp:1

You can combine this with a pre-commit hook on your SCM to make sure you don’t check in bogus stuff.

How should I specify the package version? or user properties?
Often you’ve added a package, but not 100% sure how to pass the version string  to ensure => or you’re not sure how to specify password hashes etc, puppet comes with something called ralsh that can interrogate a running system, some samples below:

% ralsh package httpd
package { 'httpd':
    ensure => '2.2.3-22.el5.centos.2'
}
 
% ralsh package httpd.i386
package { 'httpd.i386':
   ensure => '2.2.3-22.el5.centos.2'
}
 
% ralsh user apache
user { 'apache':
   password => '!!',
   uid => '48',
   comment => 'Apache',
   home => '/var/www',
   gid => '48',
   ensure => 'present',
   shell => '/sbin/nologin'
}

Note in the 2nd case I ran it as root, puppet needs to be able to read shadow and so forth.  The code it’s outputting is valid puppet code that you can put in manifests.

Will Puppet destroy my machine?
Maybe you’re just getting ready to run puppet on a host for the first time or you’re testing some new code and you want to be sure nothing terrible will happen, puppetd has a no-op option to make it just print what will happen, just run puppetd –test –noop

What files etc are being managed by puppet?
See my previous post for a script that can tell you what puppet is managing on your machine, note this does not yet work on 0.25.x branch of code.

Is my config changes taking effect?
Often people make puppet.conf changes and it just isn’t working, perhaps they put them in the wrong [section] in the config file, a simple way to test is to run puppetd –genconfig this will dump all the active configuration options.

Be careful though don’t just dump this file over your puppet.conf thinking you’ll have a nice commented file, this wont work as the option for genconfig will be set to true in the file and you’ll end up with a broken configuration.  In general I recommend keeping puppet.conf simple and short only showing the things you’re changing away from defaults, that makes it much easier to see what differs from standard behavior when asking for help.

Getting further help
Puppet has a irc channel #puppet on freenode, we try and be helpful and welcoming to newcomers, pop by if you have questions.

As always you need to have these wiki pages bookmarked and they should be your daily companions:

Read full storyComments { 2 }
Managing web traffic with ruby-pdns

Managing web traffic with ruby-pdns

A short while ago I wrote about releasing a Ruby Development framework for PowerDNS the release is still early days, feature complete but needs some robustness tweaks and a new release will be out in a week or so to address that.

I wanted though to highlight some success that I’ve had using it.  I have a small static farm for a client that handles around 2MiB/sec of 200×200 jpg files, this setup is for a startup so out of necessity its all built to be cheap, I host on networks I don’t own yet I need pretty good control over it, what IPs will be used to serve traffic and so forth.

The graph above shows the case before caused by the windows DNS bug, you’ll see the bottom host is working pretty hard getting a large chunk of the bandwidth.

This is a problem because come mid month this poor machine has already used up its allocation of 2.5TiB of transfer and I need to move it from the pool.

So my goal was to shift the traffic to the yellow and green machines and just generally balance things out a bit. I used the Weighted Round Robin feature of ruby-pdns to adjust the biases, it took a bit of fiddling because for some other reason even when this machine gets fewer requests per second it still seems to manage more in terms of bandwidth, this is the eventual code snippet:

ips = ["213.x.x.232",                         # dark blue
          "88.x.x.201",                       # lighter blue
          "82.x.x.180",                       # yellow
          "82.x.x.181"].randomize([1,2,2,3])  # green
 
answer.shuffle false
answer.content [:A, ips[0]]
answer.content [:A, ips[1]]
answer.content [:A, ips[2]]

The thresholds seems odd but that’s what worked after some fiddling, see the graph below.

This is much nicer balanced, it’s not perfect and I doubt I will get it perfect with just 4 machines to play with but I believe it’s already at the point where it means I can use all my machines for the entire month without hitting any limits.

Here’s another graph over the week showing things side by side:

The improvement is very obvious in this graph and you can see I’ve not lost anything in performance between first day and last day on the graph in terms of throughput (the lower days were days where lower traffic is expected).

If I look at my actual transfer used it’s better balanced now, first lets see the 12th:

08/12/09    12.67 GiB |  46.42 GiB |  59.09 GiB |   5.74 Mbit/s
08/12/09     7.71 GiB |  21.32 GiB |  29.04 GiB |   2.82 Mbit/s
08/12/09     9.05 GiB |  23.05 GiB |  32.10 GiB |   3.12 Mbit/s
08/12/09     6.94 GiB |  16.56 GiB |  23.50 GiB |   2.28 Mbit/s

Again the skew is very clear with a 23GiB on the lowest compared to 59GiB on the highest use machine, on the 17th it looked a lot better:

08/17/09     7.84 GiB |  28.55 GiB |  36.39 GiB |   3.53 Mbit/s
08/17/09     8.46 GiB |  25.66 GiB |  34.12 GiB |   3.31 Mbit/s
08/17/09    11.21 GiB |  30.70 GiB |  41.91 GiB |   4.07 Mbit/s
08/17/09    10.25 GiB |  28.20 GiB |  38.46 GiB |   3.73 Mbit/s

Obviously much better when looking at the 2nd to last column. The first column is received the increase in those is down to a slightly lower hit ratio on the caching proxy on these machines meaning it’s fetching more files from origin than the others.

Overall I am extremely pleased with this solution, I agree one should not be using DNS as a hammer to all your nails but for startups and cloud based people who do not have control over networks, BGP tables and so forth this really does represent a viable option to what would otherwise be an extremely expensive problem to solve.

Read full storyComments { 0 }

News from SA

So today I decided to follow @iol on my twitter client to try and keep in touch with things back home.

These are some of the messages I got in the first 11 hours:

  • Still no trace of missing baby http://bit.ly/cUafy
  • ‘I could not clear away images of the dead’ http://bit.ly/NjQxq
  • Woman forgives parents for killing her lover http://bit.ly/xd2P7
  • ‘Mass hysteria’ sweeps six E Cape schools http://bit.ly/yE30R
  • Bona magazine editor killed http://bit.ly/13JN08
  • Suspect shoots himself in groin http://bit.ly/29UNDM
  • Baby’s body found in car boot http://bit.ly/349oq
  • Chabaan guilty of assault http://bit.ly/XpKZs
  • Hammer attack hit straight to teacher’s core http://bit.ly/YbdD6
  • MJC rejects the slaughter of chickens claim http://bit.ly/23igtE
  • ‘I just can’t wait till this is all over’ http://bit.ly/2mWbg
  • Judge’s death: Unanswered questions remain http://bit.ly/4aqPW
  • Slain editor: Family believes he knew killers http://bit.ly/anMLt

That all in 11 hours of headlines.  Shocking.

Read full storyComments { 1 }

Ruby PowerDNS Framework

Regular readers here will know I patch bind with GeoIP extensions, this has served me well but my needs have now outgrown simply doing geo related replies.

I’ve for a long time had an itch to be able to do completely custom DNS, maybe respond to monitoring, or time of day, geographical location or even to work around some unbelievably annoying bugs in windows that breaks all round robin dns, this has not been possible with Bind.

PowerDNS has a backend that simply speaks via STDIN and STDOUT to any script, the documentation though is pretty shoddy but I quickly realized this is the way to go.  Once I figured out all the various weird things about PDNS and the Pipe backend I set about writing a framework to host many records in a single PDNS server – in a way that hides and abstracts all the PowerDNS details from the code

The end goal is that I would dump some Ruby code into a file on the server and it should just be served, when I get new code I just want to overwrite the old code, no restarts or anything it must just serve it.

I wanted the code to be trivially simple, something like this:

module Pdns
  newrecord(“www.your.net”) do |query, answer|
    case country(query[:remoteip])
      when “US”, “CA”
        answer.content “64.xx.xx.245″

      when “ZA”, “ZW”
        answer.content “196.xx.xx.10″

      else
        answer.content “78.xx.xx.140″
      end
  end
end

should be all that is needed to do GeoIP based serving, and really complex things like weighted random round robins that effectively work around the bugs in client resolvers like the windows one above:

        ips = ["1.x.x.x", "2.x.x.x", "3.x.x.x", "4.x.x.x", "5.x.x.x"]

        ips = ips.randomize([1,5,3,3,3])

        answer.shuffle false
        answer.ttl 300
        answer.content ips[0]
        answer.content ips[1]
        answer.content ips[2]

This code will take 5 ip addresses, shuffle them giving the first one least weight, the 2nd one most weight and return only 3 out of the 5 results, this would be impossible in Bind but trivial to imagine coding if only you could hook into the nameserver.

Anyway, so I wrote a framework that enables exactly this, the code snippets above are actual working snippets.  The code is hosted on Google Code as ruby-pdns and is at version 0.3 at present.

I’ve release tarball and RPM versions of the code, the code is publicly browsable and licensed under the GPLv2. 

At present I think I’ve documented it all fairly well with a good set of Wiki pages though the install instructions for non RPM based install leaves a bit to be desired, I’ll work on improving that.

I’ve been running this code myself serving 10′s of 1000s of queries a day and have used the technique above to work around windows bugs.  I’m looking for testers to start using the code and sending me feedback, there are groups, tickets and all set up for that on Goole Code.

Read full storyComments { 1 }

SSH socks proxies hanging

I use SSH’s socks proxy feature a lot, in fact I use it all the time, most of my browsing, IM, etc all goes over it out via my hosted virtual machines,

I do this to simplify my life for things like firewall rules and also to get around things like age blocks on mobile networks.  I work for a site deemed adult by most of them so I can’t even see my nagios without age verifying.

Recently they have been driving me nuts, every now and then the whole session would just lock up and sit there doing nothing, I’ve not seen this happen before and was a bit stumped.

Turns out, it chooses to speak to TCP/53 sometimes instead of UDP/53 for resolving, not sure why exactly, I’ve not tried to figure out what queries cause this – I know there are limits to response sizes which will force it to go over TCP.  Why it’s only started doing this now I don’t know, maybe a update changed behavior, I’ve never had TCP/53 open on the cache. 

My firewall was blocking TCP/53 on the local cache so this would lock up the whole ssh session, maybe the whole ssh process is single threaded and so waiting in SYN_SENT mode just hangs the whole thing, that’s a bit sucky, I might need a better proxy.

Read full storyComments { 0 }

Imposter Alert!

You’d be thinking based on the last 2 posts that someone is trying to convince the world that I’ve gone mad and do actually like Debian.

Actually I am letting some other people guest blog here, the first is Mark Webster aka LSD, he’s a developer, systems dude and all round kewl guy working in London on all sorts of interesting stuff, most recently about optimizing Linux kernels to get insane amounts of packets per second out of them.

Look out for more great posts from Mark hopefully detailing more of his experiences tuning kernels and such.

I’d also be interested to hear from other like minded people who want to guest blog here, I’ll over the next while take out some of the links and stuff that makes this site personal and more friendly to guest bloggers.

Read full storyComments { 0 }

Introductions

Hello World.

I’m Mark, and I do a lot of programming, designing of systems, working ridiculous hours, ranting about many things, and I am frequently guilty of re-inventing wheels (which shall henceforth be refered to as either ‘improvement’ or ‘learning’ :-) . Currently, I am involved in re-inventing designing a new suite of telephone conference call back-end systems for a rapidly expanding conference call company, from the ground up.
Doing this kind of work on the carrier grade level involves the convergence of a lot of technologies, and there’s a truckload of R&D involved, which is bloody fantastic since I get bored too easily when there’s nothing new to learn, or not enough diversity.
I arrived in this part of the IT industry after doing some weird things:
  • Five years in the games industry (hellishly boring; trust me – I’ll explain why another time)
  • A few years developing bespoke systems, providing services and Linux “appliances” for businesses around South Africa
  • Loads of freelance development on various platforms (mobile handsets, 8-bit embedded systems, even Windows apps)
  • An entire childhood & adolescence involved (misspent?) in the demoscene. Epic fun. Low-level programming, register fiddling, cycle counting and reverse engineering is the shit!
Anyway, talk is cheap, and I’ve clogged the Intertubes quite enough!
Coming up next, something that has been a constant thorn in my side as a wretched Debian user: building custom kernels.
Read full storyComments { 3 }

Talks in South Africa

As mentioned earlier I’ll be going to South Africa for 2 weeks soon, I have a few talks lined up.

I’ll be talking at the Gauteng LUG on the 1st of July 2009 and I will give the same talk in Cape Town LUG the following Thursday – 9th of July.

The talks will be about Configuration Management and about Puppet, both subjects are pretty huge subjects so I won’t pretend to cover either in depth.

I’ll just run through some problems teams who use Linux or Unix have encountered and how CM systems can be used to help with those problems. 

I’ll work through a sample of installing, configuring and starting Apache using Puppet.  Finally I’ll show how company standards can be encapsulated in simple re-usable Puppet logic to easily roll out large amounts of vhosts according to company standards with minimal effort.  The demonstration will be done using 2 virtual machines, one being managed by the other using Puppet.

Ideally the talks won’t be just me standing up and going on for x minutes, I’ll invite participation, answer any questions etc.

Read full storyComments { 0 }

Trip to South Africa

I’ll be heading to South Africa for roughly 2 weeks end of June to early July.  I’ll be visiting Johannesburg, Potchefstroom and Cape Town.

I’d be keen to hear from anyone interested in having some hours from me for consultation, it’s not really a lot of time to spend with people so I am not sure what would be viable time spent. 

I think it might be worthwhile from companies who are considering adopting Puppet for configuration management to meet with me, I’d definitely be able to help you decide if it’s the right tool for you or even if you should consider configuration management as a whole.

I might also be open to having a short talk about Puppet if there are any LUGs or something that would be interested in hearing about CM or Puppet.

Read full storyComments { 0 }
Load Balancing with HAProxy

Load Balancing with HAProxy

Load Balancers are some of the most expensive bits of equipment small to medium size sites are likely to buy, even more expensive than database servers.

Since I help a number of smaller and young startups a good Open Source load balancer is essential, I use HAProxy for this purpose.

HAProxy is a high performance non threaded load balancer, it supports a lot of really excellent features like regular expression based logic to route certain types of requests to different backend servers, session tracking using cookies or URL parts and has extensive documentation.

Getting a full redundant set of load balancers going with it requires the help of something like Linux-HA which I use extensively for this purpose, the combination of HAProxy and Linux-HA gives you a full active-passive cluster with failover capabilities that really does work a charm.

I recently had to reload a HAProxy instance after about a 100 day uptime, its performance stats were 1.8 billion requests, 15TB out and just short of 2TB in


Worth checking out HAProxy before shelling out GBP15 000 for 2 x hardware load balancers.

Read full storyComments { 0 }