www.devco.net

Flashpolicyd moved to Google Code

2009-10-07T22:01:13Z

Long time readers might know I have a Ruby based Flash Policy Server called flashpolicyd - I blogged about this in the past.

Since I've been very happy with Google Code for hosting ruby-pdns I thought I'd start moving some other projects there too, in the process I'll probably also release some other bits and bobs that I did not previous share the code of.

The new home for flashpolicyd is then: http://code.google.com/p/flashpolicyd/

This was my first major bit of Ruby code and looking back at it now I'm still fairly happy with it but the real proof is in the pudding; my production copy of this code is up over 300 days now, does not leak memory or threads and have served 10s of millions of requests without any problems.

Ruby PDNS External Data

2009-10-06T22:25:24Z

I develop ruby-pdns, so far things have been very static - you give it code and it serves your code, simple stuff.

But this is not what I want, I want a full API enabled DNS server where I can both code the logic for records as well as send environmental data into the DNS server to adjust it's behavior.

I think there's a big gap in the various cloud offerings today wrt to DNS, people end up hosting their DNS and make manual config changes, but this is not what clouds are about, you want to code the entire infrastructure, and it should adjust based on demand. This does not sit well with todays slow and generally crappy DNS. Combined with the fact that clouds don't let you follow the time tested methods for load balancing etc. we need to get creative.

My end game then is to scratch this itch, you should be able to host DNS servers and interact with them fully both in how they decide how to answer requests but also send data to them and adjust the behaviors on the fly.

Version 1 of Ruby PDNS will hopefully achieve this, here is a video of what you might expect to be able to do - the video sends data to the DNS server telling it to take a machine out of the pool for maintenance. There will be REST based APIs that will enable any language to do this, if you want to play the code that you'll see in the video is all in SVN.

I still have some ways to go before version 1, consider this a very early preview after 1 nights hacking on the feature.

Ruby PDNS Version 0.5

2009-10-05T22:11:20Z

I just released version 0.5 of Ruby PDNS. This mainly introduces a full statistics capability and fixes a minor bug or two.

I didn't intend on releasing another version before 1.0 but it turns out the stats feature were quite a big job worthy of a release.

The stats is complicated by the fact that PDNS runs multiple instances of my code sharing the load between them, this makes keeping stats pretty hard because I have no shared memory space like you would in a multi threaded app. So I'd somehow need to dump per process stats and calculate totals. Everyone I asked suggested writing a log and summarizing the log data but this just never felt right to me - there would be issues with log rotation and knowing till when you last counted stats and so forth - so I kept looking for a solution.

I struggled to come up with a good approach for a long time, finally settling on each process writing a YAML file with it's stats and a cron job to aggregate it regularly. This turned out to be quite elegant - much more so than processing logs for example - as the cron job cleans up files it already processed and also ignore too old ones etc. If the cron job keeps running it should be zero maintenance.

I added a script - pdns-get-stats.rb - to query the aggregated stats:

$ pdns-get-stats.rb --usec --record puppet
usagecount:896 totaltime:165844 averagetime:185

$ pdns-get-stats.rb --usec --record ruby-pdns-totals
usagecount:68705 totaltime:11056529 averagetime:160

The puppet record is using GeoIP so you'd expect it to be a bit slower than ones that just shuffle records and this is clear above, though I am quite pleased with the performance of the GeoIP code. Times are in microseconds.

Those who know Cacti will recognise the format as being compatible with the Cacti plugin standard - if there is demand I can easily add Munin, Collectd etc query tools

Using the details I was able to draw a quick little graph below using Cacti

The release is available in RPM, tgz and gem at the project site.

The next major step is to write a API that is externally accessible, perhaps using REST over HTTP, this means you could send data from your monitoring or CMDB and adjust weights or really anything you can code using external data.

Simple Puppet Module Structure

2009-09-28T20:31:57Z

Puppet supports something called Modules, it's a convenient collection of templates, files, classes, types and facts all related to one thing. I find it makes it easy to think about a problem in an isolated manner and it should be the default structure everyone use.

Usually though the problem comes with how to lay out modules and how to really use the Puppet classes system, below a simple module layout that can grow with you as your needs extend. I should emphasis the simple here - this example does not cater well for environments with mix of Operating Systems, or weird multi site setups with overrides per environment, but once you understand the design behind this you should be able to grow it to cater for your needs. If you have more complex needs, see this post as a primer on using the class relationships effectively.

For the most part I find I make modules for each building block - apache, syslog, php, mysql - usually the kind of thing that has:

One or more packages - or at least some install process.
One or more config files or config actions
A service or something that you do to start it.

You can see there's an implied dependency tree here, the config steps require the package to be installed and the service step require the config to be done. Similarly if any config changes the services might need to restart.

We'd want to cater for the case where you could have 10 config files, or 10 packages, or 100s of files needed to install the service in question. You really wouldn't want to create a big array of require => resources to control the relationships or to do the notifications, this would be a maintenance nightmare.

We're going to create a class for each of the major parts of the module - package, config and service. And we'll use the class systems and relationships, notifies etc to ensure the ordering, requires and notifies are kept simple - especially so that future changes can be done in one place only without editing everywhere else that has requires.

class ntp::install {
    package{"ntpd":
        ensure => latest
    }
}

class ntp::config {
   File{
      require => Class["ntp::install"],
notify => Class["ntp::service"],
owner => "root",
group => "root",
mode => 644
   }

   file{"/etc/ntp.conf":
      source => "puppet:///ntp/ntp.conf";
         "/etc/ntp/step-tickers":
      source => "puppet:///ntp/step-tickers";
   }
}
}

class ntp::service {
   service{"ntp":
ensure => running,
enable => true,
require => Class["ntp::config"],
   }
}

class ntp {
   include ntp::install, ntp::config, ntp::service
}

If we now just do include ntp everything will happen cleanly and in order.

There's a few things you should notice here. We're using the following syntax in a few places:

require => Class["ntp::something"]
notify => Class["ntp::service"]
before => Class["ntp::something"]
subscribe => Class["ntp::config"]

All of the usual relationship meta parameters can operate on classes, so what this means is if you require Class["ntp::config"] then both ntp.conf and step-tickers will be created before the service starts without having to list the files in your relationships. Similarly if you notify a class then all resources in that class gets notifies.

The preceding paragraph is very important to grasp, typically we'd try to maintain arrays of files in require trees and later if we add a new config file we have to go hunt everywhere that needs to require it and update those places, by using this organization approach we never have to update anything. If I add 10 services and 100 files into the config and service classes everything else that requires ntp::service will still work as expected without needing updates.

This is a very important decoupling of class contents with class function. Your other classes should never be concerned with class contents only class function.

As a convention this layout is a good sample, if all your daemons gets installed in the daemon::install class you'll have no problem knowing where to find the code to adjust the install behavior - perhaps to upgrade to the next version - you should try to come up with a set of module naming conventions that works for you and stick to it throughout your manifests - I have a rake task to setup my usual module structure, they're all the same and the learning curve for new people is small.

By building up small classes that focus on their task or subtask you can build up ever more powerful modules. We have a simple example here, but you can see how you could create ntp::master and ntp::client wrapper classes that would simply reuse the normal ntp::install and ntp::service classes.

As I mentioned this is a simple example for simple needs, the real take away here is to use class relationships and decouple the contents of those classes with their intent this should give you a more maintainable code set.

Ultimately conventions such as these will be needed to get us all one step closer to being able to share modules amongst each other as the specifics of making Apache on different distributions work will not matter - we'd just know that we want Class["apache::service"] for example.

Vim and Puppet

2009-09-22T13:38:25Z

I use vim for almost all my editing needs, till now I often thought I should give using TextMate another try due to the nice Bundle that Masterzen made but there's just something about desktop editing that puts me off.

I've also been renewing my efforts to better my VIM use and came across the excellent snipMate that enables you to do almost the same thing as textmate - the syntax for the snippets is even roughly the same.

I made a quick snippet file this morning for some common used bits that you can find here, see it in action below or view the movie better here:

Each time you see the input cursor jump between fields in the templates I'm simply pressing tab, shift-tab would cycle backward.

There are some nice things here you might not notice, for example editing a file test.pp and adding a class it will automatically fill in the class name as test. Same for nodes, it's really very powerful and the snippets are trivial to write.

I am using the vim syntax file that comes in the puppet tarball and the adaryn color scheme that ships with vim.

Snappy Snaps 120 Development

2009-09-14T08:21:28Z

Since moving into my current flat I've not really had a convenient place to develop film so have been putting the black and white shooting on the backburner.

During the last week though I for some other reason thought to ask the Snappy Snaps near me and sure enough they did do 120 black and white. Problem is they wanted £18/roll developed and scanned and want 3 days to do it in.

So I asked around and found out that the Snappy Snaps in Wardour Street London does it for £10/roll scanned on a standard 1 hour wait, that's very good.

Below a scan direct from their scanner, I didn't touch it in any way (click for full size):

You can see some more from this roll here it was taken on Ilford FP4 with my Bronica SQA, this is also the first time in over a year that I touched this camera so was fumbling around a bit, will get back into it now I think especially with a quick development place just 5 minutes from my office.

More on Puppet 0.25 upgrade

2009-09-11T21:25:06Z

Previously I posted a quick bit of information about Puppet 0.25 upgrade and how it sped up file transfers, below some more information.

First some metrics from the puppetmaster before and after, this particular master handles around 50 clients on a 30 minute interval. The master is a Mongrel + Apache master with 4 worker processes running on a 64bit CentOS 5.3 VM with 1GB RAM

First the CPU graph, blue is user time red is system time:

The next is Load Average, it's a stack of 5, 10 and 15 minute averages:

And finally here is a bandwidth of the master blue is outbound and green is incoming:

So this is the master, it's very interesting that the master is a lot busier than before, even with just 50 nodes this is a significant CPU increase, I do not know how this will map to say 500 or 600 nodes but I think larger sites will want to be pretty careful about updating a large chunk of machines without testing the impact.

Finally here is a graph from a node, the node has 450 file resources and doesn't do anything but run Puppet all day - at night in this graph for a short period it did backups after that it idled again. In this case combined with the massive drop in run time the cpu time is also way down, I think this is a massive win - you can always add more masters easily but suffering on all your nodes under puppetd is pretty bad. This on its own for me is a pretty big win for upgrading to Puppet 0.25.

This graph is obviously taken some hours later but it's the same basic scale.

I did not see a noticeable change in memory profile on either master of nodes so no graphs included here for that.

Overall I think this is a big win, but be careful of what happens on your masters when you upgrade.

Puppet 0.25.0 upgrade

2009-09-09T17:39:46Z

Reductive Labs released version 0.25.0 of Puppet recently, there are some teething problems but I got most of it sorted out by upgrading Mongrel.

Today I decided to upgrade 60 of my nodes and they've slowly been running in a for loop all day, the results on machines with many file resources are staggering.

One node that uses snippets heavily has 450 file resources and used to take a while to run, here's some before and after stats:

Finished catalog run in 235.63 seconds
Finished catalog run in 231.43 seconds
Finished catalog run in 42.74 seconds
Finished catalog run in 32.14 seconds

Very impressed.

Snow Leopard and Samba

2009-09-01T21:34:19Z

I did a fresh install of Snow Leopard on my Macbook and soon realized my Samba shares were broken through finder - but still worked from the CLI.

Worse still once I tried to access the shares Finder would basically be dead, you'd need to Force Quit it to make it work again.

Eventually I reached for tcpdump and wireshark and found it's the pesky .DS_Store files again, seems my QNAP is denying access to them, Finder did not cope well with this.

A quick bit of hackery of my smb.conf solved it:

veto files = /.AppleDB/.AppleDouble/.AppleDesktop/.DS_Store/:2eDS_Store/Network Trash Folder/Temporary Items/TheVolumeSettingsFolder/.@__thumb/.@__desc/
delete veto files = yes

Once I got this removed and samba restarted my shares were working again in Snow Leopard. A bit annoying but not too hard in the end.

Complex data and Puppet

2009-08-31T20:29:58Z

Often while writing Puppet manifests you find yourself needing data, things like the local resolver, SMTP relay, SNMP Contact, Root Aliases etc. once you start thinking about it the amount of data you deal with is quite staggering.

It's strange then that Puppet provides no way to work with this in a flexible way. By flexible I mean:

A way to easily retrieve it
A way to choose data per host, domain, location, data center or any other criteria you could possibly wish
A way to provide defaults that allow your code to degrade gracefully
A way to make it a critical error should expected data not exist
A way that works with LDAP nodes, External Nodes or normal node{} blocks

This is quite a list of requirements, and in vanilla puppet you'd need to use case statements, if statements etc.

For example, here's a use case, set SNMP Contact and root user alias. Some machines for a specific client should have different contact details than other, indeed even some machines should have different contact details. There should be a fall back default value should nothing be set specifically for a host.

You might attempt to do this with case and if statements:

class snmp::config {
   if $fqdn == "some.box.your.com" {
      $contactname = "Foo Sysadmin"
      $contactemail = "sysadmin@foo.com"
   }

   if $domain == "bar.com" {
      $contactname = "Bar Sysadmin"
      $contactemail = "sysadmin@bar.com"
   }

   if $location == "ldn_dc" && (! $contactname && ! $contactemail) {
      $contactname = "London Sysadmin"
      $contactemail = "ldnops@your.com"
   }

   if (! $contactname && ! $contactemail) {
      $contactname = "Sysadmin"
      $contactemail = "sysadmin@you.com"
   }
}

You can see that this might work, but it's very unwieldy and your data is all over the code and soon enough you'll be nesting selectors in case statements inside if statements, it's totally unwieldy not to mention not reusable throughout your code.

Not only is it unwieldy but if you wish to add more specifics in the future you will need to use tools like grep, find etc to find all the cases in your code where you use this and update them all. You could of course come up with one file that contains all this logic but it would be aweful, I've tried it's not viable.

What we really want to do is just this, and it should take care of all the code above, you should be able to call this wherever you want with complete disregard for the specifics of the overrides in data:

$contactname = extlookup("contactname")
$contactemail = extlookup("contactemail")

I've battled for ages with ways to deal with this and have come up with something that fits the bill perfectly, been using it and promoting it for almost a year now and so far found it to be totally life saver.

Sticking with the example above, first we should configure a lookup order that will work for us, here is my actual use:

$extlookup_precedence = ["%{fqdn}", "location_%{location}", "domain_%{domain}", "country_%{country}", "common"]

This sets up the lookup code to first look for data specified for the host, then the location the host is hosted at, then the domain, country and eventually a set of defaults.

My current version of this code uses CSV files to store the data simply because it was convenient and universally available with no barrier to entry. It would be trivial to extend the code to use a database, LDAP or other system like that.

For my example if I put into the file some.box.your.com.csv the following:

contactemail,sysadmin@foo.com
contactname,Foo Sysadmin

And in common.csv if I put:

contactemail,sysadmin@you.com
contactname,Sysadmin

The lookup code will use this data whenever extlookup("contactemail") gets called on that machine, but will use the default when called from other hosts. If you follow the logic above you'll see this completely replace the case statement above with simple data files.

Using a system like this you can model all your data needs and deal with the data and your location, machine, domain etc specific data outside of your manifests.

The code is very flexible, you can reuse existing variables in your code inside your data, for example:

ntpservers,1.pool.%{country}.ntp.org,2.pool.%{country}.ntp.org

In this case if you have $country defined in your manifest the code will use this variable and put it into the answer. This snippet of data also shows that it supports arrays.

Here is another use case:

package{"screen":
ensure => extlookup("pkg_screen", "absent")
}

This code will ensure that, unless otherwise specified, I do not want to have screen installed on any of my servers. I could now though decide that all machines in a domain, or all machines in a location, country or specific hosts could have screen installed by simply setting them to present in the data file.

This makes the code not only configurable but configurable in a way that suits every possible case as it depends on the precedence defined above. If your use case does not rely on countries for example you can just replace the country ordering with whatever works for you.

I use this code in all my manifests and it's helped me to make an extremely configurable set of manifests. It has proven to be very flexible as I can use the same code for different clients in different industries and with different needs and network layouts without changing the code.

The code as it stands is available here: http://www.devco.net/code/extlookup.rb

Follow the comments in the script for install instructions and full usage guides.

Ruby PDNS 0.4

2009-08-31T17:51:08Z

Yesterday I released version 0.4 of my Ruby PowerDNS development framework.

Version 0.3 was a feature complete version but lacked in some decent error handling in all cases which resulted in weird unexplained crashes when things didn't work as hoped, for example syntax errors in records could kill the whole thing.

Version 0.4 is a big push in stability, I've added tons of exception handling there should now be very few cases of unexpected terminations, I know of only one case and that's when the log can't be written too, all other cases should be logged and recovered from in some hopefully sane way.

I've written loads of unit tests using Test::Unit and have created a little testing harness that can be used to test your records without putting them on the server, using this for example you can test GeoIP based records easily since you can specify any source address.

Overall I think this is a production ready release, it would be a 1.0 release was it not for some features I wish to add before calling it 1.0. The features are about logging stats and about consuming data from external sources, these will be my next priorities.

hetzner.de hardware policies

2009-08-24T17:02:46Z

So I use Hetzner a lot for my machines, I've about 10 to 15 of their machines now across various clients and am mostly quite happy with them. They provide a service that matches the price - ie. good enough.

One area of their service though really grates me, they give you old machines, when those machines fail they replace them with other old machines similarly for drives etc.

On more than one occasion now have I had hard drives fail only to see them replaced with other shitty drives. Each time they claim the drives are well tested and each time they pull the old 'it could be the cable' trick and then replace the machine and the drive.

Since this has happened to me every single time I've changed a disk so far I have to wonder if this is everyones experience?

From where I sit its simple. They made a choice to take out drives reported broken by someone, they then test it and put it back when their tests fail to find any problem, they do this to save them money knowing full well that drives will fail and all they're doing is shifting the risk onto their clients, while the clients keep subsidizing their expansion.

So given this is the quality of service they're aiming at, surely once this policy bites a good long standing user offering some kind of payback for the inconvenience would be good business practice? Apparently not.

This is pretty poor, even after complaining to them they swapped my chassis and again put a disk with > 6000 hours under its belt in my machine.

So I guess you need to be pretty sure your softraids are setup properly when you want to use this company, their support stand is clear:

I'm sorry but we don't promise anywhere that we built always new hardware into our servers. I can only ensure you that all hardware is always well tested and without any problem before we build it into a server.

Ie., screw you, we don't care for any evidence and repeated failures, and we take zero responsibility for our equipment, we'll just keep taking your money.

Tips and Tricks for puppet debugging

2009-08-19T17:16:57Z

We often have people asking 'will this work...' or 'how do I...' type questions on IRC, usually because it seems like such a big deal to upload a bit of code to your master just to test.

Here are a quick few tips and tricks for testing out bits of puppet code to get the feel for things, I'll show you how to test code without using your puppetmaster or needing root, so it's ideal for just playing around on your shell, exploring the language structure and syntax..

Getting more info
Most people know this one, but just running puppetd --test will highlight all the various steps puppet is taking, any actions its performing etc in a nice colored display, handy to just do one-offs and see what is happening.

Testing small bits of code:
Often you're not sure if you've got the right syntax, especially for case statements, selectors and such, or you just want to test out some scenarios, you can't just dump the stuff into your master because it might not even compile.

Puppet comes with a executable called 'puppet' that's perfect for this task, simply puppet some manifest into a file called test.pp and run it:

puppet --debug --verbose test.pp

This will run through your code in test.pp and execute it. You should be aware that you couldn't fetch files from the master in this case since it's purely local but see the File types reference for a explanation of how the behavior changes - you can still copy files, just not from the master.

You can do anything that puppet code can do, make classes, do defines, make sub classes, install packages, this is great for testing out small concepts in a safe way. Everything you see in this article was done in my shell like this.

What is the value of a variable?
You've set a variable in some class but you're not sure if it's set to what you're expecting, maybe you don't know the scoping rules just yet or you just want to log some state back to the node or master log.

In a usual scripting language you'd add some debug prints, puppet is the same. You can print simple values in the master log by doing this in your manifest:

notice("The value is: ${yourvar}")

Now when you run your node you should see this being printed to the syslog (by default) on your master.

To log something to your client, do this:

notify{"The value is: ${yourvar}": }

Now your puppet logs - syslog usually - will show lines on the client or you could just do puppetd --test on the client to see it run and see your debug bits.

What is in an array?
You've made an array, maybe from some external function, you want to know what is in it? This really is an extension to the above hint that would print garbage when passed arrays.

Building on the example above and the fact that puppet loops using resources, lets make a simple defined type that prints each member of an array out to either the master or the client (use the technique above to choose)

$arr = [1, 2, 3]

define print() {
notice("The value is: '${name}'")
}

print{$arr: }

This will print one line for each member in the array - or just one if $arr isn't an array at all.

$ puppet test.pp
notice: Scope(Print[1]): The value is: '1'
notice: Scope(Print[3]): The value is: '3'
notice: Scope(Print[2]): The value is: '2'

Writing shell scripts with puppet?
Puppets a great little language, you might even want to replace some general shell scripts with puppet manifest code, here's a simple hello world:

#!/usr/bin/puppet

notice("Hello world from puppet!")
notice("This is the host ${fqdn}")

If we run this we get the predictable output:

$ ./test.pp
notice: Scope(Class[main]): Hello world from puppet!
notice: Scope(Class[main]): This is the host your.box.com

And note that you can access facts and everything from within this shell script language, really nifty!

Did I get the syntax right?
If I introduce a deliberate error in the code above - remove the last " - it would blow up, you can test puppet syntax using puppet itself:

$ puppet --parseonly test.pp
err: Could not parse for environment production: Unclosed quote after '' in 'Hello world from puppet!)
' at /home/rip/test.pp:1

You can combine this with a pre-commit hook on your SCM to make sure you don't check in bogus stuff.

How should I specify the package version? or user properties?
Often you've added a package, but not 100% sure how to pass the version string to ensure => or you're not sure how to specify password hashes etc, puppet comes with something called ralsh that can interrogate a running system, some samples below:

% ralsh package httpd
package { 'httpd':
    ensure => '2.2.3-22.el5.centos.2'
}

% ralsh package httpd.i386
package { 'httpd.i386':
    ensure => '2.2.3-22.el5.centos.2'
}

# ralsh user apache
user { 'apache':
    password => '!!',
    uid => '48',
    comment => 'Apache',
    home => '/var/www',
    gid => '48',
    ensure => 'present',
    shell => '/sbin/nologin'
}

Note in the 2nd case I ran it as root, puppet needs to be able to read shadow and so forth. The code it's outputting is valid puppet code that you can put in manifests.

Will Puppet destroy my machine?
Maybe you're just getting ready to run puppet on a host for the first time or you're testing some new code and you want to be sure nothing terrible will happen, puppetd has a no-op option to make it just print what will happen, just run puppetd --test --noop

What files etc are being managed by puppet?
See my previous post for a script that can tell you what puppet is managing on your machine, note this does not yet work on 0.25.x branch of code.

Is my config changes taking effect?
Often people make puppet.conf changes and it just isn't working, perhaps they put them in the wrong [section] in the config file, a simple way to test is to run puppetd --genconfig this will dump all the active configuration options.

Be careful though don't just dump this file over your puppet.conf thinking you'll have a nice commented file, this wont work as the option for genconfig will be set to true in the file and you'll end up with a broken configuration. In general I recommend keeping puppet.conf simple and short only showing the things you're changing away from defaults, that makes it much easier to see what differs from standard behavior when asking for help.

Getting further help
Puppet has a irc channel #puppet on freenode, we try and be helpful and welcoming to newcomers, pop by if you have questions.

As always you need to have these wiki pages bookmarked and they should be your daily companions:

Managing web traffic with ruby-pdns

2009-08-17T21:43:39Z

A short while ago I wrote about releasing a Ruby Development framework for PowerDNS the release is still early days, feature complete but needs some robustness tweaks and a new release will be out in a week or so to address that.

I wanted though to highlight some success that I've had using it. I have a small static farm for a client that handles around 2MiB/sec of 200x200 jpg files, this setup is for a startup so out of necessity its all built to be cheap, I host on networks I don't own yet I need pretty good control over it, what IPs will be used to serve traffic and so forth.

The graph above shows the case before caused by the windows DNS bug, you'll see the bottom host is working pretty hard getting a large chunk of the bandwidth.

This is a problem because come mid month this poor machine has already used up its allocation of 2.5TiB of transfer and I need to move it from the pool.

So my goal was to shift the traffic to the yellow and green machines and just generally balance things out a bit. I used the Weighted Round Robin feature of ruby-pdns to adjust the biases, it took a bit of fiddling because for some other reason even when this machine gets fewer requests per second it still seems to manage more in terms of bandwidth, this is the eventual code snippet:

        ips = ["213.x.x.232",     # dark blue
               "88.x.x.201",          # lighter blue
               "82.x.x.180",          # yellow
               "82.x.x.181"].randomize([1,2,2,3])   # green

        answer.ttl 300
        answer.shuffle false
        answer.content [:A, ips[0]]
        answer.content [:A, ips[1]]
        answer.content [:A, ips[2]]

The thresholds seems odd but that's what worked after some fiddling, see the graph below.

This is much nicer balanced, it's not perfect and I doubt I will get it perfect with just 4 machines to play with but I believe it's already at the point where it means I can use all my machines for the entire month without hitting any limits.

Here's another graph over the week showing things side by side:

The improvement is very obvious in this graph and you can see I've not lost anything in performance between first day and last day on the graph in terms of throughput (the lower days were days where lower traffic is expected).

If I look at my actual transfer used it's better balanced now, first lets see the 12th:

      08/12/09     12.67 GiB |   46.42 GiB |   59.09 GiB |    5.74 Mbit/s
      08/12/09      7.71 GiB |   21.32 GiB |   29.04 GiB |    2.82 Mbit/s
      08/12/09      9.05 GiB |   23.05 GiB |   32.10 GiB |    3.12 Mbit/s
      08/12/09      6.94 GiB |   16.56 GiB |   23.50 GiB |    2.28 Mbit/s

Again the skew is very clear with a 23GiB on the lowest compared to 59GiB on the highest use machine, on the 17th it looked a lot better:

      08/17/09      7.84 GiB |   28.55 GiB |   36.39 GiB |    3.53 Mbit/s
      08/17/09      8.46 GiB |   25.66 GiB |   34.12 GiB |    3.31 Mbit/s
      08/17/09     11.21 GiB |   30.70 GiB |   41.91 GiB |    4.07 Mbit/s
      08/17/09     10.25 GiB |   28.20 GiB |   38.46 GiB |    3.73 Mbit/s

Obviously much better when looking at the 2nd to last column. The first column is received the increase in those is down to a slightly lower hit ratio on the caching proxy on these machines meaning it's fetching more files from origin than the others.

Overall I am extremely pleased with this solution, I agree one should not be using DNS as a hammer to all your nails but for startups and cloud based people who do not have control over networks, BGP tables and so forth this really does represent a viable option to what would otherwise be an extremely expensive problem to solve.

News from SA

2009-08-13T21:05:56Z

So today I decided to follow @iol on my twitter client to try and keep in touch with things back home.

These are some of the messages I got in the first 11 hours:

Still no trace of missing baby http://bit.ly/cUafy
'I could not clear away images of the dead' http://bit.ly/NjQxq
Woman forgives parents for killing her lover http://bit.ly/xd2P7
'Mass hysteria' sweeps six E Cape schools http://bit.ly/yE30R
Bona magazine editor killed http://bit.ly/13JN08
Suspect shoots himself in groin http://bit.ly/29UNDM
Baby's body found in car boot http://bit.ly/349oq
Chabaan guilty of assault http://bit.ly/XpKZs
Hammer attack hit straight to teacher's core http://bit.ly/YbdD6
MJC rejects the slaughter of chickens claim http://bit.ly/23igtE
'I just can't wait till this is all over' http://bit.ly/2mWbg
Judge's death: Unanswered questions remain http://bit.ly/4aqPW
Slain editor: Family believes he knew killers http://bit.ly/anMLt

That all in 11 hours of headlines. Shocking.