Archive | Uncategorized RSS feed for this section

Effective adhoc commands in clusters

by R.I. Pienaar on 28. Aug, 2010 in Uncategorized

Last night I had a bit of a mental dump on twitter about structured data and non structured data when communicating with a cluster or servers – Twitter fails at this kind of stuff so figured I’ll follow up with a blog post.

I started off asking for a list of tools in the cluster admin space and got some great pointers which I am reproducing here:

fabric, cap, func, clusterssh, sshpt, pssh, massh, clustershell, controltier, rash (related), dsh, chef knife ssh, pdsh+dshbak and of course mcollective. I was also sent a list of ssh related tools which is awesome.

The point I feel needs to be made is that in general these tools just run commands on remote servers. They are not aware of the commands output structure, what denotes pass or fail in the context of the command etc. Basically the commands people run are commands designed for ages to be looked at by human eyes and then parsed by a human mind. Yes they are easy to pipe and grep and chop up, but ultimately it was always designed to be run on one server at a time.

The parallel ssh’ers run these commands in parallel and you tend to get a mash of output. The output is mixed STDOUT and STDERR and often output from different machines are multiplexed into each other so you get a stream of text that is hard to decipher even on 2 machines, not to mention 200 at once.

Take as an example a simple yum command to install a package:

% yum install zsh
Loaded plugins: fastestmirror, priorities, protectbase, security
Loading mirror speeds from cached hostfile
372 packages excluded due to repository priority protections
0 packages excluded due to repository protections
Setting up Install Process
Package zsh-4.2.6-3.el5.i386 already installed and latest version
Nothing to do

When run on one machine you pretty much immediately know whats going on, package was already there so nothing got done, now lets see cap invoke:

# cap invoke COMMAND="yum -y install zsh"
  * executing `invoke'
  * executing "yum -y install zsh"
    servers: ["web1", "web2", "web3"]
    [web2] executing command
    [web1] executing command
    [web3] executing command
 ** [out :: web2] Loaded plugins: fastestmirror, priorities, protectbase, security
 ** [out :: web2] Loading mirror speeds from cached hostfile
 ** [out :: web3] Loaded plugins: fastestmirror, priorities, protectbase
 ** [out :: web3] Loading mirror speeds from cached hostfile
 ** [out :: web3] 495 packages excluded due to repository priority protections
 ** [out :: web2] 495 packages excluded due to repository priority protections
 ** [out :: web3] 0 packages excluded due to repository protections
 ** [out :: web3] Setting up Install Process
 ** [out :: web2] 0 packages excluded due to repository protections
 ** [out :: web2] Setting up Install Process
 ** [out :: web1] Loaded plugins: fastestmirror, priorities, protectbase
 ** [out :: web3] Package zsh-4.2.6-3.el5.x86_64 already installed and latest version
 ** [out :: web3] Nothing to do
 ** [out :: web1] Loading mirror speeds from cached hostfile
 ** [out :: web1] Install       1 Package(s)
 ** [out :: web2] Package zsh-4.2.6-3.el5.x86_64 already installed and latest version
 ** [out :: web2] Nothing to do
 ** [out :: web1] 548 packages excluded due to repository priority protections
 ** [out :: web1] 0 packages excluded due to repository protections
 ** [out :: web1] Setting up Install Process
 ** [out :: web1] Resolving Dependencies
 ** [out :: web1] --> Running transaction check
 ** [out :: web1] ---> Package zsh.x86_64 0:4.2.6-3.el5 set to be updated
 ** [out :: web1] --> Finished Dependency Resolution
 ** [out :: web1]
 ** [out :: web1] Dependencies Resolved
 ** [out :: web1]
 ** [out :: web1] ================================================================================
 ** [out :: web1] Package      Arch            Version                Repository            Size
 ** [out :: web1] ================================================================================
 ** [out :: web1] Installing:
 ** [out :: web1] zsh          x86_64          4.2.6-3.el5            centos-base          1.7 M
 ** [out :: web1]
 ** [out :: web1] Transaction Summary
 ** [out :: web1] ================================================================================
 ** [out :: web1] Install       1 Package(s)
 ** [out :: web1] Upgrade       0 Package(s)
 ** [out :: web1]
 ** [out :: web1] Total download size: 1.7 M
 ** [out :: web1] Downloading Packages:
 ** [out :: web1] Running rpm_check_debug
 ** [out :: web1] Running Transaction Test
 ** [out :: web1] Finished Transaction Test
 ** [out :: web1] Transaction Test Succeeded
 ** [out :: web1] Running Transaction
 ** [out :: web1] Installing     : zsh                                                      1/1
 ** [out :: web1]
 ** [out :: web1]
 ** [out :: web1] Installed:
 ** [out :: web1] zsh.x86_64 0:4.2.6-3.el5
 ** [out :: web1]
 ** [out :: web1] Complete!
    command finished
zlib(finalizer): the stream was freed prematurely.
zlib(finalizer): the stream was freed prematurely.
zlib(finalizer): the stream was freed prematurely.

Most of this stuff scrolled off my screen and at the end all I had was the last bit of output. I could scroll up and still figure out ok what was going on – 2 of the 3 already had it installed, one got it. Now imagine 100 or 500 of these machines output all mixed in? Just parsing this output would be prone to human error and you’re likely to miss that something failed.

So here is my point, your cluster management tool need to provide an API around the every day commands like packages, process listing etc. It should return structured data and you could use the structured data to create tools more fit for the purpose of using on large amount of machines. Being that the output is standardized it should provide generic tools that just do the right thing out of the box for you.

With the package example above knowing that all 500 machines had spewed out a bunch of stuff while installing isn’t important, you just want to know the result in a nice way. Here’s what mcollective does:

$ mc-package install zsh
 
 * [ ============================================================> ] 3 / 3
 
web2.my.net                      version = zsh-4.2.6-3.el5
web3.my.net                      version = zsh-4.2.6-3.el5
web1.my.net                      version = zsh-4.2.6-3.el5
 
---- package agent summary ----
           Nodes: 3 / 3
        Versions: 3 * 4.2.6-3.el5
    Elapsed Time: 16.33 s

In the case of a package you want to just know the version post the event and a summary of status. Just by looking at the stats I know the desired result was achieved, if I had different versions listed I could very quickly identify the problem ones.

Here’s another example – NRPE this time:

% mc-rpc nrpe runcommand command=check_disks
 
 * [ ============================================================> ] 47 / 47
 
 
dev1.my.net                      Request Aborted
   CRITICAL
          Exit Code: 2
   Performance Data:  /=4111MB;3706;3924;0;4361 /boot=26MB;83;88;0;98 /dev/shm=0MB;217;230;0;256
             Output: DISK CRITICAL - free space: / 24 MB (0% inode=86%);
 
 
Finished processing 47 / 47 hosts in 766.11 ms

Here notice I didn’t use a NRPE specific mc- command, I just used the generic rpc caller and the caller knows that I am only interesting in seeing the results of machines that are in WARNING or CRITICAL state. If you run this on your console you’d see the ‘Request Aborted’ would be red and the ‘CRITICAL’ would be yellow. Immediately pulling your eye to the important information. Also note how the result shows human friendly field names like ‘Performance Data’.

The formatting, highlighting, knowledge to only show failing resources and human friendly headings all happen automatically, no programming of client side UI is required you get the ability to do this for free simply from the fact that mcollective focuses on putting structure around outputs.

Here’s the earlier package install example with the standard rpc caller not with a specialized package frontend:

% mc-rpc package install package=zsh
Determining the amount of hosts matching filter for 2 seconds .... 47
 
 * [ ============================================================> ] 47 / 47
 
Finished processing 47 / 47 hosts in 2346.05 ms

Everything worked, all 47 machines have the package installed and your desired action was taken. So no point in spamming you with pages of junk, who cares to see all the Yum output? Had an install failed you’d have had usable error message just for the host that failed. The output would be equally usable on one or a thousand hosts with very little margin for human error in knowing the result of your request.

This happens because mcollective has a standard structure of responses, each response has a absolute success value that tells you if the request failed or not and by using this you can get generic CLI, Web, etc tools that displays large amounts of data from a network of hosts in a way that is appropriate and context aware.

For reference here’s the response as received on the client:

{:sender=>"dev1.my.net",
 :statuscode=>1,
 :statusmsg=>"CRITICAL",
 :data=>
  {:perfdata=>
    " /=4111MB;3706;3924;0;4361 /boot=26MB;83;88;0;98 /dev/shm=0MB;217;230;0;256",
   :output=>"DISK CRITICAL - free space: / 24 MB (0% inode=86%);",
   :exitcode=>2}}

Only by thinking about CLI and admin tasks in this way do I believe we can take the Unix utilities that we call on remote hosts and turn them into something appropriate for large scale parallel use that doesn’t overwhelm the human at the other end with information. Additionally since this is an API that is computer friendly it makes those tools usable in many other places like code deployers – for example to enable your continues deployment using robust use of unix tools via such an API.

There are many other advantages to this approach. Requests are authorized on a very fine level, requests are audited. API wrappers are code that’s versioned, that can be tested in development and makes the margin for error much smaller than just running random unix commands ad hoc. Finally if you’re using the code on a CLI ad-hoc as above or in your continues deployer you share the same code that you’ve already tested and trust.

Read full story • Comments { 4 }

Marionette Collective version 0.4.8

by R.I. Pienaar on 20. Aug, 2010 in Uncategorized

I just released version 0.4.8 of mcollective. It’s a small maintenance release fixing a few bugs and adding a few features. I wasn’t planning on another 0.4.x release before the big 1.0.0 but want to keep 1.0.0 close as possible to something that’s been out there for a while.

The only major feature it introduces is custom reports of your infrastructure.

It supports two types of scriptlet for building reports. The first is a little DSL that uses printf style format strings:

inventory do
    format "%s:\t\t%s\t\t%s"
 
    fields { [ identity, facts["serialnumber"], facts["productname"] ] }
end

Which does something like this:

$ mc-inventory --script hardware.mc
web1:           KKxxx1H         IBM eServer BladeCenter HS20 -[8832M1X]-
rep1:           KKxxx5Z         IBM eServer BladeCenter HS20 -[8832M1X]-
db4:            KDxxxZY         IBM System x3655 -[794334G]-
man2:           KDxxxR0         eserver xSeries 336 -[88372CY]-
db2:            KDxxxGD         IBM System x3655 -[79855AG]-

The other – perhaps more ugly – is using a Perl like format method. To use this you need the formatr gem installed, and a report might look like this:

formatted_inventory do
    page_length 20
 
    page_heading <<TOP
 
            Node Report @<<<<<<<<<<<<<<<<<<<<<<<<<
                        time
 
Hostname:         Customer:     Distribution:
-------------------------------------------------------------------------
TOP
 
    page_body <<BODY
 
@<<<<<<<<<<<<<<<< @<<<<<<<<<<<< @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
identity,    facts["customer"], facts["lsbdistdescription"]
                                @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                                facts["processor0"]
BODY
end

And the resulting report is something like this:

$ mc-inventory --script hardware.mc
            Node Report Fri Aug 20 21:49:39 +0100
 
Hostname:         Customer:     Distribution:
-------------------------------------------------------------------------
 
web1              rip           CentOS release 5.5 (Final)
                                Intel(R) Xeon(R) CPU           L5420  
 
web2              xxxxxxx       CentOS release 5.5 (Final)
                                Intel(R) Xeon(R) CPU           X3430

The report will be paged 20 nodes per page. The result is very pleasing even if the report format is a bit grim, but it would be much worse to write yet another reporting DSL!

See the full release notes for details on bug fixes and other features.

Read full story • Comments { 0 }
Making machine metadata visible

Making machine metadata visible

by R.I. Pienaar on 29. Jul, 2010 in Uncategorized

I’m quite the fan of data, metadata and querying these to interact with my infrastructure rather than interacting by hostnames and wanted to show how far I am down this route.

This is more an iterative ongoing process than a fully baked idea at this point since the concept of hostnames is so heavily embedded in our Sysadmin culture. Today I can’t yet fully break away from it due to tools like nagios etc still relying heavily on the hostname as the index but these are things that will improve in time.

The background is that in the old days we attempted to capture a lot of metadata in hostnames, domain names and so forth. This was kind of OK since we had static networks with relatively small amounts of hosts. Today we do ever more complex work on our servers and we have more and more servers. The advent of cloud computing has also brought with it a whole new pain of unpredictable hostnames, rapidly changing infrastructures a much bigger emphasis on role based computing.

My metadata about my machines comes from 3 main sources:

  • My Puppet manifests – classes and modules that gets put on a machine
  • Facter facts with the ability to add many per machine easily
  • MCollective stores the meta data in a MongoDB and let me query the network in real time

Puppet manifests based on query

When setting up machines I keep some data like database master hostnames in extlookup but in many cases I am now moving to a search based approach to finding resources. Here’s a sample manifest that will find the master database for a customers development machines:

$masterdb = search_nodes("{'facts.customer': '${customer}', 'facts.environment':${environment}, classes: 'mysql::master'}")

This is MongoDB query against my infrastructure database, it will find for a given node the name of a node that has the class mysql::master on it, by convention there should be only one per customer in my case. When using it in a template I can get back full objects with all the meta data for a node. Hopefully with Puppet 2.6 I can get full hashes into puppet too!

Making Metadata Visible

With machines doing a lot of work, filling a lot of roles etc and with more and more machines you need to be able to tell immediately what machine you are on.

I do this in several places, first my MOTD can look something like this:

   Welcome to Synchronize Your Dogmas 
            hosted at Hetzner, Germany
 
        Puppet Modules:
                - apache
                - iptables
                - mcollective member
                - xen dom0 skeleton
                - mw1.xxx.net virtual machine

I build this up using snippet from my concat module, each important module like apache can just put something like this in:

motd::register{"Apache Web Server": }

Being managed by my snippet library, if you just remove the include line from the manifests the MOTD will automatically update.

With a big block of welcome done, I now need to also be able to show in my prompts what a machine does, who its for a importantly what environment it is in.

Above a shot of 2 prompts in different environments, you see customer name, environment and major modules. Like with the motd I have a prompt::register define that module use to register into the prompt.

SSH Based on Metadata

With all this meta data in place, mcollective rolled out and everything integrated it’s very easy to now find and access machines based on this.

MCollective does real time resource discovery, so keeping with the mysql example above from puppet:

$ mc-ssh -W "environment=development customer=acme mysql::master"
Running: ssh db1.acme.net
Last login: Thu Jul 29 00:22:58 2010 from xxxx

$

Here i am ssh’ing to a server based on a query, if it found more than one machine matching the query a menu would be presented offering me a choice.

Monitoring Based on Metatdata

Finally setting up monitoring and keeping it in sync with reality can be a big challenge especially in dynamic cloud based environments, again I deal with this through discovery based on meta data:

$ check-mc-nrpe -W "environment=development customer=acme mysql::master"  check_load
check_load: OK: 1 WARNING: 0 CRITICAL: 0 UNKNOWN: 0|total=1 ok=1 warn=0 crit=0 unknown=0 checktime=0.612054

Summary

This is really the tip of the ice berg, there is a lot more that I already do – like scheduling puppet runs on groups of machines based on metadata – but also a lot more to do this really is early days down this route. I am very keen to get views from others who are struggling with shortcomings in hostname based approaches and how they deal with it.

Read full story • Comments { 0 }

MCollective Components, Terminology and Flow

by R.I. Pienaar on 28. Jul, 2010 in Uncategorized

I often see some confusion about terminology in use in MCollective, what the major components are, where software needs to be installed etc.

I attempted to address this in a presentation and screen cast covering:

  • What middleware is and how we use it.
  • The major components and correct terminology.
  • Anatomy of a request life cycle.
  • And an actual look inside the messages we sent and receive.

You can grab the presentation from Slideshare or view a video of it on blip.tv. Below find an embedded version of the slideshare deck including audio. I suggest you view it full screen as there’s some code in it.

Architecture, Flow and Terminology

Read full story • Comments { 0 }

EC2 Bootstrap Helper

by R.I. Pienaar on 12. Jul, 2010 in Uncategorized

I’ve been working a bit on streamlining the builds I do on EC2 and wanted a better way to provision my machines. I use CentOS and things are pretty rough to non existent for nicely built EC2 images. I’ve used the Rightscale ones till now and while they’re nice they are also full of lots of code copyrighted by Rightscale.

What I really wanted was something as full featured as Ubuntu’s CloudInit but also didn’t feel much like touching any Python. I hacked up something that more or less do what I need. You can get it on GitHub. It’s written and tested on CentOS 5.5.

The idea is that you’ll have a single multi purpose AMI that you can easily bootstrap onto your puppet/mcollective infrastructure using this system. Below for some details.

I prepare my base CentOS AMI with the following mods:

  • Install Facter and Puppet – but not enabled
  • Install the EC2 utilities
  • Setup the usual getsshkeys script
  • Install the ec2-boot-init RPM
  • Add a custom fact that reads /etc/facts.txt – see later why. Get one here.

With this in place you need to create some ruby scripts that you will use to bootstrap your machines. Examples of this would be to install mcollective, configure it to find your current activemq. Or to set up puppet and do your initial run etc.

We host these scripts on any webserver – ideally S3 – so that when a machine boots it can grab the logic you want to execute on it. This way you can bug fix your bootstrapping without having to make new AMIs as well as add new bootstrap methods in future to existing AMIs.

Here’s a simple example that just runs a shell command:

newaction("shell") do |cmd, ud, md, config|
    if cmd.include?(:command)
        system(cmd[:command])
    end
end

You want to host this on any webserver in a file called shell.rb. Now create a file list.txt in the same location that just have this:

shell.rb

You can list as many scripts as you want. Now when you boot your instance pass it data like this:

--- 
:facts: 
  role: webserver
:actions: 
- :url: http://your.net/path/to/actions/list.txt
  :type: :getactions
- :type: :shell
  :command: date > /tmp/test

The above will fetch the list of actions – our shell.rb – from http://your.net/path/to/actions/list.txt and then run using the shell action the command date > /tmp/test. The actions are run in order so you probably always want getactions to happen first.

Other actions that this script will take:

  • Cache all the user and meta data in /var/spool/ec2boot
  • Create /etc/facts.txt with all your facts that you passed in as well as a flat version of the entire instance meta data.
  • Create a MOTD that shows some key data like AMI ID, Zone, Public and Private hostnames

The boot library provides a few helpers that help you write scripts for this environment specifically around fetching files and logging:

    ["rubygems-1.3.1-1.el5.noarch.rpm",
     "rubygem-stomp-1.1.6-1.el5.noarch.rpm",
     "mcollective-common-#{version}.el5.noarch.rpm",
     "mcollective-#{version}.el5.noarch.rpm",
     "server.cfg.templ"].each do |pkg|
        EC2Boot::Util.log("Fetching pkg #{pkg}")
        EC2Boot::Util.get_url("http://foo.s3.amazonaws.com/#{pkg}", "/mnt/#{pkg}")
     end

This code fetches a bunch of files from a S3 bucket and save them into /mnt. Each one gets logged to console and syslog. Using this GET helper has the advantage that it has sane retrying etc built in for you already.

It’s fairly early days for this code but it works and I am using it, I’ll probably be adding a few more features soon, let me know in comments if you need anything specific or even if you find it useful.

Read full story • Comments { 0 }

MCollective Data Definition Language

by R.I. Pienaar on 25. Jun, 2010 in Uncategorized

I mentioned in my recent post about mcollective Road Map about the DDL.

The DDL is used to describe agents in a way that is accessible by other programs, web applications, client libraries and so forth to help those various client tools to configure themselves correctly.

An actual example of a DDL file can be found here if you want to have a good look at it and full docs here.

I’ve created a short video showing the DDL and some of the features of the upcoming 0.4.7 release, you probably want to view it full screen to really see what’s going on.

And a quick note about the colors, I know people tend to feel strongly about this kind of thing, you can disable them in the config file of the client :)

This is also my first attempt at using blip.tv, please let me know if you see any problems.

Read full story • Comments { 0 }

MCollective pgrep

by R.I. Pienaar on 23. May, 2010 in Uncategorized

The unix pgrep utility is great, it lets you grep through your process list and find interesting things. I wanted to do something similar but for my entire server group so built something quick ontop of MCollective.

I am using the Ruby sys-proctable gem to do the hard work, it returns a massive amount of information about each process and have written a simple agent on top of this.

The agent supports grepping the process tree but also supports kill and pgre+kill though I have not yet implemented more than the basic grep on the command line. Frankly the grep+kill combination scares me and I might remove it. A simple grep slipup and you will kill all processes on all your machine :) Sometimes too much power is too much and should just be avoided.

At the moment mc-pgrep outputs a set format but I intend to make that configurable on the command line, here’s a sample:

% mc-pgrep -C /dev_server/ ruby
 
 * [ ============================================================> ] 4 / 4
 
dev1.my.com
       root   9833  ruby /usr/sbin/mcollectived --pid=/var/run/mcollectived.pid 
       root  21608  /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_pass
 
dev2.my.com
       root  14568  /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_pass
       root  31595  ruby /usr/sbin/mcollectived --pid=/var/run/mcollectived.pid 
 
dev3.my.com
       root   1620  /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_pass
       root  14093  ruby /usr/sbin/mcollectived --pid=/var/run/mcollectived.pid 
 
dev4.my.com
       root   3231  /usr/lib/ruby/gems/1.8/gems/passenger-2.2.2/lib/phusion_pass
       root  20557  ruby /usr/sbin/mcollectived --pid=/var/run/mcollectived.pid 
 
   ---- process list stats ----
        Matched hosts: 4
    Matched processes: 8
        Resident Size: 37.264KB
         Virtual Size: 629.578MB

You can also limit it to only find zombies with the -z option.

This has been quite interesting for me, if I limit the pgrep to “.” (the pattern is regex) every machine will send back a Sys::ProcTable hash for all its processes. This is a 50 to 70 KByte payload per server. I’ve so far seen no problem getting his much traffic through ActiveMQ + MCollective and processing it all in a very short time:

% time mc-pgrep -F "country=/uk|us/" .
 
   ---- process list stats ----
        Matched hosts: 20
    Matched processes: 1958
        Resident Size: 1.777MB
         Virtual Size: 60.072GB
 
mc-pgrep -F "country=/uk|us/" .  0.19s user 0.06s system 7% cpu 3.420 total

That 3.4 seconds is with a 2 second discovery overhead client machine in Germany and the filter matching UK and US machines – all the way to the West Coast – my biggest delay here is network and not MC or ActiveMQ.

The code can be found at my GitHub account and still a bit of a work in progress, wiki pages will follow once I am happy with it.

And as an aside, I am slowly migrating at least my code to GitHub if not wiki and ticketing. So far my Plugins have moved, MC will move soon too.

Read full story • Comments { 0 }

DKIM with CentOS 5 and Exim

by R.I. Pienaar on 14. May, 2010 in Uncategorized

DomainKeys Identified Mail – DKIM – is a recent attempt to add some sender verification to email. Read more here, here and in the RFC 4871 to get some background info.

If you’re sending any newsletters you really want to be investigating this, if you’re doing anti spam it’s good to start looking at tracking this and really everyone should have DKIM on their domains. Exim recently – as of 4.70 – have decent support for it but CentOS is still on 4.63 thanks to RHEL.

To get a new Exim on your CentOS machine I suggest just using ATrpms who as of writing has 4.71 packages available for Exim and the other bits you need. I needed:

exim-4.71-40.el5.i386.rpm
exim-mysql-4.71-40.el5.i386.rpm
libspf2_2-1.2.5-5.0.el5.i386.rpm
libsrs_alt1-1.0-3_rc1.0.el5.i386.rpm

As well as the 64bit versions, you can just add ATrpms to your systems but really you should have your own repos and carefully control the packages that goes out to your estate.

Once you have upgraded your stock Exim to these versions – it’s a totally clean and compatible upgrade – configuring Exim to automagically sign outgoing mail with DKIM is pretty easy. We’ll make it so it looks for keys in a specific location based on outgoing mail domain so if you’re a relay for many domains you just need to drop down the certs.

Put the following near the top of our /etc/exim/exim.conf file, this just sets some macros we’ll use later on:

DKIM_DOMAIN = ${lc:${domain:$h_from:}}
DKIM_FILE = /etc/exim/dkim/${lc:${domain:$h_from:}}.pem
DKIM_PRIVATE_KEY = ${if exists{DKIM_FILE}{DKIM_FILE}{0}}

This will use, based on sender domain, a private key in /etc/exim/dkim/sender_domain.pem. By default exim just logs DKIM verification, doesn’t block any incoming mail I won’t cover doing blocks here just sending.

Next find your remote_smtp transport later in the file and change it to look like this:

remote_smtp:
  driver = smtp
  dkim_domain = DKIM_DOMAIN
  dkim_selector = x
  dkim_private_key = DKIM_PRIVATE_KEY
  dkim_canon = relaxed
  dkim_strict = 0

This will make Exim do the DKIM signing on outgoing mail but only if it can find a certificate.

To make the certificates is pretty easy, we’ll use a domain example.com:

$ mkdir /etc/exim/dkim/ && cd /etc/exim/dkim/
$ openssl genrsa -out example.com.pem 768 
$ openssl rsa -in example.com.pem -out example.com-public.pem -pubout -outform PEM

All that’s left now is to update your dns, sticking to example.com you’d add something like this into your bind zone file the text to add after p= is the stuff you’ll find in the public key called example.com-public.pem in our example:

x._domainkey     IN      TXT     "v=DKIM1\; t=y\; k=rsa\; p=MIGfMA0<snip>AQAB"
_domainkey       IN      TXT     "t=y\; o=~\;"

The x matches up with your dkim_selector in the SMTP transport above. The t=y tells the world you’re still testing your setup so remove that only when you’re all 100% certain it works. The o=~ tells everyone you will sign only some mail. You can make that o=- if all mail from you would be signed.

You can verify your DNS is right like this:

$ dig +short txt x._domainkey.example.com
"v=DKIM1\; k=rsa\; p=MIGfMA0<snip>AQAB"

And finally if you’re sending mail you should now see a header in the mail like this:

DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=example.com; s=x;
	h=From:To:Message-Id:Date; bh=g3zLY<snip>5uGs=; b=fonAB<snip>bceHhQ==;

Finally you can send an email to check-auth@verifier.port25.com and it will reply with all sorts of test output about your domain including DKIM validation details.

Read full story • Comments { 3 }

Puppet Concat 20100507

by R.I. Pienaar on 07. May, 2010 in Uncategorized

I’ve had quite a lot of contributions to my Puppet Concat module and after some testing by various people I’m ready to do a new release.

Thanks to Paul Elliot, Chad Netzer and David Schmitt for patches and assistance.

For background of what this is about please see my earlier post: Building files from fragments with Puppet

You can download the release here. Please pay special attention to the upgrade instructions below.

Changes in this release

  • Several robustness improvements to the helper shell script.
  • Removed all hard coded paths in the helper script to improve portability.
  • We now use file{} to copy the combined file to its location. This means you can now change the ownership of a file by just changing the owner/group in concat{}.
  • You can specify ensure => “/some/other/file” in concat::fragment to include the contents of another file in the fragment. Even files not managed by puppet.
  • The code is now hosted on Github and we’ll accept patches there.

Upgrading

When upgrading to this version you need to take particular care. All the fragments are now owned by root, the shell script runs as root and we use file{} to copy the resulting file out.

This means you’ll see the diff of not just the fragments but also the final file when running puppetd –test but unfortunately it also means the first time you run puppet with the new code your Puppet will fire off all notifies that you have on your concat{} resources. You’ll also see a lot of changes to resources in the fragments directory on first run. This is normal and expected behavior.

So if say you’re using the concat to create my.cf and notify the service to restart automatically then simply upgrading this module will result in MySQL restarting. This is a one off notify that happens only the first time, from then on it will be as normal. So I’d suggest when upgrading to disable those notifies till this upgrade is running everywhere and then put it back.

Read full story • Comments { 1 }

London DevOps meet 28/04/2010

by R.I. Pienaar on 14. Apr, 2010 in Uncategorized

I have finalized speakers for the next London DevOps get together, I sent the mail below to the list, looking forward to seeing everyone there!

Hello,

I am glad to announce speakers for our first meet hosted by The Guardian.

We will meet at their shiny new offices in Kings Cross to start at 7pm, those who went to
Scale Camp will know the venue.

We have two talks of roughly 30 minutes each lined up:

We will have some time for a few lightning talks if there’s any interest before retiring to a nearby pub. If anyone has Pub suggestions please send them along.

Map and details can be found as usual at http://londondevops.org/meetings/

Thanks again to The Guardian for the venue, if anyone out there want to sponsor some sodas or something for at the venue please get in contact.

I will try to set up some RSVP system, if you mention the meet on twitter please use the #ldndevops hashtag!

Read full story • Comments { 3 }
Older Entries »