Trigger Puppet runs though Git hooks

04/28/2012

Since shutting down my business I now run a small 25 node network with no Puppet Masters and I do not schedule regular Puppet runs – I run them just when needed.

Till now I’ve just done puppet runs via MCollective, basically I’d edit some puppet files and after comitting them just send off a puppet run with mcollective, supplying filters by hand so I only trigger runs on the appropriate nodes.

I started looking into git commit hooks to see if I can streamline this. I could of course just trigger a run on all nodes after a commit, there is no problem with capacity of masters etc to worry about. This is not very elegant so I thought I’d write something to parse my git push and trigger runs on just the right machines.

I’ll show a simplified version of the code here, the full version of the post-receive hook can be found here. I’ve removed the parse_hiera, parse_node and parse_modules functions from this but you can find them in the code linked to. To use this code you will need MCollective 2.0.0 that is due in a few days.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
#!/usr/bin/env ruby
 
require 'rubygems'
require 'grit'
require 'mcollective'
 
include MCollective::RPC
 
@matched_modules = []
@matched_nodes = []
@matched_facts = []
 
# read each git ref in the push and process them
while msg = gets
  old_sha, new_sha, ref = msg.split(' ', 3)
 
  repo = Grit::Repo.new(File.join(File.dirname(__FILE__), '..'))
 
  commit = repo.commit(new_sha)
 
  case ref
    when %r{^refs/heads/(.*)$}
      branch = $~[1]
      if branch == "master"
        puts "Commit on #{branch}"
        commit.diffs.each do |diff|
          puts "    %s" % diff.b_path
 
          # parse the paths and save them to the @matched_* arrays
          # these functions are in the full code paste linked to above
          case diff.b_path
            when /^hieradb/
              parse_hiera(diff.b_path)
            when /^nodes/
              parse_node(diff.b_path)
            when /^common\/modules/
              parse_modules(diff.b_path)
            else
              puts "ERROR: Do not know how to parse #{diff.b_path}"
          end
        end
      else
        puts "Commit on non master branch #{branch} ignoring"
      end
  end
end
 
unless @matched_modules.empty? && @matched_nodes.empty? && @matched_facts.empty?
  puppet = rpcclient("puppetd")
 
  nodes = []
  compound_filter = []
 
  nodes << @matched_nodes
 
  # if classes or facts are found then do a discover
  unless @matched_modules.empty? && @matched_facts.empty?
    compound_filter << @matched_modules << @matched_facts
 
    puppet.comound_filter compound_filter.flatten.uniq.join(" or ")
 
    nodes << puppet.discover
  end
 
  if nodes.flatten.uniq.empty?
    puts "No nodes discovered via mcollective or in commits"
    exit
  end
 
  # use new mc 2.0.0 pluggable discovery to supply node list
  # thats a combination of data discovered on the network and file named
  puppet.discover :nodes => nodes.flatten.uniq
 
  puts
  puts "Files matched classes: %s" % @matched_modules.join(", ") unless @matched_modules.empty?
  puts "Files matched nodes: %s" % @matched_nodes.join(", ") unless @matched_nodes.empty?
  puts "Files matched facts: %s" % @matched_facts.join(", ") unless @matched_facts.empty?
  puts
  puts "Triggering puppet runs on the following nodes:"
  puts
  puppet.discover.in_groups_of(3) do |nodes|
    puts "   %-20s %-20s %-20s" % nodes
  end
 
  puppet.runonce
 
  printrpcstats
else
  puts "ERROR: Could not determine a list of nodes to run"
end

The code between lines 14 and 46 just reads each line of the git post-receive hook STDIN and process them, you can read more about these hooks @ git-scm.com.

For each b path in the commit I parse its path based on puppet module conventions, node names, my hiera structure and some specific aspects of my file layouts. These end up in the @matched_modules, @matched_nodes and @matched_facts arrays.

MCollective 2.0.0 will let you supply node names not just from network based discovery but from any source really. Here I get node names from things like my node files, file names in iptables rules and such. Version 2.0.0 also supports a new query language for discovery which we use here. The goal is to do a network discovery only when I have non specific data like class names – if I found just a list of node names I do not need to do go out to the network to do discovery thanks to the new abilities of MCollective 2.0.0

In lines 48 to 90 I create a MCollective client to the puppetd agent, discover matching nodes and do the puppet runs.

If I found any code in the git push that matched either classes or facts I need to do a full MCollective discover based on those to get a node list. This is done using the new compound filtering language, the filter will look something like:

/some_class/ or some::other::class or fact=value

But this expensive network wide discovery is only run when there are facts or classes matched out of the commit.

Line 72 will supply the combined MCollective discovered nodes and node names discovered out of the code paths as discovery data which later in line 85 will get used to trigger the runs.

The end result of this can be seen here, the commit matched only 5 out of my 25 machines and only those will be run:

$ git push origin master
Counting objects: 13, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (7/7), 577 bytes, done.
Total 7 (delta 4), reused 0 (delta 0)
remote: Commit on master
remote:     common/modules/mcollective/manifests/client.pp
remote:
remote: Files matched classes: mcollective::client
remote:
remote: Triggering puppet runs on the following nodes:
remote:
remote:    node1                node2            node3
remote:    node4                node5
remote:
remote: 5 / 5
remote:
remote: Finished processing 5 / 5 hosts in 522.15 ms
To git@git:puppet.git
   7590a60..10ee4da  master -> master