It basically has these features:<\/p>\n
- \n
- Discover the list of nodes to manage based on a supplied filter, I have regional masters so I will manage groups of Puppet nodes independently<\/li>\n
- Evenly spreads out the Puppet runs over an interval, if I have 10 nodes and a 30 minute interval I will get a run every 3 minutes.<\/li>\n
- Nodes run at a predictable time every time, even after reboots since the node list is just run through alphabetically. If the node list stays constant you’ll always run at the same time give or take 10 seconds. If nodes get added the behavior will be predictable.\n
- Before scheduling a run it checks the overall concurrency of Puppet runs, if it exceeds a limit it will skip a background run. I want to give priority to runs that I run by hand with –test, this ensures that happens.<\/li>\n
- If the client it is about to run ran its Catalog recently – maybe via –test – it will skip that run<\/li>\n<\/ul>\n
The result is pretty good, spreading 6 nodes out over 30 minutes I get a nice even CPU spread, the spike in the graph after the change is when the node itself runs Puppet. The 2nd graph is eth0 network output, the dip is when localhost is running:<\/p>\n
![](\"http:\/\/www.devco.net\/images\/spread_puppet.png\")
<\/center><\/p>\n![](\"http:\/\/www.devco.net\/images\/spread_puppet_network.png\")
<\/center><\/p>\nThe resulting CPU usage is much smoother, there aren’t periods of no CPU usage and there are no spikes caused by nodes bunching up together.<\/p>\n
Below output from a C&C session managing 3 machines with an interval of 1 minute and a max concurrency of 1, these machines were still running cron based puppetd so you can see the C&C is not scheduling runs when it hits the concurrency limit due to cron runs:<\/p>\n
<\/p>\n\r\n$ puppetcommander.rb --interval 1 -W \/dev_server\/ --max-concurrent 1\r\nWed Mar 17 08:31:29 +0000 2010> Looping clients with an interval of 1 minute(s)\r\nWed Mar 17 08:31:29 +0000 2010> Restricting to 1 concurrent puppet run(s)\r\nWed Mar 17 08:31:31 +0000 2010> Found 3 puppet nodes, sleeping for ~20 seconds between runs\r\nWed Mar 17 08:31:31 +0000 2010> Current puppetds running: 1\r\nWed Mar 17 08:31:31 +0000 2010> Puppet run for client dev1.my.net skipped due to current concurrency of 1\r\nWed Mar 17 08:31:31 +0000 2010> Sleeping for 20 seconds\r\nWed Mar 17 08:31:51 +0000 2010> Current puppetds running: 1\r\nWed Mar 17 08:31:51 +0000 2010> Puppet run for client dev2.my.net skipped due to current concurrency of 1\r\nWed Mar 17 08:31:51 +0000 2010> Sleeping for 20 seconds\r\nWed Mar 17 08:32:12 +0000 2010> Current puppetds running: 0\r\nWed Mar 17 08:32:12 +0000 2010> Running agent for dev3.my.net\r\nWed Mar 17 08:32:15 +0000 2010> Sleeping for 16 seconds\r\n<\/pre>\n
<\/code><\/p>\n
There are many advantages to this approach over some other that’s been suggested: <\/p>\n
- \n
- No host lists to maintain, it reconfigures itself dynamically on demand. <\/li>\n
- It doesn’t rely on some other on-master fact like signed certificates that breaks models where the CA is separate<\/li>\n
- It doesn’t rely on stored configs that doesn’t work well at scale or on a setup with many regional masters. <\/li>\n
- It doesn’t suffer from issues if a node isn’t available but it’s in your host lists. <\/li>\n
- It understands the state of the entire platform and so you can control concurrency and therefore resources on your master.<\/li>\n
- It’s easy to extend with our own logic or demands, the current version of the code is only 90 lines of Ruby including CLI options parsing.<\/li>\n
- Concurrency control can mitigate other problems. Have a cluster of 10 nodes, don’t want your config change to restart them all at the same time, no problem. Just make sure you only run 2 a time.\n<\/ul>\n
In reality this means I can remove 256MB RAM from my master – since I can now run fewer puppetmasterd processes, this will save me $15\/month hosting fee on this specific master, it’s small change but always good to control my platform costs.<\/p>\n","protected":false},"excerpt":{"rendered":"
Scheduling Puppet runs is a hard problem, you either run the daemon or run it through cron, both have drawbacks. There’s been some discussion about decoupling this or to improve the remote control abilities of Puppet, this is my entry into that discussion. Running the daemon it’s all about the memory problems of pretty much […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","footnotes":""},"categories":[1],"tags":[85,78,21],"_links":{"self":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/1338"}],"collection":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/comments?post=1338"}],"version-history":[{"count":19,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/1338\/revisions"}],"predecessor-version":[{"id":1704,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/1338\/revisions\/1704"}],"wp:attachment":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/media?parent=1338"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/categories?post=1338"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/tags?post=1338"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}