{"id":1356,"date":"2010-03-27T17:16:38","date_gmt":"2010-03-27T16:16:38","guid":{"rendered":"http:\/\/www.devco.net\/?p=1356"},"modified":"2010-08-17T12:14:36","modified_gmt":"2010-08-17T11:14:36","slug":"infrastructure_testing_with_mcollective_and_cucumber","status":"publish","type":"post","link":"https:\/\/www.devco.net\/archives\/2010\/03\/27\/infrastructure_testing_with_mcollective_and_cucumber.php","title":{"rendered":"Infrastructure testing with MCollective and Cucumber"},"content":{"rendered":"<p>Some time ago I showed some sample code I had for <a href=\"http:\/\/www.devco.net\/archives\/2009\/11\/06\/test_driven_deployment_-_mcollective_puppet_cucumber.php\">driving MCollective with Cucumber<\/a>.  Today I&#8217;ll show how I did that with <a href=\"http:\/\/marionette-collective.org\/simplerpc\/\">SimpleRPC<\/a>.<\/p>\n<p><a href=\"http:\/\/wiki.github.com\/aslakhellesoy\/cucumber\">Cucumber<\/a> is a testing framework, it might not be the perfect fit for systems scripting but you can achieve a lot if you bend it a bit to your will.  Ultimately I am building up to using it for testing, but we need to start with how to drive MCollective first.<\/p>\n<p>The basic idea is that you wrote a <a href=\"http:\/\/marionette-collective.org\/simplerpc\/agents.html\">SimpleRPC Agent<\/a> for your needs like the one I showed <a href=\"http:\/\/marionette-collective.org\/introduction\/screencasts.html#mcollective_deployer\">here<\/a>.   The specific agent has a number of tasks it can perform:<\/p>\n<ul>\n<li>Install, Uninstall and Update Packages<\/li>\n<li>Query NRPE status for a specific NRPE command<\/li>\n<li>Start, Stop and Restart Services<\/li>\n<\/ul>\n<p>These features are all baked into a single agent, perfect for driving from a set of Cucumber features.  The sample I will show here is only driving the <a href=\"http:\/\/code.google.com\/p\/mcollective-plugins\/source\/browse\/#svn\/trunk\/agent\/iptables-junkfilter\">IPTables<\/a> agent since that code is public and visible.<\/p>\n<p>First I&#8217;ll show the feature I want to build, we&#8217;re still concerned with driving the agent here not testing so much &#8211; though the steps are tested and idempotent:<\/p>\n<p><code><\/p>\n<pre lang=\"text\">\r\nFeature: Manage the iptables firewall\r\n\r\n    Background:\r\n    Given the load balancer has ip address 192.168.1.1\r\n    And I want to update hosts with class \/dev_server\/\r\n    And I want to update hosts with fact country=de\r\n    And I want to pre-discover the nodes to manage\r\n       \r\n    Scenario: Manage the firewall\r\n        When I block the load balancer\r\n        Then traffic from the load balancer should be blocked\r\n\r\n        # other tasks like package management, service restarts\r\n        # and monitor tasks would go here\r\n\r\n        When I unblock the load balancer\r\n        Then traffic from the load balancer should be unblocked\r\n<\/pre>\n<p><\/code><\/p>\n<p>To realize the above we&#8217;ll need some setup code that fires up our RPC client and manage options in a single place, we&#8217;ll place this in in <em>support\/env.rb<\/em>:<\/p>\n<p><code><\/p>\n<pre lang=\"ruby\">\r\nrequire 'mcollective'\r\n\r\nWorld(MCollective::RPC)\r\n\r\nBefore do\r\n    @options = {:disctimeout => 2,\r\n                :timeout     => 5,\r\n                :verbose     => false,\r\n                :filter      => {\"identity\"=>[], \"fact\"=>[], \"agent\"=>[], \"cf_class\"=>[]},\r\n                :config      => \"etc\/client.cfg\"}\r\n\r\n    @iptables = rpcclient(\"iptables\", :options => @options)\r\n    @iptables.progress = false\r\nend\r\n<\/pre>\n<p><\/code><\/p>\n<p>First we load up the MCollective code and install it into the <a href=\"http:\/\/wiki.github.com\/aslakhellesoy\/cucumber\/a-whole-new-world\">Cucumber World<\/a>, this achieves more or less what <em>include MCollective::RPC<\/em> would in a Cucumber friendly way.<\/p>\n<p>We then set some sane default options and start our RPC client.<\/p>\n<p>Now we can go onto writing some steps, we store these in <em>step_definitions\/mcollective_steps.rb<\/em>, first we want to capture some data like the load balancer IP and filters:<\/p>\n<p><code><\/p>\n<pre lang=\"ruby\">\r\nGiven \/^the (.+) has ip address (\\d+\\.\\d+\\.\\d+\\.\\d+)$\/ do |device, ip|\r\n    @ips = {} unless @ips\r\n\r\n    @ips[device] = ip\r\nend\r\n\r\nGiven \/I want to update hosts with fact (.+)=(.+)$\/ do |fact, value|\r\n    @iptables.fact_filter fact, value\r\nend\r\n\r\nGiven \/I want to update hosts with class (.+)$\/ do |klass|\r\n    @iptables.class_filter klass\r\nend\r\n\r\nGiven \/I want to pre-discover the nodes to manage\/ do\r\n    @iptables.discover\r\n\r\n    raise(\"Did not find any nodes to manage\") if @iptables.discovered.size == 0\r\nend\r\n<\/pre>\n<p><\/code><\/p>\n<p>Here we&#8217;re just creating a table of device names to ips and we manipulate the <a href=\"http:\/\/marionette-collective.org\/simplerpc\/clients.html#applying_filters_programatically\">MCollective Filters<\/a>.  Finally we do a discover and we check that we are actually matching any hosts.  If your filters were not matching any nodes the cucumber run would bail out.<\/p>\n<p>Now we want to first do the work to block and unblock the load balancers:<\/p>\n<p><code><\/p>\n<pre lang=\"ruby\">\r\nWhen \/^I block the (.+)$\/ do |device|\r\n    raise(\"Unknown device #{device}\") unless @ips.include?(device)\r\n\r\n    @iptables.block(:ipaddr => @ips[device]) \r\n\r\n    raise(\"Not all nodes responded\") unless @iptables.stats[:noresponsefrom].size == 0\r\nend\r\n\r\nWhen \/^I unblock the (.+)$\/ do |device|\r\n    raise(\"Unknown device #{device}\") unless @ips.include?(device)\r\n\r\n    @iptables.unblock(:ipaddr => @ips[device])\r\n\r\n    raise(\"Not all nodes responded\") unless @iptables.stats[:noresponsefrom].size == 0\r\nend\r\n<\/pre>\n<p><\/code><\/p>\n<p>We do some very basic sanity checks here, simply catching nodes that did not respond and bailing out if there are any.  Key is to note that to actually manipulate firewalls on any number of machines is roughly 1 line of code.<\/p>\n<p>Now that we&#8217;re able to block and unblock IPs we also need a way to confirm those tasks were 100% done:<\/p>\n<p><code><\/p>\n<pre lang=\"ruby\">\r\nThen \/^traffic from the (.+) should be blocked$\/ do |device|\r\n    raise(\"Unknown device #{device}\") unless @ips.include?(device)\r\n\r\n    unblockedon = @iptables.isblocked(:ipaddr => @ips[device]).inject(0) do |c, resp|\r\n        c += 1 if resp[:data][:output] =~ \/is not blocked\/    \r\n    end\r\n\r\n    raise(\"Not blocked on: #{unblockedon} \/ #{@iptables.discovered} hosts\") if unblockedon \r\n    raise(\"Not all nodes responded\") unless @iptables.stats[:noresponsefrom].size == 0\r\nend\r\n\r\nThen \/^traffic from the (.+) should be unblocked$\/ do |device|\r\n    raise(\"Unknown device #{device}\") unless @ips.include?(device)\r\n\r\n    blockedon = @iptables.isblocked(:ipaddr => @ips[device]).inject(0) do |c, resp|\r\n        c += 1 if resp[:data][:output] =~ \/is blocked\/    \r\n    end\r\n\r\n    raise(\"Still blocked on: #{blockedon} \/ #{@iptables.discovered} hosts\") if blockedon \r\n    raise(\"Not all nodes responded\") unless @iptables.stats[:noresponsefrom].size == 0\r\nend\r\n<\/pre>\n<p><\/code><\/p>\n<p>This code does actual verification that the clients have the IP blocked or not.  This code also highlights that perhaps my iptables agent needs some refactoring, I have two if blocks that checks for the existence of a string pattern in the result, I could make the agent return Boolean in addition to human readable results.  This would make using the agent easier to use from a program like this.<\/p>\n<p>That&#8217;s all there is to it really, MCollective RPC makes reusing code very easy and it makes addressing networks very easy.  <\/p>\n<p><center><img decoding=\"async\" src=\"http:\/\/www.devco.net\/images\/cucuber-mcollective.png\"><\/center><\/p>\n<p>&nbsp;<\/p>\n<h4>Monitoring \/ Infrastructure Testing<\/h4>\n<p>The above code demonstrates how using MCollective+Cucumber you can address any number of machines, perform actions and get states within a testing framework.  This seems an uncomfortable fit &#8211; since Cucumber is a testing framework &#8211; but it doesn&#8217;t need to be.<\/p>\n<p>Above I am using cucumber to drive actions but it would be great to use this combination to do testing of infrastructure states using something like <a href=\"http:\/\/auxesis.github.com\/cucumber-nagios\/\">cucumber-nagios<\/a>.  The great thing that MCollective brings to the table here is that you can have sets of tests that changes behavior with the environment while having the ability to break out of the single box barriers.<\/p>\n<p>With this you can easily write a kind of infrastructure test that transcends machine boundaries.  You could check the state of one set of variables on one set of machines, and based on the value of those go and check that other machines are in a state that makes those variables valid variables to have.  <\/p>\n<p>We&#8217;re able to answer those &#8216;this machine is doing x, did the admin remember to do y on another machine?&#8217; style questions.  Examples of this could be:  <\/p>\n<ul>\n<li>If the backups are running, did the cron job that takes a database out of the service pool get run?  This would flag up at any time, even if someone is doing a manual run of backups.<\/li>\n<li>How many Puppet Daemons is currently actively doing manifests on all our nodes, alert if more than 10.  Even this simple case is hard &#8211; you need a view of the status of an application in real time across many nodes, and requires information from <em>now<\/em> rather than the usual 5 minute window of Nagios.<\/li>\n<li>If there are 10 concurrent puppetd runs happening right now, is the puppet master coping?  This test would stay green, and not care for the master until the time comes that there are many puppetd&#8217;s doing manifest runs.  This way if your backups or sysadmin action pushes the load up on the master the check will stay green, it will only trigger if you&#8217;re seeing many Puppet clients running.  This could be useful indicators for capacity planning.<\/li>\n<\/ul>\n<p>These simple cases are generally hard for systems like Nagios to do, it&#8217;s hard to track state of many checks, apply logic and then go CRITICAL if a combination of factors combine to give a failure, we can build such test cases with MCollective and Cucumber fairly easily.  <\/p>\n<p>The code here does not really show you how to do that per se, but what it does show is how natural and easy it is to interact with your network of hosts via MCollective and Ruby.  In future I might post some more code here to show how we can build on these ideas and create test suites as described.  As a example a test case for the above Puppet Master example might be:<\/p>\n<p><code><\/p>\n<pre lang=\"text\">\r\nFeature: Monitor the capacity of the Puppet Master\r\n\r\n    Background:\r\n    Given we know we can run 10 concurrent Puppet clients\r\n    And the Puppet Master load average should be below 2\r\n\r\n    Scenario: Monitor the Puppet Master capacity \r\n        When there are more than usual Puppet clients running\r\n        Then the Puppet Master should have an acceptable load average\r\n<\/pre>\n<p><\/code><\/p>\n<p>Running this under <em>cucumber-nagios<\/em> we&#8217;ll achieve our stated goals.<\/p>\n<p>As a small post note, figuring out how many Puppet Daemons are currently running their manifests is trivial with the <a href=\"http:\/\/code.google.com\/p\/mcollective-plugins\/wiki\/AgentPuppetd\">Puppet Agent<\/a>:<\/p>\n<p><code><\/p>\n<pre lang=\"ruby\">\r\np = rpcclient(\"puppetd\")\r\np.progress = false\r\n\r\nrunning = p.status.inject(0) {|c, status| c += status[:data][:running]}\r\nputs(\"Currently running: #{running}\")\r\n<\/pre>\n<p><\/code><\/p>\n<p><code><\/p>\n<pre lang=\"shell\">\r\n$ ruby test.rb\r\nCurrently running: 3\r\n<\/pre>\n<p><\/code><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Some time ago I showed some sample code I had for driving MCollective with Cucumber. Today I&#8217;ll show how I did that with SimpleRPC. Cucumber is a testing framework, it might not be the perfect fit for systems scripting but you can achieve a lot if you bend it a bit to your will. Ultimately [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","footnotes":""},"categories":[1],"tags":[121,85,78,13],"_links":{"self":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/1356"}],"collection":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/comments?post=1356"}],"version-history":[{"count":41,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/1356\/revisions"}],"predecessor-version":[{"id":1387,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/1356\/revisions\/1387"}],"wp:attachment":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/media?parent=1356"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/categories?post=1356"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/tags?post=1356"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}