<\/p>\n\r\n# modules\/cspec\/lib\/puppet\/datatypes\/cspec\/suite.rb\r\nPuppet::DataTypes.create_type(\"Cspec::Suite\") do\r\n interface <<-PUPPET\r\n attributes => {\r\n \"description\" => String,\r\n \"fail_fast\" => Boolean,\r\n \"report\" => String\r\n },\r\n functions => {\r\n it => Callable[[String, Callable[Cspec::Case]], Any],\r\n }\r\n PUPPET\r\n\r\n load_file \"puppet_x\/cspec\/suite\"\r\n\r\n implementation_class PuppetX::Cspec::Suite\r\nend\r\n<\/pre>\n<\/code><\/p>\n
As you can see from the line of code cspec::suite(“filemgr agent tests”, $fail_fast, $report) |$suite| {….}<\/em> we pass 3 arguments: a description of the test, if the test should fail immediately on any error or keep going and there to write the report of the suite to. This corresponds to the attributes<\/em> here. A function that will be shown later takes these and make our instance.<\/p>\nWe then have to add our it()<\/em> function which again takes a description and yields out `Cspec::Case`, it returns any value.<\/p>\nWhen Puppet needs the implementation of this code it will call the Ruby class PuppetX::Cspec::Suite<\/em>.<\/p>\nHere is the same for the Cspec::Case<\/em>:<\/p>\n<\/p>\n\r\n# modules\/cspec\/lib\/puppet\/datatypes\/cspec\/case.rb\r\nPuppet::DataTypes.create_type(\"Cspec::Case\") do\r\n interface <<-PUPPET\r\n attributes => {\r\n \"description\" => String,\r\n \"suite\" => Cspec::Suite\r\n },\r\n functions => {\r\n assert_equal => Callable[[Any, Any], Boolean],\r\n assert_task_success => Callable[[Choria::TaskResults], Boolean],\r\n assert_task_data_equals => Callable[[Choria::TaskResult, Any, Any], Boolean]\r\n }\r\n PUPPET\r\n\r\n load_file \"puppet_x\/cspec\/case\"\r\n\r\n implementation_class PuppetX::Cspec::Case\r\nend\r\n<\/pre>\n<\/code><\/p>\n
Adding the implementation<\/H3>
\nThe implementation is a Ruby class that provide the logic we want, I won’t show the entire thing with reporting and everything but you’ll get the basic idea:<\/p>\n
<\/p>\n\r\n# modules\/cspec\/lib\/puppet_x\/cspec\/suite.rb\r\nmodule PuppetX\r\n class Cspec\r\n class Suite\r\n # Puppet calls this method when it needs an instance of this type\r\n def self.from_asserted_hash(description, fail_fast, report)\r\n new(description, fail_fast, report)\r\n end\r\n\r\n attr_reader :description, :fail_fast\r\n\r\n def initialize(description, fail_fast, report)\r\n @description = description\r\n @fail_fast = !!fail_fast\r\n @report = report\r\n @testcases = []\r\n end\r\n\r\n # what puppet file and line the Puppet DSL is on\r\n def puppet_file_line\r\n fl = Puppet::Pops::PuppetStack.stacktrace[0]\r\n\r\n [fl[0], fl[1]]\r\n end\r\n\r\n def outcome\r\n {\r\n \"testsuite\" => @description,\r\n \"testcases\" => @testcases,\r\n \"file\" => puppet_file_line[0],\r\n \"line\" => puppet_file_line[1],\r\n \"success\" => @testcases.all?{|t| t[\"success\"]}\r\n }\r\n end\r\n\r\n # Writes the memory state to disk, see outcome above\r\n def write_report\r\n # ...\r\n end\r\n\r\n def run_suite\r\n Puppet.notice(\">>>\")\r\n Puppet.notice(\">>> Starting test suite: %s\" % [@description])\r\n Puppet.notice(\">>>\")\r\n\r\n begin\r\n yield(self)\r\n ensure\r\n write_report\r\n end\r\n\r\n\r\n Puppet.notice(\">>>\")\r\n Puppet.notice(\">>> Completed test suite: %s\" % [@description])\r\n Puppet.notice(\">>>\")\r\n end\r\n\r\n def it(description, &blk)\r\n require_relative \"case\"\r\n\r\n t = PuppetX::Cspec::Case.new(self, description)\r\n t.run(&blk)\r\n ensure\r\n @testcases << t.outcome\r\n end\r\n end\r\n end\r\nend\r\n<\/pre>\n<\/code><\/p>\n
And here is the Cspec::Case<\/em>:<\/p>\n<\/p>\n\r\n# modules\/cspec\/lib\/puppet_x\/cspec\/case.rb\r\nmodule PuppetX\r\n class Cspec\r\n class Case\r\n # Puppet calls this to make instances\r\n def self.from_asserted_hash(suite, description)\r\n new(suite, description)\r\n end\r\n\r\n def initialize(suite, description)\r\n @suite = suite\r\n @description = description\r\n @assertions = []\r\n @start_location = puppet_file_line\r\n end\r\n\r\n # assert 2 things are equal and show sender etc in the output\r\n def assert_task_data_equals(result, left, right)\r\n if left == right\r\n success(\"assert_task_data_equals\", \"%s success\" % result.host)\r\n return true\r\n end\r\n\r\n failure(\"assert_task_data_equals: %s\" % result.host, \"%s\\n\\n\\tis not equal to\\n\\n %s\" % [left, right])\r\n end\r\n\r\n # checks the outcome of a choria RPC request and make sure its fine\r\n def assert_task_success(results)\r\n if results.error_set.empty?\r\n success(\"assert_task_success:\", \"%d OK results\" % results.count)\r\n return true\r\n end\r\n\r\n failure(\"assert_task_success:\", \"%d failures\" % [results.error_set.count])\r\n end\r\n\r\n # assert 2 things are equal\r\n def assert_equal(left, right)\r\n if left == right\r\n success(\"assert_equal\", \"values matches\")\r\n return true\r\n end\r\n\r\n failure(\"assert_equal\", \"%s\\n\\n\\tis not equal to\\n\\n %s\" % [left, right])\r\n end\r\n\r\n # the puppet .pp file and line Puppet is on\r\n def puppet_file_line\r\n fl = Puppet::Pops::PuppetStack.stacktrace[0]\r\n\r\n [fl[0], fl[1]]\r\n end\r\n\r\n # show a OK message, store the assertions that ran\r\n def success(what, message)\r\n @assertions << {\r\n \"success\" => true,\r\n \"kind\" => what,\r\n \"file\" => puppet_file_line[0],\r\n \"line\" => puppet_file_line[1],\r\n \"message\" => message\r\n }\r\n\r\n Puppet.notice(\"✔\ufe0e %s: %s\" % [what, message])\r\n end\r\n\r\n # show a Error message, store the assertions that ran\r\n def failure(what, message)\r\n @assertions << {\r\n \"success\" => false,\r\n \"kind\" => what,\r\n \"file\" => puppet_file_line[0],\r\n \"line\" => puppet_file_line[1],\r\n \"message\" => message\r\n }\r\n\r\n Puppet.err(\"\u2718 %s: %s\" % [what, @description])\r\n Puppet.err(message)\r\n\r\n raise(Puppet::Error, \"Test case %s fast failed: %s\" % [@description, what]) if @suite.fail_fast\r\n end\r\n\r\n # this will show up in the report JSON\r\n def outcome\r\n {\r\n \"testcase\" => @description,\r\n \"assertions\" => @assertions,\r\n \"success\" => @assertions.all? {|a| a[\"success\"]},\r\n \"file\" => @start_location[0],\r\n \"line\" => @start_location[1]\r\n }\r\n end\r\n\r\n # invokes the test case\r\n def run\r\n Puppet.notice(\"==== Test case: %s\" % [@description])\r\n\r\n # runs the puppet block\r\n yield(self)\r\n\r\n success(\"testcase\", @description)\r\n end\r\n end\r\n end\r\nend\r\n<\/pre>\n<\/code><\/p>\n
Finally I am going to need a little function to create the suite - cspec::suite<\/em> function, it really just creates an instance of PuppetX::Cspec::Suite<\/em> for us.<\/p>\n<\/p>\n\r\n# modules\/cspec\/lib\/puppet\/functions\/cspec\/suite.rb\r\nPuppet::Functions.create_function(:\"cspec::suite\") do\r\n dispatch :handler do\r\n param \"String\", :description\r\n param \"Boolean\", :fail_fast\r\n param \"String\", :report\r\n\r\n block_param\r\n\r\n return_type \"Cspec::Suite\"\r\n end\r\n\r\n def handler(description, fail_fast, report, &blk)\r\n suite = PuppetX::Cspec::Suite.new(description, fail_fast, report)\r\n\r\n suite.run_suite(&blk)\r\n suite\r\n end\r\nend\r\n<\/pre>\n<\/code><\/p>\n
Bringing it together<\/H2>
\nSo that's about it, it's very simple really the code above is pretty basic stuff to achieve all of this, I hacked it together in a day basically.<\/p>\n
Lets see how we turn these building blocks into a test suite.<\/p>\n
I need a entry point that drives the suite - imagine I will have many different plans to run, one per agent and that I want to do some pre and post run tasks etc.<\/p>\n
<\/p>\n\r\nplan cspec::suite (\r\n Boolean $fail_fast = false,\r\n Boolean $pre_post = true,\r\n Stdlib::Absolutepath $report,\r\n String $data\r\n) {\r\n $ds = {\r\n \"type\" => \"file\",\r\n \"file\" => $data,\r\n \"format\" => \"yaml\"\r\n }\r\n\r\n # initializes the report\r\n cspec::clear_report($report)\r\n\r\n # force a puppet run everywhere so PuppetDB is up to date, disables Puppet, wait for them to finish\r\n if $pre_post {\r\n choria::run_playbook(\"cspec::pre_flight\", ds => $ds)\r\n }\r\n\r\n # Run our test suite\r\n choria::run_playbook(\"cspec::run_suites\", _catch_errors => true,\r\n ds => $ds,\r\n fail_fast => $fail_fast,\r\n report => $report\r\n )\r\n .choria::on_error |$err| {\r\n err(\"Test suite failed with a critical error: ${err.message}\")\r\n }\r\n\r\n # enables Puppet\r\n if $pre_post {\r\n choria::run_playbook(\"cspec::post_flight\", ds => $ds)\r\n }\r\n\r\n # reads the report from disk and creates a basic overview structure\r\n cspec::summarize_report($report)\r\n}\r\n<\/pre>\n<\/code><\/p>\n
Here's the cspec::run_suites<\/em> Playbook that takes data from a Choria data source and drives the suite dynamically:<\/p>\n<\/p>\n\r\nplan cspec::run_suites (\r\n Hash $ds,\r\n Boolean $fail_fast = false,\r\n Stdlib::Absolutepath $report,\r\n) {\r\n $suites = choria::data(\"suites\", $ds)\r\n\r\n notice(sprintf(\"Running test suites: %s\", $suites.join(\", \")))\r\n\r\n choria::data(\"suites\", $ds).each |$suite| {\r\n choria::run_playbook($suite,\r\n ds => $ds,\r\n fail_fast => $fail_fast,\r\n report => $report\r\n )\r\n }\r\n}\r\n<\/pre>\n<\/code><\/p>\n
And finally a YAML file defining the suite, this file describes my AWS environment that I use to do integration tests for Choria and you can see there's a bunch of other tests here in the suites<\/em> list and some of them will take data like what nodes to expect etc.<\/p>\n<\/p>\n\r\nsuites:\r\n - cspec::discovery\r\n - cspec::choria\r\n - cspec::agents::shell\r\n - cspec::agents::process\r\n - cspec::agents::filemgr\r\n - cspec::agents::nettest\r\n\r\nchoria.version: mcollective plugin 0.7.0\r\n\r\nnettest.fqdn: puppet.choria.example.net\r\nnettest.port: 8140\r\n\r\ndiscovery.all_nodes:\r\n - archlinux1.choria.example.net\r\n - centos7.choria.example.net\r\n - debian9.choria.example.net\r\n - puppet.choria.example.net\r\n - ubuntu16.choria.example.net\r\n\r\ndiscovery.mcollective_nodes:\r\n - archlinux1.choria.example.net\r\n - centos7.choria.example.net\r\n - debian9.choria.example.net\r\n - puppet.choria.example.net\r\n - ubuntu16.choria.example.net\r\n\r\ndiscovery.filtered_nodes:\r\n - centos7.choria.example.net\r\n - puppet.choria.example.net\r\n\r\ndiscovery.fact_filter: operatingsystem=CentOS\r\n<\/pre>\n<\/code><\/p>\n
Conclusion<\/H2>
\nSo this then is a rather quick walk through of extending Puppet in ways many of us would not have seen before. I spent about a day getting this all working which included figuring out a way to maintain the mutating report state internally etc, the outcome is a test suite I can run and it will thoroughly drive a working 5 node network and assert the outcomes against real machines running real software.<\/p>\n
I used to have a MCollective integration test suite, but I think this is a LOT nicer mainly due to the Choria Playbooks and extensibility of modern Puppet.<\/p>\n
<\/p>\n\r\n$ mco playbook run cspec::suite --data `pwd`\/suite.yaml --report `pwd`\/report.json\r\n<\/pre>\n<\/code><\/p>\n
The current code for this is on GitHub<\/a> along with some Terraform code to stand up a test environment, it's a bit barren right now but I'll add details in the next few weeks.<\/p>\n","protected":false},"excerpt":{"rendered":"Extending Puppet using types, providers, facts and functions are well known and widely done. Something new is how to add entire new data types to the Puppet DSL to create entirely new language behaviours. I’ve done a bunch of this recently with the Choria Playbooks and some other fun experiments, today I’ll walk through building […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","footnotes":""},"categories":[7],"tags":[126,78,21],"_links":{"self":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/3773"}],"collection":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/comments?post=3773"}],"version-history":[{"count":8,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/3773\/revisions"}],"predecessor-version":[{"id":3781,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/3773\/revisions\/3781"}],"wp:attachment":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/media?parent=3773"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/categories?post=3773"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/tags?post=3773"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}