{"id":2700,"date":"2012-06-30T10:29:39","date_gmt":"2012-06-30T09:29:39","guid":{"rendered":"http:\/\/www.devco.net\/?p=2700"},"modified":"2012-06-30T11:14:59","modified_gmt":"2012-06-30T10:14:59","slug":"mcollective-2-1-data-plugins-for-discovery","status":"publish","type":"post","link":"https:\/\/www.devco.net\/archives\/2012\/06\/30\/mcollective-2-1-data-plugins-for-discovery.php","title":{"rendered":"MCollective 2.1 – Data Plugins for Discovery"},"content":{"rendered":"

This ia a post in a series of posts I am doing about MCollective 2.0 and later<\/a>. <\/p>\n

In my previous post I covered a new syntax for composing discovery queries<\/a> and right at the end touched on a data plugin system, today I’ll cover those in detail and show you how to write and use such a plugin.<\/p>\n

Usage and Overview<\/H3>
\nThese plugins allow you to query any data available on your nodes. Examples might be stat() information for a file, sysctl settings, Augeas matches – really anything you could potentially interact with from Ruby that exist on your managed nodes can be used as discovery data. You can write your own and distribute it and we ship a few with MCollective.<\/p>\n

I’ll jump right in with an example of using these plugins:<\/p>\n

<\/p>\n

\r\n$ mco service restart httpd -S \"\/apache\/ and fstat('\/etc\/rsyslog.conf').md5 = \/51b08b8\/\"\r\n<\/pre>\n
<\/code><\/p>\n
Here we’re using the -S discovery statement so we have full boolean matching.  We match machines with the apache<\/em> class applied and then do a regular expression match over the MD5 string of the \/etc\/rsyslog.conf<\/em> file, any machines with both conditions met are discovered and apache is restarted.<\/p>\n
The fstat plugin ships with MCollective 2.1.0 and newer ready to use, we can have a look at our available plugins:<\/p>\n
<\/p>\n
\r\n$ mco plugin doc\r\n.\r\n.\r\nData Queries:\r\n  agent           Meta data about installed MColletive Agents\r\n  augeas_match    Augeas match lookups\r\n  fstat           Retrieve file stat data for a given file\r\n  resource        Information about Puppet managed resources\r\n  sysctl          Retrieve values for a given sysctl\r\n<\/pre>\n
<\/code><\/p>\n
And we can get information about one of these plugins, lets look at the agent<\/em> one:<\/p>\n
<\/p>\n
\r\n$ mco plugin doc agent\r\nAgent\r\n=====\r\n\r\nMeta data about installed MColletive Agents\r\n\r\n      Author: R.I.Pienaar \r\n     Version: 1.0\r\n     License: ASL 2.0\r\n     Timeout: 1\r\n   Home Page: http:\/\/marionette-collective.org\/\r\n\r\nQUERY FUNCTION INPUT:\r\n\r\n              Description: Valid agent name\r\n                   Prompt: Agent Name\r\n                     Type: string\r\n               Validation: (?-mix:^[\\w\\_]+$)\r\n                   Length: 20\r\n\r\nQUERY FUNCTION OUTPUT:\r\n\r\n           author:\r\n              Description: Agent author\r\n               Display As: Author\r\n\r\n           description:\r\n              Description: Agent description\r\n               Display As: Description\r\n\r\n           license:\r\n              Description: Agent license\r\n               Display As: License\r\n\r\n           timeout:\r\n              Description: Agent timeout\r\n               Display As: Timeout\r\n\r\n           url:\r\n              Description: Agent url\r\n               Display As: Url\r\n\r\n           version:\r\n              Description: Agent version\r\n               Display As: Version\r\n<\/pre>\n
<\/code><\/p>\n
This shows what the query is that this plugin is expecting and what data it returns, so we can use this to discover all machines with version 1.6 of a specific MCollective agent:<\/p>\n
<\/p>\n
\r\n$ mco find -S \"agent('puppetd').version = 1.6\"\r\n<\/pre>\n
<\/code><\/p>\n
And if you’re curious what exactly a plugin would return you can quickly find out using the rpcutil<\/em> agent:<\/p>\n
<\/p>\n
\r\n% mco rpc rpcutil get_data query=puppetd source=agent\r\n\r\ndevco.net                                \r\n         agent: puppetd\r\n        author: R.I.Pienaar\r\n   description: Run puppet agent, get its status, and enable\/disable it\r\n       license: Apache License 2.0\r\n       timeout: 20\r\n           url: https:\/\/github.com\/puppetlabs\/mcollective-plugins\r\n       version: 1.6\r\n<\/pre>\n
<\/code><\/p>\n
Writing your own plugin<\/H3>
\nLets look at writing a plugin.  We’re going to write one that can query a Linux sysctl value and let you discover against that.  We’ll want this plugin only to activate on machines where \/sbin\/sysctl<\/em> exist.<\/p>\n
When we’re done we want to be able to do discovery like:<\/p>\n
<\/p>\n
\r\n% mco service restart iptables -S \"sysctl('net.ipv4.conf.all.forwarding').value=1\"\r\n<\/pre>\n
<\/code><\/p>\n
To restart iptables on all machines with that specific sysctl enabled. Additionally we’d be able to use this plugin in any of our agents:<\/p>\n
<\/p>\n
\r\naction \"query\" do\r\n   reply[:value] = Data.sysctl(request[:sysctl_name]).value\r\nend\r\n<\/pre>\n
<\/code><\/p>\n
So these plugins really are nicely contained reusable bits of data retrieval logic shareable between discovery, agents and clients.<\/p>\n
This is the code for our plugin:<\/p>\n
\r\nmodule MCollective; module Data\r\n  class Sysctl_data\n
<\/code><\/p>\n
These plugins have to be called Something_data<\/em> and they go in the libdir called data\/something_data.rb<\/em>.<\/p>\n
On line 3 we use the activate_when<\/em> helper to ensure we don't enable this plugin on machines without sysctl. The same confinement system as you might have seen in Agents.<\/p>\n
Lines 5 to 18 we run the sysctl command and do some quick and dirty parsing of the result ensuring we return Integers and Floats so that numeric comparison works fine on the CLI. <\/p>\n
You'd think we need to do some input validation here to avoid bogus data or shell injection but below you will see that the DDL defines validation and MCollective will validate the input for you prior to invoking your code. This validation happens on both the server and the client. DDL files also help us generate the documentation you saw above, native OS packages and in some cases command line completion and web UI generation.<\/p>\n
The DDL for this plugin would be:<\/p>\n
<\/p>\n
\r\nmetadata    :name        => \"Sysctl values\",\r\n            :description => \"Retrieve values for a given sysctl\",\r\n            :author      => \"R.I.Pienaar \",\r\n            :license     => \"ASL 2.0\",\r\n            :version     => \"1.0\",\r\n            :url         => \"http:\/\/marionette-collective.org\/\",\r\n            :timeout     => 1\r\n\r\ndataquery :description => \"Sysctl values\" do\r\n    input :query,\r\n          :prompt      => \"Variable Name\",\r\n          :description => \"Valid Variable Name\",\r\n          :type        => :string,\r\n          :validation  => \/\\A[\\w\\-\\.]+\\z\/,\r\n          :maxlength   => 120\r\n\r\n    output :value,\r\n           :description => \"Kernel Parameter Value\",\r\n           :display_as  => \"Value\"\r\nend\r\n<\/pre>\n
<\/code><\/p>\n
This stuff is pretty normal anyone who has written any MCollective agents would have seen these and the input, output and metadata formats are identical.  The timeout is quite important if your plugin is doing something like talking to Augeas then set this timeout to a longer period, the client when doing discovery will wait an appropriate period of time based on these timeouts.<\/p>\n
With the DDL deployed to both the server and the client you can be sure people won't be sending you nasty shell injection attacks and if someone accidentally tries to access a non existing return they'd get an error before sending traffic over the network.<\/p>\n
You're now ready to package up this plugin we support creating RPMs and Debs of mcollective plugins:<\/p>\n
<\/p>\n
\r\n% ls data\r\nsysctl_data.ddl  sysctl_data.rb\r\n% mco plugin package\r\nCreated package mcollective-sysctl-values-data\r\n% ls -l\r\n-rw-rw-r-- 1 rip rip 2705 Jun 30 10:05 mcollective-sysctl-values-data-1.0-1.noarch.rpm\r\n% rpm -qip mcollective-sysctl-values-data-1.0-1.noarch.rpm\r\nName        : mcollective-sysctl-values-data  Relocations: (not relocatable)\r\nVersion     : 1.0                               Vendor: Puppet Labs\r\nRelease     : 1                             Build Date: Sat 30 Jun 2012 10:05:24 AM BST\r\nInstall Date: (not installed)               Build Host: devco.net\r\nGroup       : System Tools                  Source RPM: mcollective-sysctl-values-data-1.0-1.src.rpm\r\nSize        : 1234                             License: ASL 2.0\r\nSignature   : (none)\r\nPackager    : R.I.Pienaar \r\nURL         : http:\/\/marionette-collective.org\/\r\nSummary     : Retrieve values for a given sysctl\r\nDescription :\r\nRetrieve values for a given sysctl\r\n<\/pre>\n
<\/code><\/p>\n
Install this RPM on all your machines and you're ready to use your plugin.  The version and meta data like author and license in the RPM comes from the DDL file.<\/p>\n
Conclusion<\/H3>
\nThis is the second of a trio of new discovery features that massively revamped the capabilities of MCollective discovery.<\/p>\n
Discovery used to be limited to only CM Classes, Facts and Identities now the possibilities are endless as far as data residing on the nodes go.  This is only available in the current development series - 2.1.x - but I hope this one will be short and we'll get these features into the production supported code base soon.<\/p>\n
In the next post I'll cover discovering against arbitrary client side data - this was arbitrary server side data.<\/p>\n","protected":false},"excerpt":{"rendered":"
This ia a post in a series of posts I am doing about MCollective 2.0 and later. In my previous post I covered a new syntax for composing discovery queries and right at the end touched on a data plugin system, today I’ll cover those in detail and show you how to write and use […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","footnotes":""},"categories":[7],"tags":[85,78,106,13],"_links":{"self":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/2700"}],"collection":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/comments?post=2700"}],"version-history":[{"count":28,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/2700\/revisions"}],"predecessor-version":[{"id":2729,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/2700\/revisions\/2729"}],"wp:attachment":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/media?parent=2700"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/categories?post=2700"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/tags?post=2700"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}