{"id":2124,"date":"2011-06-06T15:52:10","date_gmt":"2011-06-06T14:52:10","guid":{"rendered":"http:\/\/www.devco.net\/?p=2124"},"modified":"2011-10-27T23:29:17","modified_gmt":"2011-10-27T22:29:17","slug":"puppet_backend_for_hiera","status":"publish","type":"post","link":"https:\/\/www.devco.net\/archives\/2011\/06\/06\/puppet_backend_for_hiera.php","title":{"rendered":"Puppet backend for Hiera"},"content":{"rendered":"<p><strong>Note:<\/strong> This project is now being managed by Puppetlabs, its new home is <a href=\"http:\/\/projects.puppetlabs.com\/projects\/hiera\">http:\/\/projects.puppetlabs.com\/projects\/hiera<\/a><\/p>\n<p>Yesterday I <a href=\"http:\/\/www.devco.net\/archives\/2011\/06\/05\/hiera_a_pluggable_hierarchical_data_store.php\">released a Hierarchical datastore called Hiera<\/a>, today as promised I&#8217;ll show how it integrates with Puppet.<\/p>\n<p>Extlookup has solved the basic problem of loading data into Puppet.  This was done 3 years ago at a time before Puppet supported complex data in Hashes or things like Parametrized classes.  Now as Puppet has improved a new solution is needed.  I believe the combination of Hiera and the Puppet plugin goes a very long way to solving this and making parametrized classes much more bearable.<\/p>\n<p>I will highlight a sample use case where a module author places a module on the Puppet Forge and a module user downloads and use the module.  Both actors need to create data &#8211; the author needs default data to create a just-works experience and the module user wants to configure the module behavior either in YAML, JSON, Puppet files or anything else he can code.<\/p>\n<h3>Module Author<\/h3>\n<p>The most basic NTP module can be seen below.  It has a <em>ntp::config<\/em> class that uses Hiera to read default data from <em>ntp::data<\/em>:<\/p>\n<p><b>modules\/ntp\/manifests\/config.pp<\/b><br \/>\n<code><\/p>\n<pre lang=\"puppet\">\r\nclass ntp::config($ntpservers = hiera(\"ntpservers\")) {\r\n    file{\"\/tmp\/ntp.conf\":\r\n        content => template(\"ntp\/ntp.conf.erb\")\r\n    }\r\n}\r\n<\/pre>\n<p><\/code><\/p>\n<p><b>modules\/ntp\/manifests\/data.pp<\/b><br \/>\n<code><\/p>\n<pre lang=\"puppet\">\r\nclass ntp::data {\r\n    $ntpservers = [\"1.pool.ntp.org\", \"2.pool.ntp.org\"]\r\n}\r\n<\/pre>\n<p><\/code><\/p>\n<p>This is your most basic NTP module.  By using <em>hiera(&#8220;ntpserver&#8221;)<\/em> you load <em>$ntpserver<\/em> from these variables, the first one that exists gets used.  In this case the last one. <\/p>\n<ul>\n<li>$ntp::config::data::ntpservers<\/li>\n<li>$ntp::data:ntpservers<\/li>\n<\/ul>\n<p>This would be an abstract from a forge module, anyone who use it will be configured to use the <em>ntp.org<\/em> base NTP servers.<\/p>\n<h3>Module User<\/h3>\n<p>As a user I really want to use this NTP module from the Forge and not write my own.  But what I also need is flexibility over what NTP servers I use.  Generally that means forking the module and making local edits.  Parametrized Classes are supposed to make this better but sadly the design decisions means you need an ENC or a flexible data store.  The data store was missing thus far and I really would not recommend their use without it.<\/p>\n<p>Given that the NTP module above is using Hiera as a user I now have various options to override its use.  I configure Hiera to use the (default) YAML backend for data but to also load in the Puppet backend should the YAML one not provide an answer.  I also configure it to allow me to create per-location data that gives me the flexibility I need to pick NTP servers I need.<\/p>\n<p><code><\/p>\n<pre lang=\"yaml\">\r\n:backends: - yaml\r\n           - puppet\r\n:hierarchy: - %{location}\r\n            - common\r\n<\/pre>\n<p><\/code><\/p>\n<p>I now need to decide how best to override the data from the NTP module:<\/p>\n<p>I want:<\/p>\n<ul>\n<li>Per datacenter values when needed.  The NOC at the data center can change this data without change control.<\/li>\n<li>Company wide policy the should apply over the module defaults.  This is company policy and should be subject to change control like my manifests.<\/li>\n<\/ul>\n<p>Given these constraints I think the per-datacenter policy can go into data files that is controlled outside of my VCS like with a web application or simple editor.  The common data that should apply company wide need to be controlled under my VCS and managed by the change control board.<\/p>\n<p>Hiera makes this easy.  By configuring it as above the new Puppet data search path &#8211; for a machine in <em>dc1<\/em> &#8211; would be:<\/p>\n<ul>\n<li>$data::dc1::ntpservers &#8211; based on the Hiera configuration, user specific<\/li>\n<li>$data::common::ntpservers &#8211; based on the Hiera configuration, user specific<\/li>\n<li>$ntp::config::data::ntpservers &#8211; users do not touch this, it&#8217;s for module authors<\/li>\n<li>$ntp::data:ntpservers &#8211; users do not touch this, it&#8217;s for module authors<\/li>\n<\/ul>\n<p>You can see this extends the list seen above, the module author data space remain in use but we now have a layer on top we can use.<\/p>\n<p>First we create the company wide policy file in Puppet manifests:<\/p>\n<p><b>modules\/data\/manifests\/common.pp<\/b><br \/>\n<code><\/p>\n<pre lang=\"puppet\">\r\nclass data::common {\r\n   $ntpservers = [\"ntp1.example.com\", \"ntp2.example.com\"]\r\n}\r\n<\/pre>\n<p><\/code><\/p>\n<p>As Hiera will query this prior to querying any in-module data this will effectively prevent any downloaded module from supplying NTP servers other than ours.   This is a company wide policy that applies to all machines unless specifically configured otherwise.  This lives with your code in your SCM.<\/p>\n<p>Next we create the data file for machines with fact <em>$location=dc1<\/em>.  Note this data is created in a YAML file outside of the manifests.  You can use JSON or any other Hiera backend so if you had this data in a CMDB in MySQL you could easily query the data from there:<\/p>\n<p><b>hieradb\/dc1.yaml<\/b><br \/>\n<code><\/p>\n<pre lang=\"yaml\">\r\n---\r\nntpservers: - ntp1.dc1.example.com\r\n            - ntp2.dc1.example.com\r\n<\/pre>\n<p><\/code><\/p>\n<p>And this is the really huge win.  You can create Hiera plugins to get this data from anywhere you like that magically override your in-manifest data.<\/p>\n<p>Finally here are a few Puppet node blocks:<\/p>\n<p><code><\/p>\n<pre lang=\"puppet\">\r\nnode \"web1.prod.example.com\" {\r\n   $location = \"dc1\"\r\n   include ntp::config\r\n}\r\n\r\nnode \"web1.dev.example.com\" {\r\n   $location = \"office\"\r\n   include ntp::config\r\n}\r\n\r\nnode \"oneoff.example.com\" {\r\n   class{\"ntp::config\": \r\n      ntpservers => [\"ntp1.isp.net\"]\r\n   }\r\n}\r\n<\/pre>\n<p><\/code><\/p>\n<p>These 3 nodes will have different NTP configurations based on their location &#8211; you should really make <em>$location<\/em> a fact:<\/p>\n<ul>\n<li><em>web1.prod<\/em> will use <em>ntp1.dc1.example.com<\/em> and <em>ntp2.dc1.example.com<\/em><\/li>\n<li><em>web1.dev<\/em> will use the data from <em>class data::common<\/em><\/li>\n<li><em>oneoff.example.com<\/em> is a complete weird case and you can still use the normal parametrized syntax &#8211; in this case Hiera wont be involved at all.<\/li>\n<\/ul>\n<p>And so we have a very easy to use a natural blend between using param classes from an ENC for full programmatic control without sacrificing the usability for beginner users who can not or do not want to invest the time to create an ENC.<\/p>\n<p>The plugin is available on GitHub as <em><a href=\"https:\/\/github.com\/ripienaar\/hiera-puppet#readme\">hiera-puppet<\/a><\/em> and you can just install the <em>hiera-puppet<\/em> gem.  You will still need to install the <a href=\"http:\/\/projects.puppetlabs.com\/projects\/1\/wiki\/Writing_Your_Own_Functions\">Parser Function into your master<\/a> until the work I did to make <a href=\"http:\/\/projects.puppetlabs.com\/issues\/7788\">Puppet extendable using RubyGems is merged<\/a>.  <\/p>\n<p>The example above is on GitHub and you can just use that to test the concept and see how it works without the need to plug into your Puppet infrastructure first.  See the <a href=\"https:\/\/github.com\/ripienaar\/hiera-puppet\/tree\/master\/example#readme\">README<\/a>.<\/p>\n<p>The Gem includes <em>extlookup2hiera<\/em> that can convert extlookup CSV files into JSON or YAML.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Note: This project is now being managed by Puppetlabs, its new home is http:\/\/projects.puppetlabs.com\/projects\/hiera Yesterday I released a Hierarchical datastore called Hiera, today as promised I&#8217;ll show how it integrates with Puppet. Extlookup has solved the basic problem of loading data into Puppet. This was done 3 years ago at a time before Puppet supported [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","footnotes":""},"categories":[1],"tags":[121,85,97,21,13],"_links":{"self":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/2124"}],"collection":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/comments?post=2124"}],"version-history":[{"count":18,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/2124\/revisions"}],"predecessor-version":[{"id":2280,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/2124\/revisions\/2280"}],"wp:attachment":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/media?parent=2124"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/categories?post=2124"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/tags?post=2124"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}