Note: While this post is still relevant, you should also check the newer Simple Puppet Module Structure Redux post
Puppet supports something called Modules, it’s a convenient collection of templates, files, classes, types and facts all related to one thing. I find it makes it easy to think about a problem in an isolated manner and it should be the default structure everyone use.
Usually though the problem comes with how to lay out modules and how to really use the Puppet classes system, below a simple module layout that can grow with you as your needs extend. I should emphasis the simple here – this example does not cater well for environments with mix of Operating Systems, or weird multi site setups with overrides per environment, but once you understand the design behind this you should be able to grow it to cater for your needs. If you have more complex needs, see this post as a primer on using the class relationships effectively.
For the most part I find I make modules for each building block – apache, syslog, php, mysql – usually the kind of thing that has:
- One or more packages – or at least some install process.
- One or more config files or config actions
- A service or something that you do to start it.
You can see there’s an implied dependency tree here, the config steps require the package to be installed and the service step require the config to be done. Similarly if any config changes the services might need to restart.
We’d want to cater for the case where you could have 10 config files, or 10 packages, or 100s of files needed to install the service in question. You really wouldn’t want to create a big array of require => resources to control the relationships or to do the notifications, this would be a maintenance nightmare.
We’re going to create a class for each of the major parts of the module – package, config and service. And we’ll use the class systems and relationships, notifies etc to ensure the ordering, requires and notifies are kept simple – especially so that future changes can be done in one place only without editing everywhere else that has requires.
class ntp::install {
package{"ntpd":
ensure => latest
}
}
class ntp::config {
File{
require => Class["ntp::install"],
notify => Class["ntp::service"],
owner => "root",
group => "root",
mode => 644
}
file{"/etc/ntp.conf":
source => "puppet:///ntp/ntp.conf";
"/etc/ntp/step-tickers":
source => "puppet:///ntp/step-tickers";
}
}
class ntp::service {
service{"ntp":
ensure => running,
enable => true,
require => Class["ntp::config"],
}
}
class ntp {
include ntp::install, ntp::config, ntp::service
} |
If we now just do include ntp everything will happen cleanly and in order.
There’s a few things you should notice here. We’re using the following syntax in a few places:
- require => Class[“ntp::something”]
- notify => Class[“ntp::service”]
- before => Class[“ntp::something”]
- subscribe => Class[“ntp::config”]
All of the usual relationship meta parameters can operate on classes, so what this means is if you require Class[“ntp::config”] then both ntp.conf and step-tickers will be created before the service starts without having to list the files in your relationships. Similarly if you notify a class then all resources in that class gets notifies.
The preceding paragraph is very important to grasp, typically we’d try to maintain arrays of files in require trees and later if we add a new config file we have to go hunt everywhere that needs to require it and update those places, by using this organization approach we never have to update anything. If I add 10 services and 100 files into the config and service classes everything else that requires ntp::service will still work as expected without needing updates.
This is a very important decoupling of class contents with class function. Your other classes should never be concerned with class contents only class function.
As a convention this layout is a good sample, if all your daemons gets installed in the daemon::install class you’ll have no problem knowing where to find the code to adjust the install behavior – perhaps to upgrade to the next version – you should try to come up with a set of module naming conventions that works for you and stick to it throughout your manifests – I have a rake task to setup my usual module structure, they’re all the same and the learning curve for new people is small.
By building up small classes that focus on their task or subtask you can build up ever more powerful modules. We have a simple example here, but you can see how you could create ntp::master and ntp::client wrapper classes that would simply reuse the normal ntp::install and ntp::service classes.
As I mentioned this is a simple example for simple needs, the real take away here is to use class relationships and decouple the contents of those classes with their intent this should give you a more maintainable code set.
Ultimately conventions such as these will be needed to get us all one step closer to being able to share modules amongst each other as the specifics of making Apache on different distributions work will not matter – we’d just know that we want Class[“apache::service”] for example.
Nice; like the decoupling you get from depending the Class and not the Files directly – I’ve been looking for a way to depend on an array of things, but this is a lot neater.
Are your manifests shared out anywhere (github or similar)?
Unfortunately my manifests aren’t published.
it’s great to be able to use require, before, notify and subscribe with classes… I’m semi-new to puppet, coming into it during the 0.24 -> 0.25 transition. Is this a new feature with 0.25.x? I don’t remember it being in 0.24.x.
This feature has been there for a very long time.
One thing that ought to be more explicit is the mappings between the path names and the classes they contain. Your statement:
is only true if the file lives at the path /etc/puppet/modules/ntp/manifests/init.pp ; similarly if the top-level classes get too big and need to be broken out into their own files, the file
/etc/puppet/modules/ntp/manifests/install.pp can contain the ntp::install class without any special ‘import’ directives.
ModuleStandards talks about the autoloading a little bit but that doc is otherwise pretty stale so it’s not suprising that people don’t know about it.
It’s nice and clean, but what if a simple file change does not require a service to restart… So lets say I have 10 services (unlikely) and 100 files and only a few files require a restart of lets say certain services, but not all 10. Decoupling is nice and clean, but may not be the most efficient in all cases.
Don’t get me wrong, I do like the concept. Keeping a module clean of site specific details will indeed be one step closer for us sharing modules though. Hopefully we’ll get there sooner than later so we don’t have 10 millions of writing a simple puppet module to manage mysql for instance. The module should be able to manage users/db/grant where I can map their relationship and the module will manage to realize that regardless of distros or releases.
James,
This isn’t hard, just put: notify => undef on that file, and it wont notify the service, nice and easy
That’s true. I have to get use to undef still.
Thx rip!
Ok. Now I ran into inheritance problems.
Now that I’ve broken down the classes into simpler functions such as mysql::packages, mysql::files, etc etc. And then I have one encompassing class lets say mysql::server that includes the above classes. In this situation I cannot inherit mysql::server and override my resources.
Would you agree that I’ve lost or given up my class inheritance ability to override resources? Maybe you can shed some light as you’ve done earlier with undef.
It does become harder to inherit I agree, I generally try hard to totally avoid inheritance because I don’t really think it’s the best approach in most cases, though there are of course valid uses.
Not sure what to suggest in your case without seeing what you’re trying to do, grab me on IRC.
Additional note… I cannot even override resources in my mysql::server class, let alone another class that inherits mysql::server. I get the following error:
Error 400 on SERVER: Only subclasses can override parameters at /path/to/module/file
I’ll catch up with you on IRC soon, but here’s my design which is pretty much borrowed from lab42’s solution…
At our toplevel modules we have our generic modules with the basic manifest to suit most installs. At the same level lies our project_name module that holds the infrastructure design layout you’d find from lab42. Under project_name we have sub modules which basically extends in a sense, and adds specific configurations for the particular project that we don’t want to include in the top level module, which I want to keep generic for the purpose of sharing modules with others or even between my various projects/contracts.
I guess for this reason I cannot apply this simple pattern because puppet configurations aren’t truly OO in a sense. If you’re confused by now, I understand. I’ll catchup with you on IRC and share my git repo for viewing once I’m done with this contract.
Ok, I see the problem. Labs24 uses inheritance heavily and probably won’t work too hot with this layout, with the way have it set up you pretty much have to buy into their way completely and just do exactly what they do and not deviate 🙂
There was a discussion on the list recently about extending the language with a different type of inherit that will be much nicer, I need the time to file a feature request for that still.
Actually after giving it some more thought I got the best of both worlds to work together finally. It wasn’t too bad but I don’t know if it’s the best solution. Extending puppet’s inheritance would be a much better route. For this time being though this is what I have…
# Generic top level module
class A::packages {…}
class A::files {…}
class A::configure {…}
class A::server {
include A::packages
include A::files
include A::configure
}
# Applying Lab42’s approach to infrastructure layout at the project module level
class A::company::files inherits A::files {
File[‘file.conf’] {
source => “puppet://$server/project_company/etc/app/file.conf”
}
}
class A::company::server {
include A::packages
include A::company::files
include A::configure
}
This way I still can override any of the smaller classes and then encompassing them in my extended class for the company I’m contracting with.
What do you think?
Best you can do atm really with what we have 🙂
How would the directory structure look for a submodule inside of ntp::install::database?
@shawn: manifests/install/database.pp