{"id":1580,"date":"2010-07-12T16:08:41","date_gmt":"2010-07-12T15:08:41","guid":{"rendered":"http:\/\/www.devco.net\/?p=1580"},"modified":"2010-07-12T16:11:56","modified_gmt":"2010-07-12T15:11:56","slug":"ec2_bootstrap_helper","status":"publish","type":"post","link":"https:\/\/www.devco.net\/archives\/2010\/07\/12\/ec2_bootstrap_helper.php","title":{"rendered":"EC2 Bootstrap Helper"},"content":{"rendered":"<p>I&#8217;ve been working a bit on streamlining the builds I do on EC2 and wanted a better way to provision my machines.  I use CentOS and things are pretty rough to non existent for nicely built EC2 images.  I&#8217;ve used the Rightscale ones till now and while they&#8217;re nice they are also full of lots of code copyrighted by Rightscale.<\/p>\n<p>What I really wanted was something as full featured as Ubuntu&#8217;s <a href=\"https:\/\/help.ubuntu.com\/community\/CloudInit\">CloudInit<\/a> but also didn&#8217;t feel much like touching any Python.  I hacked up something that more or less do what I need.  You can get it on <a href=\"http:\/\/github.com\/ripienaar\/ec2-boot-init\">GitHub<\/a>.  It&#8217;s written and tested on CentOS 5.5.<\/p>\n<p>The idea is that you&#8217;ll have a single multi purpose AMI that you can easily bootstrap onto your puppet\/mcollective infrastructure using this system.  Below for some details.<\/p>\n<p>I prepare my base CentOS AMI with the following mods:<\/p>\n<ul>\n<li>Install Facter and Puppet &#8211; but not enabled<\/li>\n<li>Install the EC2 utilities<\/li>\n<li>Setup the usual <em>getsshkeys<\/em> script<\/li>\n<li>Install the ec2-boot-init RPM<\/li>\n<li>Add a custom fact that reads <em>\/etc\/facts.txt<\/em> &#8211; see later why.  Get one <a href=\"http:\/\/github.com\/ripienaar\/facter-facts\/tree\/master\/etcfacts\/\">here<\/a>.<\/li>\n<\/ul>\n<p>With this in place you need to create some ruby scripts that you will use to bootstrap your machines.  Examples of this would be to install mcollective, configure it to find your current activemq.  Or to set up puppet and do your initial run etc.  <\/p>\n<p>We host these scripts on any webserver &#8211; ideally S3 &#8211; so that when a machine boots it can grab the logic you want to execute on it.  This way you can bug fix your bootstrapping without having to make new AMIs as well as add new bootstrap methods in future to existing AMIs.<\/p>\n<p>Here&#8217;s a simple example that just runs a shell command:<\/p>\n<p><code><\/p>\n<pre lang=\"ruby\">\r\nnewaction(\"shell\") do |cmd, ud, md, config|\r\n    if cmd.include?(:command)\r\n        system(cmd[:command])\r\n    end\r\nend\r\n<\/pre>\n<p><\/code><\/p>\n<p>You want to host this on any webserver in a file called <em>shell.rb<\/em>.  Now create a file <em>list.txt<\/em> in the same location that just have this:<\/p>\n<p><code><\/p>\n<pre lang=\"text\">\r\nshell.rb\r\n<\/pre>\n<p><\/code><\/p>\n<p>You can list as many scripts as you want.  Now when you boot your instance pass it data like this:<\/p>\n<p><code><\/p>\n<pre lang=\"yaml\">\r\n--- \r\n:facts: \r\n  role: webserver\r\n:actions: \r\n- :url: http:\/\/your.net\/path\/to\/actions\/list.txt\r\n  :type: :getactions\r\n- :type: :shell\r\n  :command: date > \/tmp\/test\r\n<\/pre>\n<p><\/code><\/p>\n<p>The above will fetch the list of actions &#8211; our <em>shell.rb<\/em> &#8211; from <em>http:\/\/your.net\/path\/to\/actions\/list.txt<\/em> and then run using the <em>shell<\/em> action the command <em>date > \/tmp\/test<\/em>.  The actions are run in order so you probably always want <em>getactions<\/em> to happen first.<\/p>\n<p>Other actions that this script will take:<\/p>\n<ul>\n<li>Cache all the user and meta data in <em>\/var\/spool\/ec2boot<\/em><\/li>\n<li>Create <em>\/etc\/facts.txt<\/em> with all your facts that you passed in as well as a flat version of the entire instance meta data<\/em>.  <\/li>\n<li>Create a MOTD that shows some key data like AMI ID, Zone, Public and Private hostnames<\/li>\n<\/ul>\n<p>The boot library provides a few helpers that help you write scripts for this environment specifically around fetching files and logging:<\/p>\n<p><code><\/p>\n<pre lang=\"ruby\">\r\n    [\"rubygems-1.3.1-1.el5.noarch.rpm\",\r\n     \"rubygem-stomp-1.1.6-1.el5.noarch.rpm\",\r\n     \"mcollective-common-#{version}.el5.noarch.rpm\",\r\n     \"mcollective-#{version}.el5.noarch.rpm\",\r\n     \"server.cfg.templ\"].each do |pkg|\r\n        EC2Boot::Util.log(\"Fetching pkg #{pkg}\")\r\n        EC2Boot::Util.get_url(\"http:\/\/foo.s3.amazonaws.com\/#{pkg}\", \"\/mnt\/#{pkg}\")\r\n     end\r\n<\/pre>\n<p><\/code><\/p>\n<p>This code fetches a bunch of files from a S3 bucket and save them into <em>\/mnt<\/em>.  Each one gets logged to console and syslog.  Using this GET helper has the advantage that it has sane retrying etc built in for you already.<\/p>\n<p>It&#8217;s fairly early days for this code but it works and I am using it, I&#8217;ll probably be adding a few more features soon, let me know in comments if you need anything specific or even if you find it useful.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I&#8217;ve been working a bit on streamlining the builds I do on EC2 and wanted a better way to provision my machines. I use CentOS and things are pretty rough to non existent for nicely built EC2 images. I&#8217;ve used the Rightscale ones till now and while they&#8217;re nice they are also full of lots [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","footnotes":""},"categories":[1],"tags":[88,87,121,13],"_links":{"self":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/1580"}],"collection":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/comments?post=1580"}],"version-history":[{"count":11,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/1580\/revisions"}],"predecessor-version":[{"id":1590,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/1580\/revisions\/1590"}],"wp:attachment":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/media?parent=1580"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/categories?post=1580"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/tags?post=1580"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}