{"id":3699,"date":"2017-10-09T08:37:31","date_gmt":"2017-10-09T07:37:31","guid":{"rendered":"https:\/\/www.devco.net\/?p=3699"},"modified":"2017-10-09T08:37:31","modified_gmt":"2017-10-09T07:37:31","slug":"the-choria-emulator","status":"publish","type":"post","link":"https:\/\/www.devco.net\/archives\/2017\/10\/09\/the-choria-emulator.php","title":{"rendered":"The Choria Emulator"},"content":{"rendered":"<p>In my <a href=\"https:\/\/www.devco.net\/archives\/2017\/09\/22\/what-to-consider-when-speccing-a-choria-network-size.php\">previous<\/a> <a href=\"https:\/\/www.devco.net\/archives\/2017\/09\/19\/load-testing-choria.php\">posts<\/a> I discussed what goes into load testing a Choria network, what connections are made, subscriptions are made etc.<\/p>\n<p>From this it&#8217;s obvious the things we should be able to emulate are:<\/p>\n<ul>\n<li>Connections to NATS<\/li>\n<li>Subscriptions &#8211; which implies number of agents and sub collectives<\/li>\n<li>Message payload sizes<\/li>\n<\/ul>\n<p>To make it realistically affordable to emulate many more machines that I have I made an emulator that can start numbers of Choria daemons on a single node.<\/p>\n<p>I&#8217;ve been slowly rewriting MCollective daemon side in Go which means I already had all the networking and connectors available there, so a daemon was written:<\/p>\n<pre lang=\"text\">\r\nusage: choria-emulator --instances=INSTANCES [<flags>]\r\n\r\nEmulator for Choria Networks\r\n\r\nFlags:\r\n      --help                 Show context-sensitive help (also try --help-long and --help-man).\r\n      --version              Show application version.\r\n      --name=\"\"              Instance name prefix\r\n  -i, --instances=INSTANCES  Number of instances to start\r\n  -a, --agents=1             Number of emulated agents to start\r\n      --collectives=1        Number of emulated subcollectives to create\r\n  -c, --config=CONFIG        Choria configuration file\r\n      --tls                  Enable TLS on the NATS connections\r\n      --verify               Enable TLS certificate verifications on the NATS connections\r\n      --server=SERVER ...    NATS Server pool, specify multiple times (eg one:4222)\r\n  -p, --http-port=8080       Port to listen for \/debug\/vars\r\n<\/pre>\n<p>You can see here it takes a number of instances, agents and collectives.  The instances will all respond with <em>${name}-${instance}<\/em> on any mco ping or RPC commands.  It can be discovered using the normal mc discovery &#8211; though only supports agent and identity filters.<\/p>\n<p>Every instance will be a Choria daemon with the exact same network connection and NATS subscriptions as real ones.  Thus 50 000 emulated Choria will put the exact same load of work on your NATS brokers as would normal ones, performance wise even with high concurrency the emulator performs quite well &#8211; it&#8217;s many orders of magnitude faster than the ruby Choria client anyway so it&#8217;s real enough.<\/p>\n<p>The agents they start are all copies of this one:<\/p>\n<pre lang=\"text\">\r\nemulated0\r\n=========\r\n\r\nChoria Agent emulated by choria-emulator\r\n\r\n      Author: R.I.Pienaar <rip@devco.net>\r\n     Version: 0.0.1\r\n     License: Apache-2.0\r\n     Timeout: 120\r\n   Home Page: http:\/\/choria.io\r\n\r\n   Requires MCollective 2.9.0 or newer\r\n\r\nACTIONS:\r\n========\r\n   generate\r\n\r\n   generate action:\r\n   ----------------\r\n       Generates random data of a given size\r\n\r\n       INPUT:\r\n           size:\r\n              Description: Amount of text to generate\r\n                   Prompt: Size\r\n                     Type: integer\r\n                 Optional: true\r\n            Default Value: 20\r\n\r\n\r\n       OUTPUT:\r\n           message:\r\n              Description: Generated Message\r\n               Display As: Message\r\n<\/pre>\n<p>You can this has a basic data generator action &#8211; you give it a desired size and it makes you a message that size.  It will run as many of these as you wish all called like <em>emulated0<\/em> etc.<\/p>\n<p>It has an mcollective agent that go with it, the idea is you create a pool of machines all with your normal mcollective on it and this agent.  Using that agent then you build up a different new mcollective network comprising the emulators, federation and NATS.<\/p>\n<p>Here&#8217;s some example of commands &#8211; you&#8217;ll see these later again when we talk about scenarios:<\/p>\n<p>We download the dependencies onto all our nodes:<\/p>\n<pre lang=\"text\">\r\n$ mco playbook run setup-prereqs.yaml --emulator_url=https:\/\/example.net\/rip\/choria-emulator-0.0.1 --gnatsd_url=https:\/\/example.net\/rip\/gnatsd --choria_url=https:\/\/example.net\/rip\/choria\r\n<\/pre>\n<p>We start NATS on our first node:<\/p>\n<pre lang=\"text\">\r\n$ mco playbook run start-nats.yaml --monitor 8300 --port 4300 -I test1.example.net\r\n<\/pre>\n<p>We start the emulator with 1500 instances per node all pointing to our above NATS:<\/p>\n<pre lang=\"text\">\r\n$ mco playbook run start-emulator.yaml --agents 10 --collectives 10 --instances 750 --monitor 8080 --servers 192.168.1.1:4300\r\n<\/pre>\n<p>You&#8217;ll then setup a client config for the built network and can interact with it using normal mco stuff and the test suite I&#8217;ll show later. Simularly there are playbooks to stop all the various parts etc.  The playbooks just interact with the mcollective agent so you could use mco rpc directly too.<\/p>\n<p>I found I can easily run 700 to 1000 instances on basic VMs &#8211; needs like 1.5GB RAM &#8211; so it&#8217;s fairly light.  Using 400 nodes I managed to build a 300 000 node Choria network and could easily interact with it etc.<\/p>\n<p>Finally I made a ec2 environment where you can stand up a Puppet Master, Choria, the emulator and everything you need and do load tests on your own dime.  I was able to do many runs with 50 000 emulated nodes on EC2 and the whole lot cost me less than $20.<\/p>\n<p>The code for this emulator is very much a work in progress as is the Go code for the Choria protocol and networking but the emulator <a href=\"https:\/\/github.com\/choria-io\/choria-emulator\">is here if you want to take a peek<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In my previous posts I discussed what goes into load testing a Choria network, what connections are made, subscriptions are made etc. From this it&#8217;s obvious the things we should be able to emulate are: Connections to NATS Subscriptions &#8211; which implies number of agents and sub collectives Message payload sizes To make it realistically [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","footnotes":""},"categories":[7],"tags":[126,121,85,78,21],"_links":{"self":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/3699"}],"collection":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/comments?post=3699"}],"version-history":[{"count":7,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/3699\/revisions"}],"predecessor-version":[{"id":3706,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/3699\/revisions\/3706"}],"wp:attachment":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/media?parent=3699"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/categories?post=3699"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/tags?post=3699"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}