{"id":3740,"date":"2018-03-05T10:42:33","date_gmt":"2018-03-05T09:42:33","guid":{"rendered":"https:\/\/www.devco.net\/?p=3740"},"modified":"2018-03-05T10:47:15","modified_gmt":"2018-03-05T09:47:15","slug":"choria-progress-update","status":"publish","type":"post","link":"https:\/\/www.devco.net\/archives\/2018\/03\/05\/choria-progress-update.php","title":{"rendered":"Choria Progress Update"},"content":{"rendered":"

It’s been a while since I posted about Choria and where things are. There are major changes in the pipeline so it’s well overdue a update.<\/p>\n

The features mentioned here will become current in the next release cycle – about 2 weeks from now.<\/p>\n

New choria module<\/H2>
\nThe current gen Choria modules grew a bit organically and there’s a bit of a confusion between the various modules. I now have a new choria<\/a> module, it will consume features from the current modules and deprecate them. <\/p>\n

On the next release it can manage:<\/p>\n

    \n
  1. Choria YUM and APT repos<\/li>\n
  2. Choria Package<\/li>\n
  3. Choria Network Broker<\/li>\n
  4. Choria Federation Broker<\/li>\n
  5. Choria Data Adatpaters<\/li>\n<\/ol>\n

    Network Brokers<\/H2>
    \nWe have had amazing success with the NATS broker, lightweight, fast, stable. It’s perfect for Choria. While I had a pretty good module to configure it I wanted to create a more singular experience. Towards that there is a new Choria Broker<\/em> incoming that manages an embedded NATS instance.<\/p>\n

    To show what I am on about, imagine this is all that is required to configure a cluster of 3 production ready brokers capable of hosting 50k or more Choria managed nodes on modestly specced machines:<\/p>\n

    \r\nplugin.choria.broker_network = true\r\nplugin.choria.network.peers = nats:\/\/choria1.example.net:4223, nats:\/\/choria2.example.net:4223, nats:\/\/choria3.example.net:4223\r\nplugin.choria.stats_address = ::\r\n<\/pre>\n
    Of course there is Puppet code to do this for you in choria::broker<\/em>.<\/p>\n
    That’s it, start the choria-broker<\/em> daemon and you’re done – and ready to monitor it using Prometheus.  Like before it’s all TLS and all that kinds of good stuff.<\/p>\n
    Federation Brokers<\/H2>
    \nWe had good success with the Ruby Federation Brokers but they also had issues particularly around deployment as we had to deploy many instances of them and they tended to be quite big Ruby processes.<\/p>\n
    The same choria-broker<\/em> that hosts the Network Broker will now also host a new Golang based Federation Broker network.  Configuration is about the same as before you don’t need to learn new things, you just have to move to the configuration in choria::broker<\/em> and retire the old ones.<\/p>\n
    <\/center><\/p>\n
    Unlike the past where you had to run 2 or 3 of the Federation Brokers per node you now do not run any additional processes, you just enable the feature in the singular choria-broker<\/em>, you only get 1 process. Internally each run 10 instances of the Federation Broker, its much more performant and scalable.<\/p>\n
    Monitoring is done via Prometheus.<\/p>\n
    Data Adapters<\/H2>
    \nPreviously we had all kinds of fairly bad schemes to manage registration in MCollective.  The MCollective daemon would make requests<\/em> to a registration<\/em> agent, you’d designate one or more nodes as running this agent and so build either a file store, mongodb store etc.<\/p>\n
    This was fine at small size but soon enough the concurrency in large networks would overwhelm what could realistically be expected from the Agent mechanism to manage.<\/p>\n
    I’ve often wanted to revisit that but did not know what approach to take.  In the years since then the Stream Processing world has exploded with tools like Kafka, NATS Streaming and offerings from GPC, AWS and Azure etc.<\/p>\n
    Data Adapters are hosted in the Choria Broker and provide stateless, horizontally and vertically scalable Adapters that can take data from Choria and translate and publish them into other systems.<\/p>\n
    <\/center><\/p>\n
    Today I support NATS Streaming and the code is at first-iteration quality, problems I hope to solve with this:<\/p>\n