<\/p>\n\r\nadd_route(:name => \"graphite\", :type => [\"metric\", \"status\"]) do |event, routes|\r\n routes << \"stomp:\/\/\/queue\/events.graphite\" unless event.metrics.empty?\r\nend\r\n<\/pre>\n<\/code><\/p>\n
You can see from the code that we have access to the full event, a sample event is below, and we can make decisions based on any part of the event.<\/p>\n
<\/p>\n\r\n{\"name\":\"puppetmaster\",\r\n \"eventid\":\"4d9a33eb2bce3479f50a86e0\",\r\n \"text\":\"PROCS OK: 2 processes with command name puppetmasterd\",\r\n \"metrics\":{},\"tags\":{},\r\n \"subject\":\"monitor2.foo.net\",\r\n \"origin\":\"nagios bridge\",\r\n \"type\":\"status\",\r\n \"eventtime\":1301951466,\r\n \"severity\":0}\r\n<\/pre>\n<\/code><\/p>\n
This event is of type status<\/em> and has no metrics so it would not have been routed to the graphite system, while the event below has no status only metrics:<\/p>\n<\/p>\n\r\n{\"name\":\"um.bridge.nagios\",\r\n \"created_at\":1301940072,\r\n \"eventid\":\"4d9a34462bce3479f50a8839\",\r\n \"text\":\"um internal stats for um.bridge.nagios\",\r\n \"metrics\":{\"events\":49},\r\n \"tags\":{\"version\":\"0.0.1\"},\r\n \"subject\":\"monitor2.foo.net\",\r\n \"origin\":\"um stat\",\r\n \"type\":\"metric\",\r\n \"extended_data\":{\"start_time\":1301940072},\r\n \"eventtime\":1301951558,\"severity\":0}\r\n<\/pre>\n<\/code><\/p>\n
By simply supplying this route:<\/p>\n
<\/p>\n\r\nadd_route(:name => \"um_status\", :type => [\"status\", \"alert\"]) do |event, routes|\r\n routes << \"stomp:\/\/\/queue\/events.status\"\r\nend\r\n<\/pre>\n<\/code><\/p>\n
I can be sure this non status bearing event of type metric<\/em> wouldn't reach the status system where it will just waste resources.<\/p>\nYou can see the routing system is very simple and sit at the emitting side of every part of the system. If you wanted to inject code between the main Processor and Graphite here simply route the events to your code and then back into graphite when you're done transforming the events. As long as you can speak to middleware and process JSON you can inject logic into the flow of events.<\/p>\n
I hope this will give me a totally composable infrastructure, I think the routes are trivial enough that almost anyone can write and tweak them and since I am using the most simplest of technologies like JSON almost any language can be used to plug into this framework and consume the events. Routes can be put into the system without restarting anything, just drop the files down and touch a trigger file - the routes will immediately become active.<\/p>\n
The last example I want to show is the development route I mentioned above that siphons off roughly 10% of the firehose into your dev systems:<\/p>\n
<\/p>\n\r\nadd_route(:name => \"development\", :type => \"*\") do |event, routes|\r\n routes << \"stomp:\/\/\/topic\/development.portal\" if rand(10) == 1\r\nend\r\n<\/pre>\n<\/code><\/p>\n
Here I am picking all event types, I am dumping it into a topic<\/em> called development.portal<\/em> but only in roughly 10% of cases. We're using a topic since they dont buffer or store or consume much memory when the development system is down - events will just be lost when the dev system is down. <\/p>\nI'd simply drop this into \/etc\/um\/routes\/portal\/development.rb<\/em> to configure my production portal<\/em> to emit raw events to my development event portal.<\/p>\nThat's all for today, as mentioned this stuff is old technology and nothing here really solves new problems but I think the simplicity of the routing system and how it allows people without huge amounts of knowledge to re-compose code I wrote in new and interesting ways is quite novel in the sysadmin tool space that's all too rigid and controlled.<\/p>\n","protected":false},"excerpt":{"rendered":"
I’ve been working on rewriting my proof of concept code into actual code I might not feel ashamed to show people, this is quite a slow process so hang in there. If you’ve been reading my previous posts you’ll know I aim to write a framework for monitoring and event correlation. This is a surprisingly […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","footnotes":""},"categories":[7],"tags":[121,85,64,13,96],"_links":{"self":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/1998"}],"collection":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/comments?post=1998"}],"version-history":[{"count":17,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/1998\/revisions"}],"predecessor-version":[{"id":2016,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/posts\/1998\/revisions\/2016"}],"wp:attachment":[{"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/media?parent=1998"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/categories?post=1998"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devco.net\/wp-json\/wp\/v2\/tags?post=1998"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![](\"http:\/\/www.devco.net\/images\/simple-monitor.png\")
In the most basic form it just take events, archive them, process metrics, process status and raise alerts. Most people will recognize these big parts in their monitoring systems but will also agree there is a lot more to it than this. What describes the extra bits will almost never be answered in a single description as we all have unique needs and ideas.<\/p>\n![](\"http:\/\/www.devco.net\/images\/routable-endpoints.png\")
<\/p>\n