I’ve often wondered how things will change in a world where everything is a REST API and how relevant our Unix CLI tool chain will be in the long run. I’ve known we needed CLI ways to interact with data – like JSON data – and have given this a lot of thought.<\/p>\n
MS Powershell does some pretty impressive object parsing on their CLI but I was never really sure how close we could get to that in Unix. I’ve wanted to start my journey with the grep<\/em> utility as that seemed a natural starting point and my most used CLI tool.<\/p>\n I have no idea how to write parsers and matchers but luckily I have a very talented programmer working for me<\/a> who were able to take my ideas and realize them awesomely. Pieter wrote a json grep and I want to show off a few bits of what it can do.<\/p>\n I’ll work with the document below:<\/p>\n There are a few interesting things to note about this data:<\/p>\n A simple grep for a single key in the document: <\/code><\/p>\n We can extract a single key from the result: <\/code><\/p>\n A simple grep for 2 keys in the document: <\/code><\/p>\n The nested document pose a problem though, if we were to search for contacts.protocol=twitter and contacts.address=1234567890<\/em> we will get both documents and not none, that’s because in order to effectively search the sub documents we need to ensure that these 2 values exist in the same sub document.<\/p>\n <\/code><\/p>\n Placing []<\/em> around the 2 terms works like ()<\/em> but restricts the search to the specific sub document. In this case there is no sub document in the contacts array that has both twitter and 1234567890.<\/p>\n Of course you can have many search terms:<\/p>\n <\/code><\/p>\n We can also construct entirely new documents:<\/p>\n <\/code><\/p>\n So I am adding JSON output support to MCollective, today I was rolling out a new Nagios check script to my nodes and wanted to be sure they all had it. I used the File Manager<\/a> agent to fetch the stats for my file from all the machines then printed the ones that didn’t match my expected MD5.<\/p>\n <\/code><\/p>\n Eventually you will be able to then pipe this output to mco<\/em> again and call another agent, here I take all the machines that didn’t yet have the right file and cause a puppet run to happen on them, this is very Powershell like and the eventual use case I am building this for:<\/p>\n <\/code><\/p>\n I also wanted to know the total size of a logfile across my web servers to be sure I would have enough space to copy them all:<\/p>\n <\/code><\/p>\n Now how about interacting with a webservice like the GitHub API:<\/p>\n <\/code><\/p>\n Here I fetched the most recent commits in the marionette-collective<\/em> GitHub repository, searched for ones by Pieter and returns the ID of those commits. The –start argument is needed because the top of the JSON returned is not the array we care for. The –start tells jgrep to take the commits<\/em> key and grep that.<\/p>\n Or since it’s Sysadmin Appreciation Day how about tweets about it:<\/p>\n <\/code><\/p>\n Here as before we have to grep the results<\/em> array that is contained inside the results.<\/p>\n I can also find all the restaurants near my village via SimpleGEO:<\/p>\n <\/code><\/p>\n There's a lot more I didn't show, it supports all the usual <=<\/em> etc operators and a fair few other bits.<\/p>\n You can get this utility by installing the jgrep<\/a> Ruby Gem or grab the code from GitHub<\/a>. The Gem is a library so you can use these abilities in your ruby programs but also includes the CLI tool shown here. <\/p>\n\r\n[\r\n {\"name\":\"R.I.Pienaar\",\r\n \"contacts\": [\r\n {\"protocol\":\"twitter\", \"address\":\"ripienaar\"},\r\n {\"protocol\":\"email\", \"address\":\"rip@devco.net\"},\r\n {\"protocol\":\"msisdn\", \"address\":\"1234567890\"}\r\n ]\r\n },\r\n {\"name\":\"Pieter Loubser\",\r\n \"contacts\": [\r\n {\"protocol\":\"twitter\", \"address\":\"pieterloubser\"},\r\n {\"protocol\":\"email\", \"address\":\"foo@example.com\"},\r\n {\"protocol\":\"msisdn\", \"address\":\"1234567890\"}\r\n ]\r\n }\r\n]<\/pre>\n\n
Examples<\/H3>
\nThe examples below show a few possible grep use cases:<\/p>\n
\n<\/p>\n\r\n$ cat example.json | jgrep \"name='R.I.Pienaar'\"\r\n[\r\n {\"name\":\"R.I.Pienaar\",\r\n \"contacts\": [\r\n {\"protocol\":\"twitter\", \"address\":\"ripienaar\"},\r\n {\"protocol\":\"email\", \"address\":\"rip@devco.net\"},\r\n {\"protocol\":\"msisdn\", \"address\":\"1234567890\"}\r\n ]\r\n }\r\n]\r\n<\/pre>\n
\n<\/p>\n\r\n$ cat example.json | jgrep \"name='R.I.Pienaar'\" -s name\r\nR.I.Pienaar\r\n<\/pre>\n
\n<\/p>\n\r\n% cat example.json | \r\n jgrep \"name='R.I.Pienaar' and contacts.protocol=twitter\" -s name\r\nR.I.Pienaar\r\n<\/pre>\n
<\/p>\n\r\n$ cat example.json | \r\n jgrep \"[contacts.protocol=twitter and contacts.address=1234567890]\"\r\n<\/pre>\n
<\/p>\n\r\n% cat example.json | \r\n jgrep \"[contacts.protocol=twitter and contacts.address=1234567890] or name='R.I.Pienaar'\" -s name\r\nR.I.Pienaar\r\n<\/pre>\n
<\/p>\n\r\n% cat example.json | jgrep \"name='R.I.Pienaar'\" -s \"name contacts.address\"\r\n[\r\n {\r\n \"name\": \"R.I.Pienaar\",\r\n \"contacts.address\": [\r\n \"ripienaar\",\r\n \"rip@devco.net\",\r\n \"1234567890\"\r\n ]\r\n }\r\n]\r\n<\/pre>\nReal World<\/H3><\/p>\n
<\/p>\n\r\n$ mco rpc filemgr status file=\/...\/check_puppet.rb -j | \r\n jgrep 'data.md5!=a4fdf7a8cc756d0455357b37501c24b5' -s sender\r\nbox1.example.com\r\n<\/pre>\n
<\/p>\n\r\n$ mco rpc filemgr status file=\/...\/check_puppet.rb -j | \r\n jgrep 'data.md5!=a4fdf7a8cc756d0455357b37501c24b5' |\r\n mco rpc puppetd runonce\r\n<\/pre>\n
<\/p>\n\r\n$ mco rpc filemgr status file=\/var\/log\/httpd\/access_log -W \/apache\/ -j |\r\n jgrep -s \"data.size\"|\r\n awk '{ SUM += $1} END { print SUM\/1024\/1024 \" MB\"}'\r\n2757.9093 MB\r\n<\/pre>\n<\/p>\n\r\n$ curl -s http:\/\/github.com\/api\/v2\/json\/commits\/list\/puppetlabs\/marionette-collective\/master|\r\n jgrep --start commits \"author.name='Pieter Loubser'\" -s id\r\n52470fee0b9fe14fb63aeb344099d0c74eaf7513\r\n<\/pre>\n
<\/p>\n\r\n% curl -s \"http:\/\/search.twitter.com\/search.json?q=sysadminday\"|\r\n jgrep --start results -s \"text\"\r\n\r\nRT @RedHat_Training: Did you know that today is Systems Admin Day? A big THANK YOU to all our system admins! Here's to you! http:\/\/t.co\/ZQk8ifl\r\nRT @SinnerBOFH: #BOFHers RT @linuxfoundation: Happy #SysAdmin Day! You know who you are, rock stars. http:\/\/t.co\/kR0dhhc #linux\r\nRT @google: Hey, sysadmins - thanks for all you do. May your pagers be silent and your users be clueful today! http:\/\/t.co\/N2XzFgw\r\nRT @google: Hey, sysadmins - thanks for all you do. May your pagers be silent and your users be clueful today! http:\/\/t.co\/y9TbCqb #sysadminday\r\nRT @mfujiwara: http:\/\/www.sysadminday.com\/\r\nRT @mitchjoel: It's SysAdmin Day! Have you hugged your SysAdmin today? Make sure all employees follow the rules: http:\/\/bit.ly\/17m98z #humor\r\n? @mfujiwara: http:\/\/www.sysadminday.com\/\r\n<\/pre>\n
<\/p>\n\r\ncurl -x localhost:8001 -s \"http:\/\/api.simplegeo.com\/1.0\/places\/51.476959,0.006759.json?category=Restaurant\"|\r\n jgrep --start features \"properties.distance<2.0\" -s \"properties.address \\\r\n properties.name \\\r\n properties.postcode \\\r\n properties.phone \\\r\n properties.distance\"\r\n[\r\n {\r\n \"properties.address\": \"15 Stratheden Road\",\r\n \"properties.distance\": 0.773576114771768,\r\n \"properties.phone\": \"+44 20 8858 8008\",\r\n \"properties.name\": \"The Lamplight\",\r\n \"properties.postcode\": \"SE3 7TH\"\r\n },\r\n {\r\n \"properties.address\": \"9 Stratheden Parade\",\r\n \"properties.distance\": 0.870622234751732,\r\n \"properties.phone\": \"+44 20 8858 0728\",\r\n \"properties.name\": \"Sun Ya\",\r\n \"properties.postcode\": \"SE3 7SX\"\r\n }\r\n]\r\n<\/pre>\n