Apache has its native mod_status status page that many people use to pull stats into tools such as Cacti and other RRDTool based stats packages. This works well but does not always provide enough details, questions such as these remain unanswered:
- How many of my requests are GET and how many are POST?
- How many 404 errors and 5xx errors do I get on my site as a whole and for script.php specifically?
- What is the average response time for the whole server, and for script.php?
- How many Closed, Keep Alive and Aborted connections do I have?
To answer this I wrote a script that keeps a running track of your Apache process, it has many fine grained controls that let you fine tune exactly what to keep stats on. I got the initial idea from an old ONLamp article titled Profiling LAMP Applications with Apache’s Blackbox Logs.
The article proposes a custom log format that provides the equivelant to an airplanes blackbox, a flight recorder that records more detail per request than the usual common log formats do. I suggest you read the article for background information. The article though stops short of a full data parser so I wrote one for a client who kindly agreed that I can opensource it.
Using this and some glue in my Cacti I now have graphs showing a profile of the requests I receive for the whole site, but as you are able to apply fine grained controls to select what exactly you’ll see, you could get per server overview stats and details for just a specific scripts performance and statuses:
The script creates on a regular interval a file that contains the performance data, the data is presented in variable=value data pairs, I will soon provide a Cacti and Nagios plugin to parse this output to ease integration into these tools.
The performance data includes values such as:
- Amount of requests in total
- Total size of requests separated by in and out bytes
- Average response time
- Total processing time.
- Counts of connections in Close, Keep Alive and Aborted states.
- Counts for each valid HTTP Status code, and aggregates for 1xx, 2xx, 3xx, 4xx and 5xx.
- The amount of GET and POST requests.
- And detail for each and every unique request the server serves.
See the Sample Stats for a good example, variables are pretty self explanatory. To keep the data set small and manageable 2 selectors exist, one to choose which requests to keep details for and which to keep stats for. These can be combined with standard Apache directives such as Location to provide very fine grained stats for all or a subset of your site.
You would need some glue to plug this into Cacti and Nagios, I will provide a script for this soon as I have time to write up some docs for it.
Install guide etc can be found on my Wiki there is also extensive Perdoc Docs in the script, the Wiki also have links to downloading the script, the latest is always available here
Hi,
really nice script. Do you have template for Cacti?
Thanks for your comment, but no I dont have a template I’d be happy to share yet as its a bit rough, and as I mentioned you’ll need a tool to extract the values from the dataset. I might release that soon.
The keepalives seem to always be counting up, I never see a close. Is this as expected?
My bad, I had the output going to a directory that was not owned by apache. It was interesting because it was putting things to the file, but not updating.
Very nice tool.
To make apache blackbox work with cacti took some effort.
The hard part was putting a system in place to collect the relevant information, downloading the entire stats file every so often didn’t scale for my situation, so I used an SNMP based solution.
I don’t think cacti needs the output fields in any particular order, but make sure they are named the same.
1. On the apache host, create a perl script to parse blackbox.txt for the relevant fields you want cacti to poll.
2. Follow steps from here to setup cacti portion: http://blog.bitpusher.com/2008/05/30/custom-graphing-with-cacti/
NOTE: You need to set your inputs as a COUNTER, not gauge. COUNTER will take the difference from one poll to the next, which is how blackbox stores its stats.
Nice, thanks for that Daniel, in my case I have a script that fetches the whole blackbox file once and caches it for up to 4.5 minutes on the cacti machine in /tmp.
All subsequent requests for values within the 4.5 minutes will query the local copy of the file, so this makes sure each apache only gets hit for stats once every 5 minutes
The same cache file also feeds my nagios – in the cases where nagios and cacti shares a server this helps a lot.
That solved it better for me than a SNMP solution since i only ever communicate with the host in question once,
Hi Daniel,
Many thanks for your great script. I’ve managed to collect the data and make it available through SNMP. However I’m having much trouble with getting it plotted in Cacti.
Would you be willing to share or send me by mail the template or code you used to create the graphs above?
Many thanks in advance!
Kind regards,
Peter
Yep I thinks its time the master, to add templates and the sdditional fun stuff for apache and make literally the best apache cati plugin out there!!