www.devco.net by r.i.pienaar

5Mar/088

Detailed Apache Stats

Apache has its native mod_status status page that many people use to pull stats into tools such as Cacti and other RRDTool based stats packages. This works well but does not always provide enough details, questions such as these remain unanswered:

  • How many of my requests are GET and how many are POST?
  • How many 404 errors and 5xx errors do I get on my site as a whole and for script.php specifically?
  • What is the average response time for the whole server, and for script.php?
  • How many Closed, Keep Alive and Aborted connections do I have?

To
answer this I wrote a script that keeps a running track of your Apache
process, it has many fine grained controls that let you fine tune
exactly what to keep stats on. I got the initial idea from an old ONLamp article titled Profiling LAMP Applications with Apache's Blackbox Logs.

The
article proposes a custom log format that provides the equivelant to an
airplanes blackbox, a flight recorder that records more detail per
request than the usual common log formats do. I suggest you read the
article for background information. The article though stops short of a
full data parser so I wrote one for a client who kindly agreed that I
can opensource it.

Using
this and some glue in my Cacti I now have graphs showing a profile of
the requests I receive for the whole site, but as you are able to
apply fine grained controls to select what exactly you'll see, you could get per server overview stats and details for just a specific scripts performance and statuses:

The script creates on a regular interval a file that contains the performance data, the data is presented in variable=value data pairs, I will soon provide a Cacti and Nagios plugin to parse this output to ease integration into these tools.

The
performance data includes values such as:

  •  Amount of requests in total
  • Total size
    of requests separated by in and out bytes
  • Average response time
  • Total processing time.
  • Counts of connections in Close, Keep Alive and Aborted
    states.
  • Counts for each valid HTTP Status code,
    and aggregates for 1xx, 2xx, 3xx, 4xx and 5xx.
  • The amount of GET and
    POST requests.
  • And detail for each and every unique request the server
    serves. 

See the Sample Stats
for a good example, variables are pretty self explanatory. To keep the
data set small and manageable 2 selectors exist, one to choose which
requests to keep details for and which to keep stats for. These can be
combined with standard Apache directives such as Location to provide very fine grained stats for all or a subset of your site.

You would need some glue to plug this into Cacti and Nagios, I will provide a script for this soon as I have time to write up some docs for it. 

Install guide etc can be found on my Wiki there is also extensive Perdoc Docs in the script, the Wiki also have links to downloading the script, the latest is always available here

Tagged as: , , , 8 Comments
25Aug/060

MySQL Query logger released

I previously blogged about a parser for pcap files that will show all SQL queries, you can now get the code for this tool here. It's very simple, help is in the comment block right at the beginning of the file.

Tagged as: , , No Comments
3Nov/054

Portaudit Central 1.0

Portaudit is intended to run daily against your ports directory identifying known vulnerabilities against a central database. Each night it produce an email that gets sent out and requires inspection.
The problem with this is with many FreeBSD hosts the emails can just be too many and I tend not to look at them.
Portaudit Central provides a means for delivering portaudit output to a central email box which will then produce a simple HTML based report of all machines. You can view a Sample Report produced using these scripts.
The server side script will call logger(1) with some useful diagnostic messages but the lines being logged will include some variables from the environment. I developed this under exim and the environment variables it logs are set by Exim. This will still work under another MTA, the worst that will happen is you'd have some logging entries thats missing details like the sender and message id.
I've taken some steps to prevent man-in-the-middle attacks. An attacker can in theory produce a report that says you have no vulnerabilities on a host when in fact you do have some. In order to combat this a few things are done:

  • Only 1 report per host per day, any reports after the first one will result in errors being logged, this should be a clear indication that you've either configured multiple clients with the same $hostname or someone is up to no good.
  • Each email being sent has a very simple crypto signature, the signature is basically a MD5 hash of the body of the message and a passphrase md5(body, passphrase) this means as long as your passphrases are secure - they aren't being sent along in the email so no-one can sniff them - someone else should not be able to produce a report that will pass this check regardless of report content. Even if your passphrase gets compromised you should be alerted about tampering by the log entries produced by the duplicate checking mechanism above. As long as you monitor your log files.

I've used this system myself now for around a month and have been quite happy, but no-one else has had a look at it yet so I'd appreciate feedback if anyone use it. In the future I hope to make the output from the report generator themed, right now it's very ugly but it does what it should. I will also move to a actual configuration file rather than editing some perl variables. I'm open to other suggestions.
Version 1.0 can be downloaded here: portauditcentral-current.tgz the full documentation including installation instructions can be found at the Wiki

Tagged as: , , 4 Comments
30Jan/055

Net::MovableType and Image::ExifTool

Posts for my photoblog are structured in a specific way. I put the image name in the entry body and the text in the extended entry. As long as I stick to that format my templates will automatically put in the right thumbnails, full size images etc.

While this is pretty simple to keep in mind and I have been doing it manually I wanted to automate things a bit. I also wanted to include some EXIF info for each image which Movable Type could not do on its own.

The solution was to write a simple blogging client in perl, read on for some details.

First off, lets cover the EXIF extraction, I found a excellent module in the FreeBSD ports system called Image::ExifTool. From its description:

ExifTool provides an extensible set of perl modules to read and write
meta information in image files. It reads EXIF, GPS, IPTC, XMP,
GeoTIFF, ICC Profile and Photoshop IRB meta information from JPEG,
TIFF, GIF, THM, CRW (Canon RAW), CR2 (Canon 1D Mk II RAW), MRW (Minolta
RAW), NEF (Nikon Electronic image Format), PEF (Pentax RAW), ORF
(Olympus RAW Format) and DNG (Digital Negative) images, as well as the
maker notes of many digital cameras by various manufacturers including
Canon, Casio, FujiFilm, Minolta/Konica-Minolta, Nikon, Olympus/Epson,
Panasonic/Leica, Pentax/Asahi, Sanyo and Sigma/Foveon. It writes EXIF,
GPS, IPTC, XMP and MakerNotes information to JPEG, TIFF, GIF, THM, CR2,
NEF, PEF and DNG files.

Using it is trivial, and you can read all the details at its CPAN Page. For me it was as simple as this:

my $exifTool = new Image::ExifTool;$exifInfo = $exifTool->ImageInfo("$photodir/$image");

$tagDescription = $exifTool->GetDescription($tag);

%tagValue = $exifTool->GetValue($tag);

Really simple, to see for yourself what information this will find
in a image you can run the included 'exiftool' script, here is some
sample output:

---- ExifTool ----
ExifTool Version Number : 3.72
---- File ----
File Name : 30012005.jpg
File Size : 67KB
File Type : JPG
Image Width : 366
Image Height : 550
---- EXIF ----
Software : Adobe Photoshop CS Windows
Flash : No Flash
Exif Image Width : 366
Exif Image Length : 550
Compression : JPEG (old-style)
Thumbnail Offset : 586
Thumbnail Length : 6137
---- IPTC ----
Application Record Version : 2
---- Photoshop ----
Photoshop Thumbnail : (Binary data 6137 bytes, use -b option to extract)
---- XMP ----
Version : 2.2
Raw File Name : DSC_4995.NEF
White Balance : As Shot
Exposure : +0.85
Shadows : 5
Brightness : 50
Contrast : +25
Saturation : 0
Sharpness : 25
Luminance Smoothing : 0
Color Noise Reduction : 25
Chromatic Aberration R : 0
Chromatic Aberration B : 0
Vignette Amount : 0
Shadow Tint : 0
Red Hue : 0
Red Saturation : 0
Green Hue : 0
Green Saturation : 0
Blue Hue : 0
Blue Saturation : 0
Tv(Shutter Speed) : 1/500
Shutter Speed Value : 1/500
Av(Aperture Value) : 5.6
Aperture Value : 5.6
Exposure Program : Program AE
Shooting Date/Time : 2005:01:23 13:58:27
Exposure Compensation : -0.33
Max Aperture Value : 5.7
Metering Mode : Multi-segment
Focal Length : 125.0mm
Focal Length In 35mm Format : 187
Color Space : Unknown (4294967295)
ISO Speed : 200
Flash Fired : False
Flash Return : No return detection
Flash Mode : Unknown (0)
Flash Function : False
Flash Red Eye Mode : False
Lens : 18.0-125.0 mm f/3.3-5.6
Make : NIKON CORPORATION
Camera Model Name : NIKON D70
Orientation : Horizontal (normal)
X Resolution : 300
Y Resolution : 300
Resolution Unit : inches
Creator Tool : Adobe Photoshop CS Windows
Date/Time Of Last Modification : 2005:01:23 17:21:22
Date/Time Of Digitization : 2005:01:23 17:21:22
Metadata Date : 2005:01:23 17:21:22
Derived From Instance ID : uuid:121a03ec-6d63-11d9-bfa3-cf79758759c4
Derived From Document ID : adobe:docid:photoshop:121a03eb-6d63-11d9-bfa3-cf79758759c4
Document ID : adobe:docid:photoshop:121a03ef-6d63-11d9-bfa3-cf79758759c4
Format : image/jpeg
---- Composite ----
Av(Aperture Value) : 5.6
Image Size : 366x550
Scale Factor To 35mm Equivalent : 1.5
Tv(Shutter Speed) : 1/500
Thumbnail Image : (Binary data 6137 bytes, use -b option to extract)
Focal Length : 125.0mm (35mm equivalent: 187.0mm)

This is very impressive, you can see multiple sets of data, example
is the Exposure, I shot it at -0.33 in the camera then bumped it to
+0.85 in Photoshop CS RAW Plugin.

Now that I have a good way to extract meta data from my images the next step was to do the blog posting, this was trivial using Net::MovableType, below is a quick code snippet, read all about it at CPAN

use Net::MovableType;

my $mt = new Net::MovableType('http://your.com/rsd.xml');
$mt->username('user');

$mt->password('secret');

$entry = {
   title => "Hello World from Net::MovableType",
   description => "Look ma, no hands!"
};

$postID = $mt->newPost($entry);
$mt->publishPost($postID);

Very simple, it has many other functions, you can read, edit,
categorise etc, all the stuff you can do from the standard Movable Type
API.

Tagged as: , Continue reading
13Aug/041

Calculating CIDR notation from IP ranges

Following from my previous post about blocking some more countries I discovered a bit of a short coming in the code I used to calculate CIDR notation from ranges of IP's. So a bit of searching on CPAN got me Net::CIDR.

use Net::CIDR;
$range = shift;
print (join("\n", Net::CIDR::range2cidr("$range")) . "\n" );

This will take any given range of ip address in format a.b.c.d-w.x.y.z and spew out a list of subnets required to cover the whole range:

# ./range2cidr.pl 64.139.147.0-64.139.170.255
64.139.147.0/24
64.139.148.0/22
64.139.152.0/21
64.139.160.0/21
64.139.168.0/23
64.139.170.0/24

So with this I now have hopefully a more accurate set of rules that will not block bits of New Zeeland as well by accident.

Tagged as: , , 1 Comment
27Aug/030

iScan

UPDATE: Exim now has excellent built-in spam and virus scanning, this performs better and is more effective, you should now use that. I've closed the project at SourceForge and you won't be able to get the files anymore.
This is a general mail manipulation system that features a powerfull plugin architecture using Perl. It was designed to work with Exim but plans are underway to possibly make it more MTA independant.