R.I.Pienaar

Tag Archives: perl

MCollective and other languages

by R.I. Pienaar on 07. Aug, 2010 in Code

I often get asked about MCollective and other programming languages. Thus far we only support Ruby but my hope is in time we’ll be able to be more generic.

Initially I had a few requirements from serialization:

It must retain data types
Encoding the same data – like a hash – twice should give the same result from the point of view of md5()

That was about it really. This was while we used a pre-shared key to validate requests and so the result of the encode and decode should be the same on the sender as on the receiver. With YAML this was never the case so I used Marshal.

We recently had a SSL based security plugin contributed that relaxed the 2nd requirement so we can go back to using YAML. We could in theory relax the 1st requirement but it would just inhibit the kind of tools you can build with MCollective quite a bit. So I’d strongly suggest this is a must have.

Today there are very few cross language serializers that let you just deal with arbitrary data YAML is one that seems to have a log of language support. Prior to version 1.0.0 of MCollective the SSL security system only supported Marshal but we’ll support YAML in addition to Marshal in 1.0.0.

This enabled me to write a Perl client that speaks to your standard Ruby collective (if it runs this new plugin).

You can see the Perl client here. The Perl code is roughly a mc-find-hosts written in Perl and without command line options for filtering – though you can just adjust the filters in the code. It’s been years since I wrote any Perl so that’s just the first thing that worked for me.

Point is someone should be able to take any language that has the Syck YAML bindings and write a client library to talk with Mcollective. I tried the non Syck bindings in PHP and it’s unusable, I suspect the PHP Syck bindings will work better but I didn’t try them.

As mentioned on the user list post 1.0.0 I intend to focus on long running and scheduled requests I’ll then also work on some kind of interface between Mcollective and Agents written in other languages – since that is more or less how long running scheduled tasks would work anyway. This will then use the Ruby as a transport hooking clients and agents in different languages together.

I can see that I’ll enable this but I am very unlikely to write the clients myself. I am therefore keen to speak to community members who want to speak to MCollective from languages like Python and who have some time to work on this.

Read full story • Comments { 2 }

Detailed Apache Stats

by R.I. Pienaar on 05. Mar, 2008 in Code, Front Page

Apache has its native mod_status status page that many people use to pull stats into tools such as Cacti and other RRDTool based stats packages. This works well but does not always provide enough details, questions such as these remain unanswered:

How many of my requests are GET and how many are POST?
How many 404 errors and 5xx errors do I get on my site as a whole and for script.php specifically?
What is the average response time for the whole server, and for script.php?
How many Closed, Keep Alive and Aborted connections do I have?

To answer this I wrote a script that keeps a running track of your Apache process, it has many fine grained controls that let you fine tune exactly what to keep stats on. I got the initial idea from an old ONLamp article titled Profiling LAMP Applications with Apache’s Blackbox Logs.

The article proposes a custom log format that provides the equivelant to an airplanes blackbox, a flight recorder that records more detail per request than the usual common log formats do. I suggest you read the article for background information. The article though stops short of a full data parser so I wrote one for a client who kindly agreed that I can opensource it.

Using this and some glue in my Cacti I now have graphs showing a profile of the requests I receive for the whole site, but as you are able to apply fine grained controls to select what exactly you’ll see, you could get per server overview stats and details for just a specific scripts performance and statuses:

The script creates on a regular interval a file that contains the performance data, the data is presented in variable=value data pairs, I will soon provide a Cacti and Nagios plugin to parse this output to ease integration into these tools.

The performance data includes values such as:

Amount of requests in total
Total size of requests separated by in and out bytes
Average response time
Total processing time.
Counts of connections in Close, Keep Alive and Aborted states.
Counts for each valid HTTP Status code, and aggregates for 1xx, 2xx, 3xx, 4xx and 5xx.
The amount of GET and POST requests.
And detail for each and every unique request the server serves.

See the Sample Stats for a good example, variables are pretty self explanatory. To keep the data set small and manageable 2 selectors exist, one to choose which requests to keep details for and which to keep stats for. These can be combined with standard Apache directives such as Location to provide very fine grained stats for all or a subset of your site.

You would need some glue to plug this into Cacti and Nagios, I will provide a script for this soon as I have time to write up some docs for it.

Install guide etc can be found on my Wiki there is also extensive Perdoc Docs in the script, the Wiki also have links to downloading the script, the latest is always available here

Read full story • Comments { 8 }

MySQL Query logger released

by R.I. Pienaar on 25. Aug, 2006 in Code

I previously blogged about a parser for pcap files that will show all SQL queries, you can now get the code for this tool here. It’s very simple, help is in the comment block right at the beginning of the file.

Read full story • Comments { 0 }

Portaudit Central 1.0

by R.I. Pienaar on 03. Nov, 2005 in Code

Portaudit is intended to run daily against your ports directory identifying known vulnerabilities against a central database. Each night it produce an email that gets sent out and requires inspection.
The problem with this is with many FreeBSD hosts the emails can just be too many and I tend not to look at them.
Portaudit Central provides a means for delivering portaudit output to a central email box which will then produce a simple HTML based report of all machines. You can view a Sample Report produced using these scripts.
The server side script will call logger(1) with some useful diagnostic messages but the lines being logged will include some variables from the environment. I developed this under exim and the environment variables it logs are set by Exim. This will still work under another MTA, the worst that will happen is you’d have some logging entries thats missing details like the sender and message id.
I’ve taken some steps to prevent man-in-the-middle attacks. An attacker can in theory produce a report that says you have no vulnerabilities on a host when in fact you do have some. In order to combat this a few things are done:

Only 1 report per host per day, any reports after the first one will result in errors being logged, this should be a clear indication that you’ve either configured multiple clients with the same $hostname or someone is up to no good.
Each email being sent has a very simple crypto signature, the signature is basically a MD5 hash of the body of the message and a passphrase md5(body, passphrase) this means as long as your passphrases are secure – they aren’t being sent along in the email so no-one can sniff them – someone else should not be able to produce a report that will pass this check regardless of report content. Even if your passphrase gets compromised you should be alerted about tampering by the log entries produced by the duplicate checking mechanism above. As long as you monitor your log files.

I’ve used this system myself now for around a month and have been quite happy, but no-one else has had a look at it yet so I’d appreciate feedback if anyone use it. In the future I hope to make the output from the report generator themed, right now it’s very ugly but it does what it should. I will also move to a actual configuration file rather than editing some perl variables. I’m open to other suggestions.
Version 1.0 can be downloaded here: portauditcentral-current.tgz the full documentation including installation instructions can be found at the Wiki

Read full story • Comments { 4 }

Net::MovableType and Image::ExifTool

by R.I. Pienaar on 30. Jan, 2005 in Code

Posts for my photoblog are structured in a specific way. I put the image name in the entry body and the text in the extended entry. As long as I stick to that format my templates will automatically put in the right thumbnails, full size images etc.

While this is pretty simple to keep in mind and I have been doing it manually I wanted to automate things a bit. I also wanted to include some EXIF info for each image which Movable Type could not do on its own.

The solution was to write a simple blogging client in perl, read on for some details.

First off, lets cover the EXIF extraction, I found a excellent module in the FreeBSD ports system called Image::ExifTool. From its description:

ExifTool provides an extensible set of perl modules to read and write
meta information in image files. It reads EXIF, GPS, IPTC, XMP,
GeoTIFF, ICC Profile and Photoshop IRB meta information from JPEG,
TIFF, GIF, THM, CRW (Canon RAW), CR2 (Canon 1D Mk II RAW), MRW (Minolta
RAW), NEF (Nikon Electronic image Format), PEF (Pentax RAW), ORF
(Olympus RAW Format) and DNG (Digital Negative) images, as well as the
maker notes of many digital cameras by various manufacturers including
Canon, Casio, FujiFilm, Minolta/Konica-Minolta, Nikon, Olympus/Epson,
Panasonic/Leica, Pentax/Asahi, Sanyo and Sigma/Foveon. It writes EXIF,
GPS, IPTC, XMP and MakerNotes information to JPEG, TIFF, GIF, THM, CR2,
NEF, PEF and DNG files.

Using it is trivial, and you can read all the details at its CPAN Page. For me it was as simple as this:

my $exifTool = new Image::ExifTool;$exifInfo = $exifTool->ImageInfo("$photodir/$image");

$tagDescription = $exifTool->GetDescription($tag);

%tagValue = $exifTool->GetValue($tag);

Really simple, to see for yourself what information this will find
in a image you can run the included ‘exiftool’ script, here is some
sample output:

—- ExifTool —-
ExifTool Version Number : 3.72
—- File —-
File Name : 30012005.jpg
File Size : 67KB
File Type : JPG
Image Width : 366
Image Height : 550
—- EXIF —-
Software : Adobe Photoshop CS Windows
Flash : No Flash
Exif Image Width : 366
Exif Image Length : 550
Compression : JPEG (old-style)
Thumbnail Offset : 586
Thumbnail Length : 6137
—- IPTC —-
Application Record Version : 2
—- Photoshop —-
Photoshop Thumbnail : (Binary data 6137 bytes, use -b option to extract)
—- XMP —-
Version : 2.2
Raw File Name : DSC_4995.NEF
White Balance : As Shot
Exposure : +0.85
Shadows : 5
Brightness : 50
Contrast : +25
Saturation : 0
Sharpness : 25
Luminance Smoothing : 0
Color Noise Reduction : 25
Chromatic Aberration R : 0
Chromatic Aberration B : 0
Vignette Amount : 0
Shadow Tint : 0
Red Hue : 0
Red Saturation : 0
Green Hue : 0
Green Saturation : 0
Blue Hue : 0
Blue Saturation : 0
Tv(Shutter Speed) : 1/500
Shutter Speed Value : 1/500
Av(Aperture Value) : 5.6
Aperture Value : 5.6
Exposure Program : Program AE
Shooting Date/Time : 2005:01:23 13:58:27
Exposure Compensation : -0.33
Max Aperture Value : 5.7
Metering Mode : Multi-segment
Focal Length : 125.0mm
Focal Length In 35mm Format : 187
Color Space : Unknown (4294967295)
ISO Speed : 200
Flash Fired : False
Flash Return : No return detection
Flash Mode : Unknown (0)
Flash Function : False
Flash Red Eye Mode : False
Lens : 18.0-125.0 mm f/3.3-5.6
Make : NIKON CORPORATION
Camera Model Name : NIKON D70
Orientation : Horizontal (normal)
X Resolution : 300
Y Resolution : 300
Resolution Unit : inches
Creator Tool : Adobe Photoshop CS Windows
Date/Time Of Last Modification : 2005:01:23 17:21:22
Date/Time Of Digitization : 2005:01:23 17:21:22
Metadata Date : 2005:01:23 17:21:22
Derived From Instance ID : uuid:121a03ec-6d63-11d9-bfa3-cf79758759c4
Derived From Document ID : adobe:docid:photoshop:121a03eb-6d63-11d9-bfa3-cf79758759c4
Document ID : adobe:docid:photoshop:121a03ef-6d63-11d9-bfa3-cf79758759c4
Format : image/jpeg
—- Composite —-
Av(Aperture Value) : 5.6
Image Size : 366×550
Scale Factor To 35mm Equivalent : 1.5
Tv(Shutter Speed) : 1/500
Thumbnail Image : (Binary data 6137 bytes, use -b option to extract)
Focal Length : 125.0mm (35mm equivalent: 187.0mm)

This is very impressive, you can see multiple sets of data, example
is the Exposure, I shot it at -0.33 in the camera then bumped it to
+0.85 in Photoshop CS RAW Plugin.

Now that I have a good way to extract meta data from my images the next step was to do the blog posting, this was trivial using Net::MovableType, below is a quick code snippet, read all about it at CPAN

use Net::MovableType;

my $mt = new Net::MovableType(‘http://your.com/rsd.xml’);
$mt->username(‘user’);

$mt->password(‘secret’);

$entry = {
title => “Hello World from Net::MovableType”,
description => “Look ma, no hands!”
};

$postID = $mt->newPost($entry);
$mt->publishPost($postID);

Very simple, it has many other functions, you can read, edit,
categorise etc, all the stuff you can do from the standard Movable Type
API.

(more…)

Read full story • Comments { 5 }

Calculating CIDR notation from IP ranges

by R.I. Pienaar on 13. Aug, 2004 in Code

Following from my previous post about blocking some more countries I discovered a bit of a short coming in the code I used to calculate CIDR notation from ranges of IP’s. So a bit of searching on CPAN got me Net::CIDR.

use Net::CIDR;
$range = shift;
print (join(“\n”, Net::CIDR::range2cidr(“$range”)) . “\n” );

This will take any given range of ip address in format a.b.c.d-w.x.y.z and spew out a list of subnets required to cover the whole range:

# ./range2cidr.pl 64.139.147.0-64.139.170.255
64.139.147.0/24
64.139.148.0/22
64.139.152.0/21
64.139.160.0/21
64.139.168.0/23
64.139.170.0/24

So with this I now have hopefully a more accurate set of rules that will not block bits of New Zeeland as well by accident.

Read full story • Comments { 1 }

iScan

by R.I. Pienaar on 27. Aug, 2003 in Code

UPDATE: Exim now has excellent built-in spam and virus scanning, this performs better and is more effective, you should now use that. I’ve closed the project at SourceForge and you won’t be able to get the files anymore.
This is a general mail manipulation system that features a powerfull plugin architecture using Perl. It was designed to work with Exim but plans are underway to possibly make it more MTA independant.