www.devco.net by r.i.pienaar

27May/056

On The Fly Encryption (OTFE)

I recently got a LaCie 250Gb external drive to do some off-site backups of my data. I am a bit worried about security though since it is so easy to get these USB devices talking to just about anything.
I read up about disk encryption software commonly called On The Fly Encryption - OTFE for short. I use XP and OS X as my desktop Operating Systems but I think I'll stick this drive mostly into my XP machines for now so I am focussing on software for that at the moment.
The amount of data I need to encrypt is probably much less than 5Gb, it is just things like mail, configuration files, a few database dumps and so forth, the rest could go in the plain onto the disk. However some of these tools allow encryption of full devices so that would be an ideal. I would for example not be too happy if my raw files of my photos gets stolen, this is the main chunk of data I need to arrange off-site backup for.
There are a number of free and commercial options, I tried a few in each catagory:

Product Name Cost Comments
FileDisk Free (GPL) Command line only, though the FreeOTFE author wrote a GUI front end for it. It seems to be unmaintained though and certainly was the reason for quite a few hard resets of my box today.
FreeOTFE Free (GPL) Early days in developement but looks promising. I had it stop responding a couple of times when copying large files onto it. Lacks good progress indicators for things, so you think its crashed when its just taking its time. A big plus of this product though is that it has the ability to make Linux compatible crypted disks, this could be a big selling point.
TrueCrypt Open Source (Own License) Works flawlessly so far. I particularly like the nice progress bars on creating and formating of the data files.
CryptoExpert Lite Free but restricted Has maximum file size limitation so did not try it.
Softwinter Sentry $49.95 This product also worked flawlessly, not as nice progress bars but it works.

From the above table it should be clear that amongst the products I tried TrueCrypt and Sentry are the winners, I'd consider buying Sentry if I needed very long term storage and need the kind of backing that a company tends to give, backwards compatibility and so forth.
My usage however as a off-site backup system means I will be overwriting the last backups - or perhaps rotate them for 2 or 3 months - so I most certainly do not need long term archival.
TrueCrypt can also encrypt a full partition so I also tested that and I must say it works great. The initial format over the USB2 of 200Gig would take about 5 hours - so I did a quick format for testing but this is not suggested for actual use. This works great so I will put all my data on the crypted partition and leave a 32Gig FAT32 on the drive to store the TrueCrypt software on etc. You do not need to install anything on the windows machine to run TrueCrypt so can even be run off a memory stick.
My choice therefore is TrueCrypt, kudo's to them for a very professional looking product with a good UI and great documentation to go with it.
While researching this I came across this site that has a whole lot of useful encryption related information.

Tagged as: , 6 Comments
25May/058

More on ZIP File Creation

I did some more testing with the code I posted yesterday and found it isn't 100% compatible with some unzip programs. Works with unix unzip, Mac OS X default tool, WinZip, WinRAR but annoyingly not with the default XP zip folder thing.

The problem seem to be 4 rogue bytes that gets inserted somewhere. So back to Google, eventually I found a much better library at PHP Concept Library Zip. It works more or less the same, except you must pass it files to add instead of just data in variables, either works for me.

Some sample code:

require ("incl/pclzip.lib.php");
$zipfile = new PclZip('zipfile.zip');
$v_list = $zipfile->create('incl/pclzip.lib.php');
if ($v_list == 0) {
die ("Error: " . $zipfile->errorInfo(true));
}
header("Content-type: application/octet-stream");
header("Content-disposition: attachment; filename=zipfile.zip");
readfile("zipfile.zip");

This one works with every unzip tool I've tried, Windows, OS X, command line unzip etc. It also has decent error reporting etc.

Tagged as: , 8 Comments
24May/0533

Creating ZIP files with PHP

UPDATE: You are better off using the library mentioned in this post.

As part of my previously mentioned OpenVPN CA I want to deliver keys, certs and config files to users in a single zip file that they can just extract onto their computers. PHP's own ZIP File Functions only supports reading zip files and not making them.

Some Googling discovered an article by John Coggeshall that can create zip files. It does this by creating the binary data on the fly and can output the zip files directly to the browser from memory or by writing it to disk.

I had some troubles getting hold of a usable version of this code since all these PHP code collection sites have this annoying habit of only showing the syntax highlighted versions of the code rather than give a download link. Eventually got one though and I figured I'll host a mirror of it here to help people out.

Using it is very simple, this is a quick sample that will create a ZIP file and add one directory and one file into then send it directly to the client.

<?
require ("incl/zipfile.inc.php");
$zipfile = new zipfile();
$filedata = implode("", file("incl/zipfile.inc.php"));
$zipfile->add_dir("incl/");
$zipfile->add_file($filedata, "incl/zipfile.inc.php");
header("Content-type: application/octet-stream");
header("Content-disposition: attachment; filename=zipfile.zip");
echo $zipfile->file();
?>
Tagged as: , 33 Comments
22May/050

Photoblog Shots

Quite a nice week in photoblogs, seems it's another week of sunsets or sun warmed up photos.
At The Center of Things
17 May by Ryan Rahn
Wave Crashing
Space
The Icarus Syndrome
20 May by Ryan Rahn
I also subscribed to a few new photoblogs this week, one of the new comers features twice above - Ryan Rahn.
Other new subscriptions: A Walk Through Durham Township, The Darkside Photoblog, Pinhole Media and Shots Photography.

20May/058

Apache + PHP + OpenSSL

I've been trying to write a web based certificate authority to help make signing up for my OpenVPN installation easier for the users. Till now we used OpenVPN GUI for WIndows that provides a frontend to SSL for creating the CSR's. It's all a pain and way beyond what our users can manage in general so a nice web front-end is called for.
I read that PHP has bindings to the OpenSSL libraries so I thought I'd try and use that. I had endless hassles with Apache though, it would just die the moment I call the openssl_* functions in PHP or things would just return FALSE without any useful errors. I tried this on 4 FreeBSD machines all with Apache 1.x on, eventually I found it worked fine on Apache 2 machines! Upgraded one of the systems and it's all good now.
Writing the SSL stuff is very simple with PHP, I'll soon have a full implementation of a Certificate Authority done that is fully web driven so if there are any interest in this I may clean it up and release it under some Open Source licence, will also put up some code samples later but for now just wanted to point out that to get this stuff going you need Apache 2 it seems, well at least on FreeBSD machines.
The full entry has some screenshots of where I am today with the CA so you can get an idea of what I am getting at.

Tagged as: , Continue reading
15May/050

This weeks photoblog shots

Here are the shots I liked this week on the photoblogs I read, only a handful really, for some or other reason I had some trouble finding ones that really impressed me.
Revenge of The Cranes
moma moments II
Hoop
Coney Island VII
Wet Patio
Tower Bridge at sunset
Quite a few sunset pics this week, but I have to say Revenge of The Cranes is by far my favorite.
So while on the subject of sunsets, here is one I took in Venice, this is obviously photoshopped, I applied a Diffuse Glow to it which is one of the few editing methods that significantly change an imagine that I use, I especially like it on sunsets.


Venice Sunset

14May/054

One For All Mosaic on the XBox

I bought a One For All Mosaic aka URC-9990 universal remote control to try and make sense of the mess of remote controls on my table.
Primary concern was compatibility with the xbox specifically to use with XBox Media Center (XBMC). The remote itself is nice enough, UI leaves a bit to be desired during the setup stage but I guess you can't have it all in a £40 remote.
Getting stuff going in the XBox though was another story, the Mosaic has a internet download function that plays a WAV file through your speaker into its microphone, bit like a 300 baud modem without any handshaking or carrier. I downloaded their XBOX profile this got the basics going but since the actual XBox remote control lacks volume, mute etc these were not added.
The Mosaic has a learn function like most of these remote controls I guess but I am sure there is a limited capacity for learned codes so I wanted to not use these if at all possible. It also has a Key Magic system that lets you enter codes manually, the hassle though is that they have obfuscated the codes and what actually gets sent. They say the Key Magic thing is patented etc and you need to speak to their customer support to get the right Magic key to your remote codes. Well thats just b/s all it is is a lookup against one set of numbers that will output the real number out via IR. IR only has 255 valid signals so its not that difficult.
I set the XBMC into debug mode where it shows the IR signals it receives and so made a table of Key Magic codes to actual codes, I also put in what default codes are in use on the XBox remote controller. View the full entry for the table of codes. It was actually very easy there are easily detectible patterns in their mapping from Magic codes to actual codes.
XBMC has in its latest version a global volume control that can be activated in all screens, it also has short cuts for accessing Videos, Movies, Weather and so forth, all accessible by the remote if you know the codes and can program it correctly. The keymap.xml defines what happens when keys or IR signals gets sent. The definition for the global volume up is:

<action>
<description>volumeup</description>
<id>88</id>
<gamepad>rightthumbstickup</gamepad>
<remote>volumeplus</remote>
<keyboard>+</keyboard>
</action>

Simple stuff, so to activate the volume button on the Mosaic I simply assigned a Magic Key using the table below and put the actual value that the Magic Key sends in the keymap.xml like this:

<action>
<description>volumeup</description>
<id>88</id>
<gamepad>rightthumbstickup</gamepad>
<remote></remote>
<remotecode>129</remotecode>
<keyboard>+</keyboard>
</action>

Notice that I cleared out the <remote></remote> blocks and added the <remotecode></remotecode> ones. Actual code 129 maps to Magic Key 262. Using the same simple procedure I also added quick access keys for movies and music and activate the mute button.
I am not sure why One For All feel they need to obfuscate something so fundamental to the operation of the remote control, but I hope this helps you use the device you paid good money for without having to waste additional money and time by having to call or mail them.

14May/050

Applescript

The windows world never really got into scripting GUI applications. You get your Visual Basic for Applications in the MS Office tools (and some others), other big apps have their own scripting languages or simple macro languages but it really is a mess since these things pretty much all do their own thing.

Apple has for ages had Applescript. Applescript is a single scripting language that can script any GUI application - as long as the GUI app makes some commands available but most do.

The nice thing about Applescripts is that the runtime requirement is satisfied on all Apple computers even from the old Classic ones. Things have changed a bit but its remained the same basic thing for a long time. The language is very english like and pretty simple to learn, I've read a book on the plane back from Helsinki and pretty much jumped right in and wrote something useful.

I use NetNewsWire 2 which is really all you'd want in a RSS aggregator. Specifically I sync all my subscriptions via Bloglines and just subscribe to them in NetNewsWire. The problem is when you do that you do not end up with the same groups or custom names as you have in Bloglines so it is a pain to restructure it all when you need to resubscribe to all your feeds.

So once I did the initial big job of putting all feeds into categories and renaming some - specifically I rename personal blogs to the name of the person rather than whatever people call their blogs - I wanted to export my subscriptions to OPML as a backup. NetNewsWire has a function for this but unfortunately it does not export any Bloglines feeds! Totally useless to me then. Applescript to the rescue.

I will not go through the whole app here, you can get the complete source a bit later, but here are just some snippets and a few words on each to show the basics of scripting with Applescript.

First we tell our script to speak to NetNewsWire and fetch all the group folders.

tell application "NetNewsWire"
repeat with curGroup in (every subscription whose is group is true)
end repeat
end tell

The whose is group is true reads a bit weird, 'is group' is a boolean property of each subscription, so the above will loop over all the groups thats been defined.

Next we want to pull out each subscription for the group we are in, so we grow the code to look like this:

tell application "NetNewsWire"
repeat with curGroup in (every subscription whose is group is true)
set t to display name of curGroup
-- put code here to build outline entries for each group
repeat with curSub in (every subscription whose display name of group is t)
-- put code here to build outline entries for each sub in this group
end repeat
end repeat
end tell

Here I get the display name property of the current group and search for all subscriptions with the same group.

After building the opml file and storing it in a variable I simply chose to output a new TextEdit.app document with the contents of the opml file, but could easily have written a new text file for example

tell application "TextEdit"
activate
make new document
set the text of the front document to opml as Unicode text
end tell

Simple stuff, start up the TextEdit application, make it active, create a new document and put the opml file contents into the document as Unicode Text.

So that's the basic logic, you can get the full script here. This will probably not work 100% with nested groups and I do not cater for subscriptions that does not belong to any group, it works for me though :)

This is a simple example but it does demonstrate though the absolute beauty of this, data from one app written by one software house queried and modified then output into an app written by another all by a scripting language written by a 3rd, fantastic.

Tagged as: , No Comments
10May/050

Spotlight hardcoded exclusions

I've been trying to play with the new Certificate Assistant that comes with Tiger to set up a CA but had some trouble tracking it down in the System Preferences. So I figured I'd search for it using spotlight but could not find it.
Turns out it is on the drive in /System/Library/CoreServices/Certificate Assistant.app so why did it not show up in Spotlight? Annoyingly I found out that there is a hard coded list of directories that Spotlight will exclude, this include all of /private which contains among others /etc and a whole long list of other stuff that Unix people might actually enjoy being able to find.
If you'd like to override these defaults then read this discussion board entry for all the details.
Spotlight is becoming more a pain than it's worth, I have considered just turning it off since it seems daily I discover a new way that it is somehow crippled in the name of sparing the poor clueless masses from seeing something they aren't supposed to see etc.

Tagged as: No Comments
9May/050

FreeBSD 5.4

FreeBSD today released version 5.4. This looks like a pretty good release mostly for some changes to userland tools and some firewalling bits, some of the highlights for me are:

  • CARP from OpenBSD has been added, this allows for some nice shared IP addresses between machines with failover carp(4)
  • The ipfw(8) ipfw fwd rule now supports the full packet destination manipulation when the kernel option options IPFIREWALL_FORWARD_EXTENDED is specified in addition to options IPFIREWALL_FORWARD. This kernel option disables all restrictions to ensure proper behavior for locally generated packets and allows redirection of packets destined to locally configured IP addresses.
  • The libarchive library (as well as the tar(1) command that uses it) now has support for reading ISO images (with optional RockRidge extensions) and ZIP archives (with deflate and none compression).
  • The -f option of tail(1) utility now supports more than one file at a time.
  • rc.conf(5) now supports changes of network interface names at boot time.

There are a whole bunch of other interesting changes, be sure to read the Migration Guide before attempting to upgrade.