R.I.Pienaar

Archive | Usefull Things RSS feed for this section

Useful Xen Utilities

by R.I. Pienaar on 13. Feb, 2008 in Front Page, Usefull Things

Today on freshmeat I noticed 2 useful utilities for anyone running Xen Servers.

The first is called Virt-top it is a easier to read top like tool than xm top that shows all virtual machines memory and CPU usage in a nice display including totals etc:

The other – Virt-P2V – it’s a CD Image that you can boot a physical machine with that will then convert it to a virtual machine for you. It will scp the drive image to a destination of your choice and create a config file to boot it after asking you some questions. I intend to use this to move a VMWare virtual machine to Xen soon, will post here and see how it goes.

Both of these come from a Red Hat employee, with some luck we’ll see these included in Red Hat Linux soon.

Read full story • Comments { 0 }

Extracting only certain lines from a file

by R.I. Pienaar on 09. Jan, 2008 in Usefull Things

This is probably old news to most people but I need to remember this so I figured I may as well blog it.

I made a mysqldump that just takes all databases into a single file, already I want to kick myself because I know if I ever need to import it there will be troubles because the target database will already have the mysql database etc.

Really I should have used MySQL Parallel Dump that makes files per tables etc and is much faster but it didn’t exist at the time.

So how to pull lines 8596 to 9613 from this big file? It’s trivial with sed:

here is a sample file:

$ cat > file.txt
line 1
line 2
line 3
line 4
line 5
^D
$ sed -n '2,4p;4q' file.txt
line 2
line 3
line 4

The sed command just tells it the start to end line and also to quit processing when it hits the end line, really kewl.

Read full story • Comments { 0 }

NetNewsWire is set free

by R.I. Pienaar on 09. Jan, 2008 in Usefull Things

I just noticed that the folks over at Newsgator has set pretty much their whole product suite free today, free is Newsgator for Windows, NetNewsWire for the Mac, the online version of NewsGator and so is NewGator Go! for your phone.

This is pretty huge news as all the products I just mentioned syncs with each other seamlessly and have great UI’s, NewNewsWire has been my reader of choice for ages.

For the paranoids out there though there’s this little tid bit in the new features list:

Sort by attention: NetNewsWire now tracks more information about what you do and can tell which feeds are most important to you.

So you probably want to find out exactly what that’s all about first.

Read full story • Comments { 0 }

RedHat 5.1 tunable kernel ticks per second

by R.I. Pienaar on 07. Jan, 2008 in Usefull Things

For some time the default clock rate on RedHat machines (and probably others) have been 1000HZ, this is great to keep your mouse moving smooth while something big is happening in the background, but not so great for hosting 10 virtual machines on one poor physical machine as it will have to try and satisfy 10000 ticks per second.

I’ve been using a guest kernel repository by one of the VMWare users that rebuilds the std CentOS/RedHat kernels with HZ=100 and it’s been great, chopped massive amounts off my CPU usage on the host.

Now with RedHat 5.1 this is not needed anymore see this post for a bit of a graph on the impact and the background. The short of it is, simply append divider=10 to your guest kernel boot parameters and enjoy a much happier host. I found that time keeping also becomes more predictable in the guest.

Read full story • Comments { 0 }

Processing Medical X-Ray Data

by R.I. Pienaar on 19. Dec, 2007 in Front Page, Usefull Things

Recently one of our snakes died due to an abscess in her body, we tried everything to safe the poor snake but in the end – and almost £1200 later – it passed away one night.

While undergoing treatment we were given copies of the X-Ray data, at the time I tried to read the RAW data files but failed, finally writing it off as some proprietary format specific to the X-Ray machine vendor.

Today I came across an item on MacNN mentioning OsiriX which is basically an Open Source suite to drive all things medical. Turns out the data is encoded in a standard format known as DICOM which defines the data type and also a network protocol for these machines to communicate with each other and their image storage over a network.

The data I got was in RAW format so none of the DICOM headers were present, this led me to some other software and a FAQ for importing unsupported/unknown DICOM data. Using the information there I was able to work out based on file size that my data was 512 x 512 big with an Header offset of 6480.

Armed with this information I was able to do a pretty decent import into OsiriX, the output of the one X-Ray can be seen below (click for full size).

The infection is clearly visible on the left of the image, this was removed but a second formed. Anyway, so the point of this post isn’t to go on about the poor departed snake but to mention the fantastic medical imaging tool OsiriX which is a pure Mac application and while I doubt many of my Blog audience will care for it it might still be of some use to some Googlers.

Read full story • Comments { 0 }

QNAP TS-209 pro NAS

by R.I. Pienaar on 14. Dec, 2007 in Front Page, Usefull Things

I have been looking for a good solid SOHO Network Attached Storage device for a while. I was all set on the Lacie 2big 1.5TB Network device, it is attractive does what I needed – not much more than share files – and supports multiple drives.

Problem is I have since discovered that Lacie UK are the most incompetent people on the planet. I placed the order with them after their site showed they had the unit in-stock on a 3 days delivery time, after placing my order site said the same so I was confident it was all in order. Needless to say the device never came. I emailed their sales lines, no response, I emailed their supports lines, no response. I called them (after spending about a hour tracking down phone numbers) they didn’t reply to voice mails.

After about 10 calls I eventually spoke to someone who was unhelpful to say the least, I was told next-week, next-week etc a few times, next week came and went and no drive unit so I eventually just canceled my order. No more Lacie devices in my future ever that is a certainty.

Some searching later I found a few excellent reviews over at SmallNetBuilder for this and other devices, they even have a very awesome tool for comparing different NAS devices for speed etc, I decided based on their review to get the QNAP TS-209 pro.

The TS-209 pro is an attractive yet very well built little system, all the screws and connectors are proper solid bits of kit like you’d expect on real hardware. It is a Linux box and you can ssh to it:

# uname -a
Linux vault 2.6.12.6-arm1 #2 Thu Nov 1 03:31:14 CST 2007 armv5tejl unknown
# cat /proc/cpuinfo
Processor       : ARM926EJ-Sid(wb) rev 0 (v5l)
BogoMIPS        : 332.59
Features        : swp half thumb fastmult
# cat /proc/mdstat
<snip>md0 : active raid1 sdb3[2] sda3[0]
731423296 blocks [2/1] [U_]
[==>..................]  recovery = 10.6% (78181504/731423296) finish=144.0min speed=75574K/sec

So a proper little box then, I put 2 x Seagate 750GB drives into it for the same amount of storage as I would have had in the Lacie, the total price ended being about GBP50 more or so.

That GBP50 is money really well spent in this case. The device has hot swap drives – I tested it by yanking one out live without any problems, a few beeps, a few emailed alerts and log entries:

The device has a ton of features, the usual SMB shares are there but also NFS, Appletalk, FTP, Web access. It has a MySQL server built in, a webserver with php so you can deploy whatever you want on it. An iTunes server for your MP3s and a typical UPNP media server that will work with your PS3 etc.

This is a really capable device built on solid technology, so far I am very happy with it and will recommend to anyone. If anything significant change on my experiences I’ll post more later but I suggest you read the review linked above and seriously consider this for your SOHO NAS needs.

Read full story • Comments { 2 }

Google Calendar Sync for Blackberry

by R.I. Pienaar on 13. Dec, 2007 in Front Page, Usefull Things

Google released a neat new Blackberry app recently, it lets you sync your Google Calendars with your Blackberry native calendar.

I tried it and it’s awesome, previously I synced my Mac to Google one way using ical feeds and then used the Blackberry sync tools to push to my phone, again one way. It sucked a lot. Now, the BB goes direct to source, syncs up all my calendars even ones I subscribed to and it just works.

I have one tiny gripe, I’d like to be able to pick which of my calendars new entries go into, even if they all go into the same one as long as I can choose which one, other than that it’s great.

Read full story • Comments { 0 }

Zoho Creator

by R.I. Pienaar on 07. Dec, 2007 in Usefull Things

I’ve heard a lot about Zoho before, essentially they provide Web 2.0 style office applications – many of them completely free of charge.

I didn’t pay them too much attention really, being a mostly happy Google hostage. Today I read again about their CRM which led me to look at their other offerings. Boy was I impressed.

Initially I am just trying out Zoho Creator, it is your basic RAD database application development system, fully hosted, fully free and fully on the web and mobile.

I ofcourse had an itch to scratch, the itch being that I keep a list of my billable hours, sales, recurring bills etc all in a big old nasty VI file using VIM Outliner. This approach works but its hardly high tech or cross platform.

So I created a small Creator app that lists my clients, hours billed, sales items, recurring invoice items and some reports for listing invoiced and un-invoiced of each. Some screenshots of my app:

Adding a client

Adding an invoice line

Report of unbilled time

Bulk mark items as invoiced
You can click on the images for larger versions.
This is pretty impressive, you can really imagine coding a standalone version of this for some would be fun, for me it wouldn’t be a challenge and so it would be dead boring and probably take many hours that is better spent on billable work. I’d also be stuck with something else to maintain in code rather than a snazy point-and-click UI.
So how long did this app take me? Not more than 2 hours end to end, free hosting, and it’s available online via my blackberry for adding items etc while on the road.
Creator also has an API like all good Web 2.0 stuff and so it is really easy for you to integrate something into this, but you could also just embed a specific report or specific input field into any page of your choice via a simple iframe.
This is a great solution for small little web apps like this, soon in Q1 2008 they will also integrate the database from Creator into the Reports app then you can do some really nice multi dimensional reports, graphs and such.
I’d recommend checking out Zoho apps. As always be sure to read the T’s and C’s though before handing over too much overly sensitive data.

Read full story • Comments { 1 }

Linux HA Clusters with Heartbeat

by R.I. Pienaar on 25. Nov, 2007 in Usefull Things

I played with heartbeat a year or so ago and while it was ok, I was not convinced it was 100% there yet, it lacked monitoring capabilities of resources it managed which was a major drawback in my mind.
I again had a need for a cluster recently so had another look. Since the previous time they’ve released Heartbeat version 2 which has a full resource manager like you’ll find in more mature systems.
The only time I’ve previously used a cluster system extensively was on Windows 2003 Enterprise with MS SQL Server Enterprise, the setup then was a active-passive SQL cluster with shared fibre storage. The windows cluster services works quite well, have a solid GUI and supports many nodes in a cluster.

I wouldn’t yet compare Heartbeat to commercial offerings, technically the Cluster Resource Manager introduced in version 2 is a massive step forward but configuration can be a real nightmare.
The documentation is very thin on the ground and configuration has to be done through XML files. There is a disturbing trend these days for people to think XML is an acceptable form of configuration from a human point of view but it really is not. Worse is the DTD for the XML format is the definitive source for configuration reference, as their WIKI states:

It was out of date and didn’t take into account the fact that not everyone is on the same version. Instead, you should refer to crm.dtd on your system (which is always appropriate to your version).

Heartbeat does provide a GUI but I found it immature, inconsistent and often had error messages pop up with no contents in them other than an ‘OK’ button. It also lacked some features, while evaluating it I decided if I had to rely on the GUI in any way as it stands today I would not use Heartbeat for my cluster as it would invalidate any high availability hopes I had. It is useful though to monitor and visualise your cluster, especially if you have a lot of groups.

Once I figured out the correct XML formats to do what I wanted and learned the command line tools and provided my own documentation for these I eventually got a full 2 node cluster going managing currently 5 resources with more to follow.
My main goal with this project was to manage HAProxy on the cluster not because HAProxy is in any way unstable but because I find it difficult to do maintenance with just one machine for it and as I adopt HAProxy more the hardware would be an unacceptable single point of failure.
Heartbeat lets you manage resources using several type of scripts, the best one to use would be the new OCF standard scripts which is designed specifically for managing cluster resources but it’s an emerging format so not a lot of scripts exist for it today. Heartbeat also support using standard /etc/init.d/ style rc scripts with the caveat that they have to be 100% LSB compliant. You’d think at least the scripts that Red Hat provide are LSB compliant but you’d be wrong, I had to fiddle with almost each one I wanted to use which is not optimal because I hate editing non-config files delivered with RPM and I think its very poor of RedHat who has been making a point of telling anyone who would listen that they’re completely LSB compliant.
I would also have liked to build a HA NFS server but unfortunately Heartbeat version 2 and DRBD version 8 does not yet play nicely, so that is a project for some other time.
My conclusions on Heartbeat then is that it is a good solid project especially with version 2, I think in a year or so once documentation etc had a chance to mature it would be a good choice for almost anyone, for now though it is unfortunately out of reach for the average guys.

Read full story • Comments { 0 }

GMail IMAP

by R.I. Pienaar on 01. Nov, 2007 in Usefull Things

Recently Google announced that all gmail users are getting IMAP, this is fantastic news. I won’t actually use it with an IMAP client but I will use it to backup my mail and my meta data! It might even help me to slowly import years and years of old mail into GMail.
So how to do a backup, first you need imapsync or something similar, I’ll use imapsync since that’s what I know.
Second you need a local imap server to dump your gmail account into, I am using my current server that already has other mail in it so I made a sub-folder called GMail to sync into.
Once you have imapsync installed its pretty simple with the following command line:

imapsync --host1 imap.gmail.com --ssl1 --user1 you@gmail.com \
--password1 secret --host2 your.imap.com --user2 you --password2 secret \
--prefix2 INBOX.GMail. --authuser1 you@gmail.com --authmech1 LOGIN \
--authmech2 LOGIN --syncinternaldates

You’ll need to look at that command and set appropriate options for your user, password and imap account. Also if you’re going to run this on a shared machine where other user accounts exist, you should use the –passfile1 and –passfile2 options to not show your password in the output of ps.
The process is slow, and creates some duplicates for instance you’ll have a [Gmail]/All Mail folder that contains all your mail but you’ll also have folders for each tag that would have copies of messages that are already in your All Mail. This to me is a small price to pay for the peace of mind that solid off-site backups provide as the thought of my mail on a 3rd party system has been keeping me awake