www.devco.net by r.i.pienaar

24Aug/090

hetzner.de hardware policies

So I use Hetzner a lot for my machines, I've about 10 to 15 of their machines now across various clients and am mostly quite happy with them.  They provide a service that matches the price - ie. good enough.

One area of their service though really grates me, they give you old machines, when those machines fail they replace them with other old machines similarly for drives etc.

On more than one occasion now have I had hard drives fail only to see them replaced with other shitty drives.  Each time they claim the drives are well tested and each time they pull the old 'it could be the cable' trick and then replace the machine and the drive.

Since this has happened to me every single time I've changed a disk so far I have to wonder if this is everyones experience? 

From where I sit its simple.  They made a choice to take out drives reported broken by someone, they then test it and put it back when their tests fail to find any problem, they do this to save them money knowing full well that drives will fail and all they're doing is shifting the risk onto their clients, while the clients keep subsidizing their expansion. 

So given this is the quality of service they're aiming at, surely once this policy bites a good long standing user offering some kind of payback for the inconvenience would be good business practice?  Apparently not.

This is pretty poor, even after complaining to them they swapped my chassis and again put a disk with > 6000 hours under its belt in my machine.

So I guess you need to be pretty sure your softraids are setup properly when you want to use this company, their support stand is clear:

I'm sorry but we don't promise anywhere that we built always new hardware into our servers. I can only ensure you that all hardware is always well tested and without any problem before we build it into a server.

Ie., screw you, we don't care for any evidence and repeated failures, and we take zero responsibility for our equipment, we'll just keep taking your money.

17Aug/090

Managing web traffic with ruby-pdns

A short while ago I wrote about releasing a Ruby Development framework for PowerDNS the release is still early days, feature complete but needs some robustness tweaks and a new release will be out in a week or so to address that.

I wanted though to highlight some success that I've had using it.  I have a small static farm for a client that handles around 2MiB/sec of 200x200 jpg files, this setup is for a startup so out of necessity its all built to be cheap, I host on networks I don't own yet I need pretty good control over it, what IPs will be used to serve traffic and so forth.


The graph above shows the case before caused by the windows DNS bug, you'll see the bottom host is working pretty hard getting a large chunk of the bandwidth.

This is a problem because come mid month this poor machine has already used up its allocation of 2.5TiB of transfer and I need to move it from the pool.

So my goal was to shift the traffic to the yellow and green machines and just generally balance things out a bit.   I used the Weighted Round Robin feature of ruby-pdns to adjust the biases, it took a bit of fiddling because for some other reason even when this machine gets fewer requests per second it still seems to manage more in terms of bandwidth, this is the eventual code snippet:

        ips = ["213.x.x.232",     # dark blue
               "88.x.x.201",          # lighter blue
               "82.x.x.180",          # yellow
               "82.x.x.181"].randomize([1,2,2,3])   # green

        answer.ttl 300
        answer.shuffle false
        answer.content [:A, ips[0]]
        answer.content [:A, ips[1]]
        answer.content [:A, ips[2]]

The thresholds seems odd but that's what worked after some fiddling, see the graph below.


This is much nicer balanced, it's not perfect and I doubt I will get it perfect with just 4 machines to play with but I believe it's already at the point where it means I can use all my machines for the entire month without hitting any limits.

Here's another graph over the week showing things side by side:


The improvement is very obvious in this graph and you can see I've not lost anything in performance between first day and last day on the graph in terms of throughput (the lower days were days where lower traffic is expected).

If I look at my actual transfer used it's better balanced now, first lets see the 12th:

      08/12/09     12.67 GiB |   46.42 GiB |   59.09 GiB |    5.74 Mbit/s
      08/12/09      7.71 GiB |   21.32 GiB |   29.04 GiB |    2.82 Mbit/s
      08/12/09      9.05 GiB |   23.05 GiB |   32.10 GiB |    3.12 Mbit/s
      08/12/09      6.94 GiB |   16.56 GiB |   23.50 GiB |    2.28 Mbit/s

Again the skew is very clear with a 23GiB on the lowest compared to 59GiB on the highest use machine, on the 17th it looked a lot better:

      08/17/09      7.84 GiB |   28.55 GiB |   36.39 GiB |    3.53 Mbit/s
      08/17/09      8.46 GiB |   25.66 GiB |   34.12 GiB |    3.31 Mbit/s
      08/17/09     11.21 GiB |   30.70 GiB |   41.91 GiB |    4.07 Mbit/s
      08/17/09     10.25 GiB |   28.20 GiB |   38.46 GiB |    3.73 Mbit/s

Obviously much better when looking at the 2nd to last column.  The first column is received the increase in those is down to a slightly lower hit ratio on the caching proxy on these machines meaning it's fetching more files from origin than the others.

Overall I am extremely pleased with this solution, I agree one should not be using DNS as a hammer to all your nails but for startups and cloud based people who do not have control over networks, BGP tables and so forth this really does represent a viable option to what would otherwise be an extremely expensive problem to solve.

19Jan/091

Cogent

I've not posted here for a while, been insanely busy but today those lovely people at Cogent kicked me out of my blogging slumber with a shocking display that I simply had to share.

Cogent for those who do not know is a very large Tier 1 ISP, known mostly for many disputes with other ISPs about peering, it has become so bad that in the UK at least they are basically a complete no-go zone for anyone. 

I've previously delt with Cogent when a client signed up for a few mbit of Cogent bandwidth on the basis of a £5/mbit pricing structure, they soon realised that you get what you pay for.  Even between racks in the same Data Center you could not reach each other without first hopping over to Europe.  I've attempted to resolve this at the time with Cogent and the other ISPs and  both confirmed that it is essentially a waste of time.  Cogent said they can't speak to the ISP in question since most of the UK ISP industry can't stand them.  The other ISP basically laughed out loud about being 'suckered' into buying Cogent bandwidth.

This is confirmed elsewhere, search the Renesys Blog for Cogent and you'll find a lot of information about Cogent, mostly bad news.  From an article on their blog about Cogent in the UK you will see this:

Firstly, Cogent has a fairly serious Europe problem right now. They
have been aggressively attacking the European market for a few years
now and making some solid headway. They bought a couple of carriers
(Lambdanet Spain and France, Carrier1 in Germany among them), ruthlessly
integrated them and then proceeded to undersell the market by a factor
of 50-80%. This has made them many enemies.

As a result of this approach to business, Cogent has much less
effective peering in Europe than do many of its larger competitors.
Most of the European PTTs refuse to peer with Cogent anywhere on the
European continent. Recently, some large US carriers (among them
Level (3) ) seem to have adopted a similar approach. This means that
when Cogent sells capacity in Europe, it is forced to drag that
traffic back to the US to hand it off to its peers here. Of course
that means that if the ultimate destination is European, the traffic
has to travel back. This is a burden on both Cogent and the European
carrier and, of course, the customers on both sides. But it's
unlikely to change because of just how much hate there is for Cogent
among European networkers.

This basically confirms my experience with Cogent and those of many I have spoken too.  As such if you choose to support Cogent you will basically be forced to:

  • Buy a lot of other bandwidth since if you're hoping to serve UK customers, Cogent is a terrible sole carrier to have
  • You will need to invest in extra hardware, extra admin time, extra complex routing infrastructure and additional overhead on your teams
  • You will forever be at the mercy of everyone who hates Cogent, you will find your self randomly falling off the internet, randomly de-peering with vast swaths of the internet and basically the whole thing will be a pain in the behind.

For these reasons, everyone who I know with Cogent bandwidth use them as last resort backup carriers, they are cheap and basically shit, but ok enough to use as a backup when everything else failed.

Over the last few years Cogent has contacted me direct via email to attempt to sell their wares, always the threads end withe me saying something along these lines:

Furthermore we'd prefer to use companies who do not directly
contact us with marketing material, please remove us from your lists
for future contact.

Today again one of my clients got a mention on Techcrunch which resulted in more spam from Cogent, again to an email address totally unrelated to my business activities, not listed in whois records for the client or anything like that.  The sales person even had the nerve to copy the email that the above quote is from in his mail to me asking if I can have a conference with him.

My response was the usual, no we don't deal with spammers, you were told to leave us alone now please stop bothering us.  Which resulted in an amazingly pushy email from the sales person, quoted below:

No doubt that writing when being asked not to is, well, borderline. That
said, it is both of our responsibilities to make sure that all options
are explored. You need to confirm that you are aware of all vendors
information, and mine includes getting it out there. 

.

.

Admittedly, this is difficult to resolve via email. However, if I didn't
think that we could compliment your service, I wouldn't persist.

This is just amazing, this person really think he can presume to tell me what my responsibilities are, what I need to do, and that I have to indulge his blatant b/s.

After I again pointed out that they were asked to stop mailing me and I pointed out that they were using a private email address held by a UK citizen and as such under the data protection laws they need to stop contacting me when asked, they once again mailed me demanding further information about my customers.  They really are on par with simple Viagra spammers.

Does anyone really think this kind of heavy handed tactic gets them business?

The worst part of it is the ISP who currently provide a large part of our b/w are Cogent customers, Cogent Sales people do not think twice to approach clients of their clients and try to undercut them - effectively trying to steal their customers business away from them. 

Why would any business support such a company?  I would not, I would effectively be negligent in my duties to my clients to ever recommend these clowns for anything since they are just a nightmare waiting to happen.

Tagged as: , , 1 Comment
1Oct/082

Layeredtech’s thanks to old customers

I have been a customer of Layeredtech for years, at present I have only 2 machines there but at times I've had 7 or 8.  My one machine is pretty old, I think I got it circa 2002 or so and it's been doing well, same hardware etc.

Yesterday I received the following email from them:

Layered Tech is committed to being the leader of the Hosted
Infrastructure market by providing our customers with the best products
backed by the best service.  In an effort to improve our customer
experience, we have determined that a small number of existing servers
will need to be relocated from their current data center.  As you are
receiving this message, we have identified that you have one or more
servers in the in area of the Savvis facility that will need to be
moved.  It is our intention to minimize any interruption in service and
we will do our best to work within predetermined time frames that are
convenient to you.

Due to the form factor (chassis type) of
this server, we will need to migrate your data to a new server. We will
work with you so that the impact is as minimal as possible.  

Below
are the servers that are affected by this migration.  Please respond to
this message acknowledging the need to relocate your server(s).  At
that point, we will move this ticket to our Operations Department where
we will work with you on a migration schedule.

From reading this you might assume they will assist you with the migrate and this is a notice of an impending change, perhaps a month or two from now?

In reality the situation is that no, they will not help you migrate your data.  They want you to take out a contract for a new machine and then migrate your data yourself - something which even at best will take 5 to 10 hours on oldish machines like this.

They do not offer any compensation, and when pressed on that point only offer 1 month...the cherry on the cake is that all this has to be done for 18 days from now, in effect they are terminating your old machine forcing you to take a new one and doing it with less than the agreed 30 days notice.  Like it or not.

The sales person who has been coordinating this from their side is incredibly unhelpful and frankly useless, only after much pushing back by me do I even get a hint that anything other than do-it-yourself migration is an option, at this point still waiting for details.

This kind of disregard for customers is typical of large hosting centres, they have thousands of customers and their hard handed handling of their customers is acceptable because at worse they'll loose a fraction of a percentage of customers, so being unhelpful really does pay off for them since most people will probably just take this crap.

This is shockingly poor service, if you value your data, avoid Layeredtech.

13Apr/080

Online Regex Testing

Back in 2004 I posted about The Regex Coach, its a great app that I still use today, however it only really works on Windows so I have been looking for some alternatives.

There is a really great resources called Regular-Expressions.info it even has a cheap tool that you can use to do something similar to The Regex Coach.  Today Lifehacker mentioned RegExr, its a great web app but also has standalone versions for Windows, Linux and OS X, full of sample regular expressions, good explanations of what a regular expression parses as etc, it is a perfect replacement for The Regex Coach, worth checking it out!

17Jan/080

Library of Congress and Flickr

Flickr and The Library of Congress announced a project together to put a whole load of the Libraries photos up and to ask the public to create meta data for these photos - tags, notes etc.

This is a phenomenal achievement for Flickr in my mind, looking through these photos there are some really absolutely amazing shots showing American life in the early 1900s, depression years, the war etc.

I spent some time over lunch browsing some of these, the machinery, clothes, culture, cars, architecture, it is all just amazing I wish there were such a good record of the UK available to the public.

Some of the images are just great to look at like the one below from 1911, that's a hand held large format camera, amazing.


Others show the truly amazing work that photographers did in those days and frankly makes me wish I can even come close to this kind of shot on my digital cameras.

  

Click on these images and look at them, they are phenomenally well done the richness and dynamic range of color in those shots far out paces the results I tend to see on digital.  I wish self developing color slides wasn't such a pain else I'd start doing medium format color transparencies right away.

11Jan/081

Performance of Social Media Sites

I am under contract to be systems administrator at a Social Media company called Handmade Mobile, their prime product is Flirtomatic (You might want to wait till you're home from work to visit the site), a site for adults who just want to chat and meet fun people, it's not a dating site in the normal sense it's more like MSN with with some profile searching, pictures, etc added.

I got involved with them about 2 or 2.5 years ago they were having some issues and needed a re-do of their servers, network, etc.  I designed a simple easy to manage network and put in servers built the way I liked them using Open Source technologies which I still maintain and administer today.

They've been making some good waves in the last while, 2 items I think are worth mentioning:

Back in September they set a in-house record for monthly WAP impressions with more than 100 Million WAP impressions in the month.  One report on this mentioned that In March the amount of searches across all the major search engines combined done from phones were around 20 million, so this puts the 100 Million WAP impressions in a extremely good light.

Today I got news of another impressive bit of stat about them - A company called WatchMouse who specializes in monitoring web site response and stability did a test against 104 Social Media sites for performance, time to load the whole page etc. they also penalized your points for any failed loads etc.

Out of the lot Facebook were the worst, Flirtomatic though came fourth which I have to say I am very impressed by as Flirtomatic has huge amounts of photos on their front page compared to number 1 faceparty for example.  Faceparty's front page weighs in at a very light 44KB (1KB for the HTML) while Flirtomatic is 630KB (17KB for the HTML) so I think being 4th fastest given its bulk is great.

Anyway, kudos to them :)

Tagged as: 1 Comment
19Dec/070

Processing Medical X-Ray Data

Recently one of our snakes died due to an abscess in her body, we tried everything to safe the poor snake but in the end - and almost £1200 later - it passed away one night.

While undergoing treatment we were given copies of the X-Ray data, at the time I tried to read the RAW data files but failed, finally writing it off as some proprietary format specific to the X-Ray machine vendor.

Today I came across an item on MacNN mentioning OsiriX which is basically an Open Source suite to drive all things medical.  Turns out the data is encoded in a standard format known as DICOM which defines the data type and also a network protocol for these machines to communicate with each other and their image storage over a network.

The data I got was in RAW format so none of the DICOM headers were present, this led me to some other software and a FAQ for importing unsupported/unknown DICOM data.  Using the information there I was able to work out based on file size that my data was 512 x 512 big with an Header offset of 6480.

Armed with this information I was able to do a pretty decent import into OsiriX, the output of the one X-Ray can be seen below (click for full size).

The infection is clearly visible on the left of the image, this was removed but a second formed.  Anyway, so the point of this post isn't to go on about the poor departed snake but to mention the fantastic medical imaging tool OsiriX which is a pure Mac application and while I doubt many of my Blog audience will care for it it might still be of some use to some Googlers.

16Dec/070

New Google Calendar Feature

I've previously mentioned a major problem I had with Google Apps for Domains Calendar function:

GCal also has a feature that mails you a short daily agenda, much like
what you see if you hit the Agenda button. Problem is the agenda mail
function ONLY reads from your primary calendar, it does not include
events from any subscribed calendars etc. This means that one of the
biggest selling points of Google for Domains is crippled, if you share
calendars you can't use them even in the rudimentary tools provided.

I am glad to report that this is now fixed, my Daily Agenda mails started arriving 2 or 3 days ago and they include entries from all my Calendars,

Brilliant.

7Dec/071

Zoho Creator

I've heard a lot about Zoho before, essentially they provide Web 2.0 style office applications - many of them completely free of charge.

I didn't pay them too much attention really, being a mostly happy Google hostage.  Today I read again about their CRM which led me to look at their other offerings.  Boy was I impressed.

Initially I am just trying out Zoho Creator, it is your basic RAD database application development system, fully hosted, fully free and fully on the web and mobile.

I ofcourse had an itch to scratch, the itch being that I keep a list of my billable hours, sales, recurring bills etc all in a big old nasty VI file using  VIM Outliner.  This approach works but its hardly high tech or cross platform.

So I created a small Creator app that lists my clients, hours billed, sales items, recurring invoice items and some reports for listing invoiced and un-invoiced of each.  Some screenshots of my app:


Adding a client


Adding an invoice line


Report of unbilled time


Bulk mark items as invoiced

You can click on the images for larger versions.
This is pretty impressive, you can really imagine coding a standalone version of this for some would be fun, for me it wouldn't be a challenge and so it would be dead boring and probably take many hours that is better spent on billable work. I'd also be stuck with something else to maintain in code rather than a snazy point-and-click UI.
So how long did this app take me? Not more than 2 hours end to end, free hosting, and it's available online via my blackberry for adding items etc while on the road.
Creator also has an API like all good Web 2.0 stuff and so it is really easy for you to integrate something into this, but you could also just embed a specific report or specific input field into any page of your choice via a simple iframe.
This is a great solution for small little web apps like this, soon in Q1 2008 they will also integrate the database from Creator into the Reports app then you can do some really nice multi dimensional reports, graphs and such.
I'd recommend checking out Zoho apps. As always be sure to read the T's and C's though before handing over too much overly sensitive data.

Tagged as: , 1 Comment