I’ve a machine at Hetzner DE and have been pretty happy with it and their service in general. I used to run VMWare Server on it for virtualisation but have become a bit annoyed at the totally unfixable clock problems that seems to be the norm with VMWare guests.
First the good news. I decided to give XEN a try on it, same setup same spec virtual machine on the host doing the same workload as before, below a CPU usage graph before and after, enough said.
I know it’s not the same technology etc, but XEN VMs will do what I need now so it seems a good option for me.
Now for the bad, I had a few problems:
- Hetzner uses a DHCP/TFTP rescue system that is very useful, if I make a mistake in a network config for example I can sort it out via this. XEN by default works in a bridged mode that will change the MAC address for your eth0 to FE:FF:FF:FF:FF:FF. The problem is that Hetzner uses Realtek NICs which as most people know are uber crap. When the machine gets a normal alt-ctrl-del or other reboot other than a power cycle the NIC does not reset to the original MAC. So if I get a crash I wont ever get to the rescue system. This cost me 119 Euro this weekend in support fees.
- In the past I had a few crashes on this machine, adding acpi=force irqpoll noapic to the kernel command line solved it, however the XEN kernel won’t poll my drives properly without removing the noapic bit, so I hope my machine doesn’t become unstable all of a sudden.
I am using CentOS 5.0 on this machine that came with XEN out of the box so installation was dead easy, creating new VM’s are easy etc and very importantly I can manage the VM’s all through text consoles which was a pain for me under VMWare as I don’t use Windows or Linux, getting console access was tough.
So about the MAC address, this turned out incredibly easy to fix by turning the machine into a routed network setup instead of a bridge, so previously the network was bridging eth0 with the vif’s from the virtual machines onto the wire, this required the MAC address of eth0 to change. Moving to a routed setup required the following changes:
/etc/sysctl.conf:
net.ipv4.ip_forward = 1
net.ipv4.conf.all.proxy_arp = 1
This basically enables routing and sets up proxy arp across all interfaces, this is to work with Hetzner’s routing where they don’t actually add a route to my subnet over my eth0 but over to the switch is connected too. Proxy ARP sorts this out.
/etc/xen/xend-config.sxp:
#(network-script network-bridge)
#(vif-script vif-bridge)
(vif-script vif-route)
(network-script network-route)
This points XEN to its provided route scripts rather than the bridge ones.
Reboot virtual machines and thats that. Machine is perfectly happy, routing traffic and MAC address is correct.
Bonus is now all traffic gets routed through my Dom0 and I can now do all the firewalling there outside of my DomU’s which enhances security as it is essentially a border firewall.
Hi!
Very interesting article.
How did you actually configure the network interface on both Dom0 and DomU?
Just curious.
Claudius
Nothing fancy, eth0 on dom0 is as normal, just the ip on eth0 and
ditto for domu’s, xen will when starting up each domU create vifx.x on
the dom0 machine without you needing to do anything.
Wow, that was a fast turnaround…
Dom0 does, in fact, create a vifx.0 for each domU. However, on my system that is bound to the machine’s main IP rather than to the first of the Hetzner assigned additional IP range that the DomUs live in. The domU does not bring up eth0 at all, hence has no network. Attempts to force-enable eth0 result in SIOCADDRT: No such device
I create my vif in the config file like so:
vif = [ ‘ip=hetzner.ip.address.here’ ]
and then the domU has a valid nic
Thanks a lot!
So do I, using one of the Hetzner-assigned additional IPs…
In xen-tools.conf I have dutifully
(vif-script vif-route)
(network-script network-route)
enabled and all alternatives disabled.
In /etc/network/interfaces on dom0 I have
******
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address MainIP
broadcast MainBroadcast
netmask MainNetmask
gateway MainGateway
auto eth0:1
iface eth0:1 inet static
address 1stAdditionalIPAfterAdditionalGateway
netmask AdditionalNetmask
broadcast AdditionalBroadcast
gateway AdditionalGateway
********
In xen-tools.conf finally I entered
*******
gateway = 1stAdditionalIPAfterAdditionalGateway
netmask = AdditionalNetmask
broadcast = AdditionalBroadcast
*******
These values correctly end up in DomU’s /etc/network/interfaces for eth0 with the IP being 2ndAdditionalIPAfterAdditionalGateway. Yet, DomU’s eth0 does not come up…
things might be different when using xen-tools, but I notice you edited xen-tools.conf for the route settings you should be editing xend-config.sxp if you’re on a standard xen setup other than that I dont know, I am using stock std xen.
sorry for the error —
(vif-script vif-route)
(network-script network-route) is, in fact, specified in xend-config.sxp as it should be, not in xen-tools.conf as mistakenly noted in my previous post
You should also not be binding the ip to eth0:1 on the dom0, don’t need to do anything with them at all, just leave them uncofigured let your domU take them. If you put the same one in your dom0 and U there will be a clash and the routing won’t know where to send packets
Thanks for the tip, but, alas, no luck 🙁 I removed the eth0:1 binding on Dom0, but that doesn’t make DomU any happier. What surprises me is that the device does not even show up in DomU in whatever form and refuses to be created. Seems Ubuntu may have an issue here…
yip, gotta be something ubuntu related unfortunately I am a RedHat guy so can’t help much there.
Maybe a lesson to be learnt there, I fear… I, for one, have used Debian for many years now, but am fed up by the Ubuntu + Xen combination… Pity, CentOS is not one of Hetzner’s prepared install images.
In any case, many thanks for your help!
While CentOS isn’t supported right now it used to be, and I wrote some documentation on the Hetzner wiki on getting CentOS going on the new hardware:
http://wiki.hetzner.de/index.php/DS8000_/_CentOS_5_/_VNC_Install
…well someone put it there after I posted that Howto on the forums.
I’ve searched over internet to find a solution to install vmware on hetzner using bridged network setup. I’ve not found anything…
I’ve studied and I’ve installed VMWARE SERVER correctly. I would share my post hoping that is usefully for you
http://www.smokinglinux.com/vmware-server-virtualization/install-vmware-server-on-debian-etch-in-an-hetzner-rootserver
I’m experiencing virtual problems of the exact same kind as Claudius did a few months ago (failure to bring up eth0 inside a guest, using XEN 3.1 and Ubuntu Feisty). Did a solution perchance emerged in the meantime? Or is there a better way to go about virtualization with Hetzner hosting: a set-up known to work? I have a server idly spinning there, and am, of course, quite eager to put it to use.
HI,
we’re also using Hetzner server – it costs a lot of time to get XEN working 🙂 but works now.
I’m trying to get my 2nd NIC (LAN) into the DomUs!
Do you use a 2nd nic for your server, and are you providing the 2nd nic to the domU’s as well?
Could you give me a hint?
thanks,
pmo
Unfortunately I only use the one nic.
I built a new Xen machine last night from scratch, CentOS 5.1 (not officially supported on Hetzner) and end to end it took me less than a hour…not so much work 🙂
Now i would say exactly the same – quite easy to install xen!
But you have to know little thing (like hetzner’s subnet is already routed to the server’s ip, brigding is not supported, grub in debian4.0 only recognizes 3.5 gb of 8gb …)
I setted up the whole system yesterday in more or less than an hour.
Thanks for your answer.
Even easier: reset the Realtec NIC during boot. Edit /etc/sysconfig/network-scripts/ifcfg-eth0 and change the keyword HWADDR to MACADDR.
See also http://www.centos.org/docs/5/html/5.1/Deployment_Guide/s2-networkscripts-interfaces-eth0.html
Hetzner today took our machines of the network becouse the nic uses the ff:ff:ff mac. claiming this is spoofing….
I wonder how many other customers are affected.
Hi,
Is it possible to contact you via email. We’re also trying to get centos 5.1 + xen working on a hetzner server but we’re experiencing some network issues.
Hi,
When using Xen routing, am I correct that you need to provide the DomU’s IP address in the config file? E.g:
vif = [“ip=123.123.123.123”]
That’s what I’ve been doing and it works well, however I don’t know how to get virt-install to create a config file without trying to use a bridge. Do you use virt-install for your guests, if so, how do you get it to create a correct vif line in the config?
Thanks!
Yeah unfort for the routed setup you need to gice it the ip in that way its a pain.
My configuration management system – puppet – does my installs, not virt install so i control it through there.
I’ve abandoned using virt-install now, as it isn’t possible to tell it to set up vifs in a routed way (it always tries to set up a bridged vif)
Thanks!
Why not use xen-tools? There’s no problem using alien to get debootstrap and rinse on centos. Just grab debian packages. 🙂