There are a few gotchas, mostly well covered in the release notes, the only other odd thing I found was that /etc/snmp/snmpd.options has now moved to /etc/sysconfig/snmpd.options ditto for snmptrapd.options.  It’s a bit of a weird change, while it makes the SNMPD config a bit more like the rest of the RedHat system, it still is different, you’d think based on all the other files in /etc/sysconfig that this one would have been called /etc/sysconfig/snmpd rather than have the .options bit tacked on.

Other changes that I noticed is that Xen is behaving a lot better now on suspends, if I reboot a dom0 and then bring it back up the domU’s resume where they were and unlike the past the clocks do not go all over the place, in fact I’ve even seen SSH sessions stay up between reboots.  Though SNMP still sometimes stop working after resume.

The general overall look of the distribution is much better, the artwork has been redone through out and now forms a nice cohesive look and feel through out.

While investigating the cause of the /etc/snmp/snmpd.options file mysteriously going missing I once again had the miss fortune of having to deal with #centos on freenode.  It really is one of the most hostile channels I’ve come across in the opensource world, people are just outright arseholes, every one including the project leaders. 

Immediately assuming you have no clue, don’t know what you’re talking about and generally just treating everyone like shit who dare suggest something is broken with the usual ‘works for me’ ‘read the docs’ or ‘its in the release notes’ or ‘looking at the source will not help’ style responses to every question.  When as it turns out every one of those remarks were just plain wrong. No it didn’t work for them, their files also got moved by the installer.  No it was not in the docs or release notes.  No looking at the source would have helped a lot more than they did because I would have then been able to see for myself that the post install of the RPM moves the files etc.  It took literally over a hour to get even one of them to actually make the effort to be helpful compared to about 2 minutes it would have taken if the SRPMs were available at release time.

I think they’re really doing the project a big disservice by not sorting out the irc channel in fact they actively defend and even promote the hostility shown there, in contrast to the puppet irc channel for instance it really is a barbaric bit of the 3rd world.

The first is called Virt-top it is a easier to read top like tool than xm top that shows all virtual machines memory and CPU usage in a nice display including totals etc:

Both of these come from a Red Hat employee, with some luck we’ll see these included in Red Hat Linux soon.

I’ve been using a guest kernel repository by one of the VMWare users that rebuilds the std CentOS/RedHat kernels with HZ=100 and it’s been great, chopped massive amounts off my CPU usage on the host.

Now with RedHat 5.1 this is not needed anymore see this post for a bit of a graph on the impact and the background.  The short of it is, simply append divider=10 to your guest kernel boot parameters and enjoy a much happier host.  I found that time keeping also becomes more predictable in the guest.

xenstored: corruption detected by connection 0: \
err No such file or directory: Write  failed
xenstored: clean_store: '/local/domain/0/backend/vbd/16/51712/sectors' is orphaned!

At this point you’re pretty much screwed, try and reboot and xend won’t even run, no VM’s will start.
Fixing it is pretty easy in the end once you’ve done tons of Googling and found the 2 year old bug in the Xen bugtracker about this exact problem – complete with the xen guys trying to close it in a routine ‘cleanup of tickets’ rather than actually fixing the bug.
First, shut down all things Xen, if you can even boot from a non Xen kernel. Once you’re sure its all down just delete /var/lib/xenstored/tdb* and reboot, it should all be fine after that.
You must be sure you don’t have xenstored running while doing this, else it will write its in-memory corrupted state back to disk when you reboot and it will look like your fix didn’t work.

• Hetzner uses a DHCP/TFTP rescue system that is very useful, if I make a mistake in a network config for example I can sort it out via this. XEN by default works in a bridged mode that will change the MAC address for your eth0 to FE:FF:FF:FF:FF:FF. The problem is that Hetzner uses Realtek NICs which as most people know are uber crap. When the machine gets a normal alt-ctrl-del or other reboot other than a power cycle the NIC does not reset to the original MAC. So if I get a crash I wont ever get to the rescue system. This cost me 119 Euro this weekend in support fees.
• In the past I had a few crashes on this machine, adding acpi=force irqpoll noapic to the kernel command line solved it, however the XEN kernel won’t poll my drives properly without removing the noapic bit, so I hope my machine doesn’t become unstable all of a sudden.

I am using CentOS 5.0 on this machine that came with XEN out of the box so installation was dead easy, creating new VM’s are easy etc and very importantly I can manage the VM’s all through text consoles which was a pain for me under VMWare as I don’t use Windows or Linux, getting console access was tough.
So about the MAC address, this turned out incredibly easy to fix by turning the machine into a routed network setup instead of a bridge, so previously the network was bridging eth0 with the vif’s from the virtual machines onto the wire, this required the MAC address of eth0 to change. Moving to a routed setup required the following changes:
/etc/sysctl.conf:

net.ipv4.ip_forward = 1
net.ipv4.conf.all.proxy_arp = 1

This basically enables routing and sets up proxy arp across all interfaces, this is to work with Hetzner’s routing where they don’t actually add a route to my subnet over my eth0 but over to the switch is connected too. Proxy ARP sorts this out.
/etc/xen/xend-config.sxp:

#(network-script network-bridge)
#(vif-script vif-bridge)
(vif-script vif-route)
(network-script network-route)

This points XEN to its provided route scripts rather than the bridge ones.
Reboot virtual machines and thats that. Machine is perfectly happy, routing traffic and MAC address is correct.
Bonus is now all traffic gets routed through my Dom0 and I can now do all the firewalling there outside of my DomU’s which enhances security as it is essentially a border firewall.

Virtualization enhanced by Intel Virtualization Technology will allow a platform to run multiple operating systems and applications in independent partitions. With virtualization, one computer system can function as multiple “virtual” systems. With processor and I/O enhancements to Intel’s various platforms, Intel Virtualization Technology can improve the performance and robustness of today’s software-only virtual machine solutions.

I’ve been very excited to get my hands on both hardware and software together that lets me explore this exciting technology and today that was delivered in the form of my Apple iMac Core Duo machine running Parallels Workstation.

Parallels Workstation 2.0 is the first desktop virtualization solution to include a lightweight hypervisor, a mature technology originally developed in the 1960s to maximize the power of large mainframes. Hypervisor technology dramatically improves virtual machine stability, security and performance by using a thin layer of software, inserted between the machine’s hardware and the primary operating system, to directly control some of the host machine’s hardware profiles and resources. It not only makes Parallels Workstation-powered virtual machines secure, stable and efficient, but also empowers users to immediately realize the benefits associated with Intel VT hardware virtualization architecture.

The good thing is it works on OS X on Intel hardware and allows you to run most operating systems. I grabbed a copy of FreeBSD 6.0 and immediately knew it was fast, only later did I realise how fast.
I wanted to see if I can get my 2nd Core to appear as a 2nd CPU in the guest system so I built the included GENERIC SMP kernel, the build time was just short of 17 minutes. The VM was allocated 256Mb RAM and was running on my 1800mhz Intel iMac. At the time I had Thunderbird, Deerpark, NetNewsWire, Terminal, Adium, Activity Monitor and MS Word (under Rosetta) open.
I then took my AMD 1800Mhz with 512Mb RAM machine also running FreeBSD 6.0 and did the same build, the build time on that was 18 minutes.
I realise this isn’t the most scientific of tests, but it far exceeds any performance indication on any VM I’ve ever seen, I’m really looking forward to one day having a Bladecenter full of machines capable of running this technology and doing a full VMWare based architecture, the advantages that virtualization brings to the table is staggering. If you ever get a chance to see a real demo of VMWare’s enterprise products I’d strongly suggest you jump at the oppertunity.