SSH socks proxies hanging
I use SSH's socks proxy feature a lot, in fact I use it all the time, most of my browsing, IM, etc all goes over it out via my hosted virtual machines,
I do this to simplify my life for things like firewall rules and also to get around things like age blocks on mobile networks. I work for a site deemed adult by most of them so I can't even see my nagios without age verifying.
Recently they have been driving me nuts, every now and then the whole session would just lock up and sit there doing nothing, I've not seen this happen before and was a bit stumped.
Turns out, it chooses to speak to TCP/53 sometimes instead of UDP/53 for resolving, not sure why exactly, I've not tried to figure out what queries cause this - I know there are limits to response sizes which will force it to go over TCP. Why it's only started doing this now I don't know, maybe a update changed behavior, I've never had TCP/53 open on the cache.
My firewall was blocking TCP/53 on the local cache so this would lock up the whole ssh session, maybe the whole ssh process is single threaded and so waiting in SYN_SENT mode just hangs the whole thing, that's a bit sucky, I might need a better proxy.
Compiling Custom Kernels in Debian
Things aren't how they used to be.
It seems I spent most of the '90s and the early '00s (damn you Y2K!) building my own kernels. The preposterous thought of sticking with the default kernel of your chosen distribution simply never crossed a lot of minds. For one thing, your hardware was pretty much guaranteed not to work out of the box, and RAM was expensive! It was in your best interests to tweak the hell out of the settings to get the best performance out of your hardware, and to avoid compiling in any unnecessary code or modules, to reduce your system memory footprint.
This involved much learning of new terminology and options, many failures and unbootable systems, and many trips to the local steakhouse or coffee shop, since each compile would take hours on your trusty 80386 or 80486.
Things aren't the way they used to be, thank goodness. Chances are, the vanilla kernel you received with your latest Masturbating Monkey Ubuntu 13.0 installation performs well and works with most of your hardware. You don't need to roll your own kernel. In fact, you should probably avoid it, especially if you're thinking of installing servers in a live production environment.
Well, usually. Sometimes, you really need to tweak some code to get the feature or performance you were counting on, or try out some awesome new patch which might just revolutionise the systems you are developing.
I'm a sucker for Debian. I love dpkg and apt(-itude). The package manager is powerful and I enjoy using it. I like everything except building packages from source. It's rarely as straightforward as it should be, and sometimes it's incredibly difficult to obtain a package which installs the same way and into the same places with the same features as the upstream pre-built package that you're supposedly building.
Building kernel packages is worse yet. Far worse. When rolling your own kernel, especially if you don't want the package manager to install the latest version over your own, you are forced to play ball with apt, and you must play by apt's rules.
I've tried a multitude of incantations of dpkg-buildpackage, debuild, make-kpkg, etc. All I want to be able to do is patch the kernel source, make some changes, append a custom version tag, and build a .deb which I can safely install, yet each HOWTO or set of instructions I tried failed to do this to my (misguided?) specifications. I had particularly nasty problems with the grub update post-inst scripts in all cases.
Load Balancing with HAProxy
Load Balancers are some of the most expensive bits of equipment small to medium size sites are likely to buy, even more expensive than database servers.
Since I help a number of smaller and young startups a good Open Source load balancer is essential, I use HAProxy for this purpose.
HAProxy is a high performance non threaded load balancer, it supports a lot of really excellent features like regular expression based logic to route certain types of requests to different backend servers, session tracking using cookies or URL parts and has extensive documentation.
Getting a full redundant set of load balancers going with it requires the help of something like Linux-HA which I use extensively for this purpose, the combination of HAProxy and Linux-HA gives you a full active-passive cluster with failover capabilities that really does work a charm.
I recently had to reload a HAProxy instance after about a 100 day uptime, its performance stats were 1.8 billion requests, 15TB out and just short of 2TB in
Worth checking out HAProxy before shelling out GBP15 000 for 2 x hardware load balancers.
SixXS IPv6 and CentOS
I thought its high time I get to spend some time with IPv6 so I signed up for a static tunnel from sixxs.net, apart from taking some time it's a fairly painless process to get going.
I chose a static tunnel since I am just 9ms from one of their brokers and my machine is up all the time anyway, they have some docs on how to get RedHat machines talking to them but it was not particularly accurate, this is what I did:
You'll get a mail from them listing your details, something like this:
Tunnel Id : T21201
PoP Name : dedus01 (de.speedpartner [AS34225])
Your Location : Gunzenhausen, de
SixXS IPv6 : 2a01:x:x:x::1/64
Your IPv6 : 2a01:x:x:x::2/64
SixXS IPv4 : 91.184.37.98
Tunnel Type : Static (Proto-41)
Your IPv4 : 78.x.x.x
Using this you can now configure your CentOS machine to bring the tunnel up, you need to edit these files:
/etc/sysconfig/network
NETWORKING_IPV6=yes
IPV6_DEFAULTDEV=sit1
/etc/sysconfig/network-scripts/ifcfg-sit1
DEVICE=sit1
BOOTPROTO=none
ONBOOT=yes
IPV6INIT=yes
IPV6_TUNNELNAME="sixxs"
IPV6TUNNELIPV4="91.184.37.98"
IPV6TUNNELIPV4LOCAL="78.x.x.x"
IPV6ADDR="2a01:x:x:x::2/64"
IPV6_MTU="1280"
TYPE=sit
Just replace the values from your email into the files above, once you have this in place reboot or restart your networking and you should see something like this:
% ifconfig sit1
sit1 Link encap:IPv6-in-IPv4
inet6 addr: 2a01:x:x:x::2/64 Scope:Global
inet6 addr: fe80::4e2f:c3c6/128 Scope:Link
UP POINTOPOINT RUNNING NOARP MTU:1480 Metric:1
RX packets:6383 errors:0 dropped:0 overruns:0 frame:0
TX packets:6431 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4072476 (3.8 MiB) TX bytes:921722 (900.1 KiB)
% ping6 -c 3 -n noc.sixxs.net
PING noc.sixxs.net(2001:838:1:1:210:dcff:fe20:7c7c) 56 data bytes
64 bytes from 2001:838:1:1:210:dcff:fe20:7c7c: icmp_seq=0 ttl=57 time=20.2 ms
64 bytes from 2001:838:1:1:210:dcff:fe20:7c7c: icmp_seq=1 ttl=57 time=28.4 ms
64 bytes from 2001:838:1:1:210:dcff:fe20:7c7c: icmp_seq=2 ttl=57 time=20.1 ms--- noc.sixxs.net ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2008ms
rtt min/avg/max/mdev = 20.181/22.934/28.406/3.869 ms, pipe 2
Since this is a remote machine it took me some time to figure out how to get browsing going through it, but once I reconnected my SSH SOCKS tunnel it immediately became IPv6 aware and were happily routing me to sites like ipv6.google.com. To do this just run from your desktop:
ssh -D 1080 yourbox.net
Now set your firefox network.proxy.socks_remote_dns setting to true in about:config, and point your browser at localhost:1080 as a socks proxy, your SSH should now work as a perfectly effective ipv4-to-6 gateway. You can test it by browsing to either the sixxs.net homepage or ipv6.google.com - watch out for the special google logo.
Location aware Bind for RedHat 5.3
Previously I wrote about RPMs I built to GeoIP enable Bind using the original patches at http://www.caraytech.com/geodns/.
I have now refreshed this for the latest CentOS 5.3, the details of the patch, install instructions etc has not changed, read the previous article I wrote for the details, the new RPMs are below:
NOTE: When you install these RPMs you won't see a /etc/named.conf being created and a few other odd things, these are bugs that exist in the CentOS provided RPMs, they do the same.
bind-9.3.4-10.P1geodns.el5.i386.rpm
bind-chroot-9.3.4-10.P1geodns.el5.i386.rpm
bind-devel-9.3.4-10.P1geodns.el5.i386.rpm
bind-libbind-devel-9.3.4-10.P1geodns.el5.i386.rpm
bind-libs-9.3.4-10.P1geodns.el5.i386.rpm
bind-utils-9.3.4-10.P1geodns.el5.i386.rpm
bind-sdb-9.3.4-10.P1geodns.el5.i386.rpm
caching-nameserver-9.3.4-10.P1geodns.el5.i386.rpm
bind-9.3.4-10.P1geodns.el5.x86_64.rpm
bind-chroot-9.3.4-10.P1geodns.el5.x86_64.rpm
bind-libbind-devel-9.3.4-10.P1geodns.el5.x86_64.rpm
bind-devel-9.3.4-10.P1geodns.el5.x86_64.rpm
bind-libs-9.3.4-10.P1geodns.el5.x86_64.rpm
bind-sdb-9.3.4-10.P1geodns.el5.x86_64.rpm
bind-utils-9.3.4-10.P1geodns.el5.x86_64.rpm
caching-nameserver-9.3.4-10.P1geodns.el5.x86_64.rpm
CentOS 5.3
CentOS 5.3 was released on the 1st of April, I've since updated a whole lot of my machines to this version and been very happy.
There are a few gotchas, mostly well covered in the release notes, the only other odd thing I found was that /etc/snmp/snmpd.options has now moved to /etc/sysconfig/snmpd.options ditto for snmptrapd.options. It's a bit of a weird change, while it makes the SNMPD config a bit more like the rest of the RedHat system, it still is different, you'd think based on all the other files in /etc/sysconfig that this one would have been called /etc/sysconfig/snmpd rather than have the .options bit tacked on.
Other changes that I noticed is that Xen is behaving a lot better now on suspends, if I reboot a dom0 and then bring it back up the domU's resume where they were and unlike the past the clocks do not go all over the place, in fact I've even seen SSH sessions stay up between reboots. Though SNMP still sometimes stop working after resume.
The general overall look of the distribution is much better, the artwork has been redone through out and now forms a nice cohesive look and feel through out.
While investigating the cause of the /etc/snmp/snmpd.options file mysteriously going missing I once again had the miss fortune of having to deal with #centos on freenode. It really is one of the most hostile channels I've come across in the opensource world, people are just outright arseholes, every one including the project leaders.
Immediately assuming you have no clue, don't know what you're talking about and generally just treating everyone like shit who dare suggest something is broken with the usual 'works for me' 'read the docs' or 'its in the release notes' or 'looking at the source will not help' style responses to every question. When as it turns out every one of those remarks were just plain wrong. No it didn't work for them, their files also got moved by the installer. No it was not in the docs or release notes. No looking at the source would have helped a lot more than they did because I would have then been able to see for myself that the post install of the RPM moves the files etc. It took literally over a hour to get even one of them to actually make the effort to be helpful compared to about 2 minutes it would have taken if the SRPMs were available at release time.
I think they're really doing the project a big disservice by not sorting out the irc channel in fact they actively defend and even promote the hostility shown there, in contrast to the puppet irc channel for instance it really is a barbaric bit of the 3rd world.
MySQL Defaults and Load time
We all know not to use the default mysql config, right?
Well I accidentally left a machine to defaults, then tried to load a massive dump file into it, a month later I finally killed the process loading the data. I gave up on it ages ago but it got to the point where it was some curiosity to see just how long it will take.
iptables chains
A lifetime ago when I still gave a damn for FreeBSD I wrote about ipfw tables, I still really love ipfw's simple syntax and really wish there was something similar for Linux rather than Human Error Guaranteed convoluted syntax mess that's iptables.
Anyway, so in my case I have machines all over, one off VPS machines, dom0's with a subnet routed to them and so forth. I often have rules that need to match on all my ips, things like allow data into my backup server, allow config retrieval from my puppetmaster etc. I do not want to maintain my total list of ips 10 times over so how to deal with it?
This is a good fit for ipfw tables, you create a table - essentially an object group like in a Cisco PIX or ASA - and then use it to match source IPs.
In the last week I've asked quite a few people how they'd do something similar with iptables but no-one seemed to know, I had people who were happy to maintain the same list many times. People who would use tools like sed to insert it into their rules and everything in between. I think I know a better way so I figured I'll blog about it because it's obviously something people do not just understand.
Iptables ofcourse use chains, and you can jump to and from chains all you want, this is very simple, so lets create a chain with all my IPs
-A my_ips -s 192.168.1.1 -m comment --comment "box1.com" -j ACCEPT
-A my_ips -s 192.168.2.1 -m comment --comment "box2.com" -j ACCEPT
-A my_ips -s 192.168.3.1 -m comment --comment "box3.com" -j ACCEPT
This creates a chain my_ips that just accepts all traffic from my IP addresses, now lets see how we'd allow all my ip addresses into my webserver?
-A INPUT -p tcp --dport 80 -m tcp -j my_ips
So this is something almost as good as a ipfw table, I can reuse it many times on many machines and my overall configuration is much more simple. It's not quite as powerful as a table but for my needs it's fine.
Combined with a tool like Puppet that manages your configurations you can ensure that this chain is installed on any machine that uses iptables, ready to be used and also trivial to update whenever you need too without having to worry about human error incurred from having to maintain many copies of essentially the same data.
In my environment when I update this table, I check it into SVN and within 30 minutes every machine in my control has the new table and they've all reloaded their iptables rules to activate it. Testing is very easy since puppet allows you to use environments similar to Rails has and so if I really need to I can easily test firewall changes on a small contained set of machines, distributed object group management with version control and everything rolled into one.
RedHat Linux <-> Cisco ASA IPSEC VPN
I've previously mentioned the really great syscfg integrated IPSEC on RedHat Linux here but thought I'd now show a real world example of a Cisco ASA and a RedHat machine talking since it is not totally obvious and it is not something I seen specifically documented anywhere using Google.
A quick recap: RedHat now lets you build IPSEC VPNs using just simple ifcfg-eth0 style config files.
I'll quickly show both sides of the config to build a site to site VPN, Site A is a Linux machine with a real IP address while Site B is a Cisco ASA with a private network behind it, the Linux machine has this in /etc/sysconfig/network-scripts/ifcfg-ipsec1:
TYPE=IPSEC
ONBOOT=yes
IKE_METHOD=PSK
SRCGW=1.2.3.4
DSTGW=2.3.4.5
SRCNET=1.2.3.4/32
DSTNET=10.1.1.0/24
DST=2.3.4.5
AH_PROTO=none
The pre-shared key is in /etc/sysconfig/network-scripts/keys-ipsec1 as per the RedHat documentation.
The Cisco ASA does not support AH so the big deal here is to disable AH which turns out to be the magic knob to tweak here to make it work.
In this case the Linux Server on Site A has the IP address 1.2.3.4 while the ASA is running on 2.3.4.5, the private network at Site B is 10.1.1.0/24.
On the Cisco the relevant lines of config are:
object-group network siteb_to_sitea_local_hosts
description Site B to Site A VPN Local hosts
network-object 10.1.1.0 255.255.255.0
object-group network siteb_to_sitea_remote_hosts
description Site B to Site A VPN Remote Hosts
network-object 1.2.3.4 255.255.255.255access-list siteb_to_sitea_vpn extended permit ip object-group siteb_to_sitea_local_hosts object-group siteb_to_sitea_remote_hosts
access-list inside_nat_bypass extended permit ip object-group siteb_to_sitea_local_hosts object-group siteb_to_sitea_remote_hosts
nat (inside) 0 access-list inside_nat_bypass
crypto map outside_map 20 match address siteb_to_sitea_vpn
crypto map outside_map 20 set pfs
crypto map outside_map 20 set peer 1.2.3.4
crypto map outside_map 20 set transform-set ESP-3DES-SHA
crypto map outside_map 20 set security-association lifetime seconds 3600
crypto map outside_map 20 set security-association lifetime kilobytes 4608000
crypto map outside_map interface outside
crypto isakmp enable outsidecrypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto isakmp policy 20
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 28800tunnel-group 1.2.3.4 type ipsec-l2l
tunnel-group 1.2.3.4 ipsec-attributes
pre-shared-key secret
Using these specific phase 1 and phase 2 parameters - timings, pfs, crypto etc - means that it will match up with the default out-the-box parameters as per /etc/racoon/racoon.conf thereby minimizing the amount of tweaking needed on the RedHat machine.
All that is needed now is to start the VPN using /etc/sysconfig/network-scripts/ifup ifcfg-ipsec1 and you should be able to communicate between your nodes.
Detailed Apache Stats
Apache has its native mod_status status page that many people use to pull stats into tools such as Cacti and other RRDTool based stats packages. This works well but does not always provide enough details, questions such as these remain unanswered:
- How many of my requests are GET and how many are POST?
- How many 404 errors and 5xx errors do I get on my site as a whole and for script.php specifically?
- What is the average response time for the whole server, and for script.php?
- How many Closed, Keep Alive and Aborted connections do I have?
To
answer this I wrote a script that keeps a running track of your Apache
process, it has many fine grained controls that let you fine tune
exactly what to keep stats on. I got the initial idea from an old ONLamp article titled Profiling LAMP Applications with Apache's Blackbox Logs.
The
article proposes a custom log format that provides the equivelant to an
airplanes blackbox, a flight recorder that records more detail per
request than the usual common log formats do. I suggest you read the
article for background information. The article though stops short of a
full data parser so I wrote one for a client who kindly agreed that I
can opensource it.
Using
this and some glue in my Cacti I now have graphs showing a profile of
the requests I receive for the whole site, but as you are able to
apply fine grained controls to select what exactly you'll see, you could get per server overview stats and details for just a specific scripts performance and statuses:
The script creates on a regular interval a file that contains the performance data, the data is presented in variable=value data pairs, I will soon provide a Cacti and Nagios plugin to parse this output to ease integration into these tools.
The
performance data includes values such as:
- Amount of requests in total
- Total size
of requests separated by in and out bytes - Average response time
- Total processing time.
- Counts of connections in Close, Keep Alive and Aborted
states. - Counts for each valid HTTP Status code,
and aggregates for 1xx, 2xx, 3xx, 4xx and 5xx. - The amount of GET and
POST requests. - And detail for each and every unique request the server
serves.
See the Sample Stats
for a good example, variables are pretty self explanatory. To keep the
data set small and manageable 2 selectors exist, one to choose which
requests to keep details for and which to keep stats for. These can be
combined with standard Apache directives such as Location to provide very fine grained stats for all or a subset of your site.
You would need some glue to plug this into Cacti and Nagios, I will provide a script for this soon as I have time to write up some docs for it.
Install guide etc can be found on my Wiki there is also extensive Perdoc Docs in the script, the Wiki also have links to downloading the script, the latest is always available here
Links
- @ripienaar
- Flashpolicyd
- Flickr
- London DevOps Group
- MCollective Plugins
- R.I.Pienaar
- Ruby PDNS
- The Marionette Collective
- Wiki
Tweets
- It's ironic how you cannot send email with google charts api images to gmail users and expect it to work. way to go.
- Google charting API really is very sexy
- High Scalability: Saying Yes to NoSQL; Going Steady with Cassandra at Digg http://bit.ly/9Kli5f
- Caffiene infused. 2am to bed day before delivering the last talk of the day is not recommended
- Once again discovering that something as simple as correct MX and NS records is a very effective CAPTCHA to weed out idiot hosting companies
- I love the E4 ipad ripoff ad http://srt.ly/epad
- RT @garnaat: The only accurate model of the system is the system itself. All other representations are a compromise. <- what he said
- Parallels announces support for Google Chrome Operating System http://bit.ly/aCXXCJ
- RT @brunns: RT @jaywgraves: This is pretty awesome. http://bit.ly/aJ12Gb
- Kelway are totally wicked hardware suppliers. Threw away a £100 cable by accident, they gave me a freebie replacement
Reading
Planned books:
- The Science of Fear: How the Culture of Fear Manipulates Your Brain by Daniel Gardner
- Relentless by Dean Koontz
- Mind Over Ship by David Marusek
- The New Space Opera by Gardner Dozois
- The New Space Opera 2 by Gardner Dozois, Jonathan Strahan
Current books:
-
Ariel: A Book of the Change by Steven R. Boyett
Recent books:
- Hacking Vim: A Cookbook to Get the Most Out of the Latest Vim Editor by Kim Schulz
- Beautiful Architecture: Leading Thinkers Reveal the Hidden Beauty in Software Design by Diomidis Spinellis , Georgios Gousios
Categories
Archives
Licence
