I have a number of FreeBSD machines with jails on them that require me to keep stats and graphs of their bandwidth usage.
The solution I came up with is to add counter rules in the kernel IPFW firewall table and then plug a simple perl script into Net SNMP which will put each ipfw counter rule’s current byte count on a unique OID that you can query and graph using something Cacti.
This same technique can be used to graph things like only HTTP, SMTP, etc traffic, or infact anything that you can express as a IPFW counter rule.
Read the full entry for details on how I implemented this.
A number of components make up the final working solution:
- FreeBSD
- IPFW counter rules
- A working installation of Net SNMP
- A tool to collect and graph the data, I use RRDtool and Cacti but you can also use MRTG
- A script to integrate into Net SNMP, I will provide a sample below.
I will not get into setting up the OS, Jails or base IPFW functionality, you can use the Handbook for that.
My example will have two jails configured with IP addresses 192.168.1.10 and 192.168.1.11. You will need counter rules that resemble these, my network interface is fxp0, replace with yours:
00100 67118324 49154926178 count ip from 192.168.1.10 to any out via fxp0
00110 59209784 11219191580 count ip from any to 192.168.1.10 in via fxp0
00120 59861887 21490069162 count ip from 192.168.1.11 to any out via fxp0
00130 60701763 10491223999 count ip from any to 192.168.1.11 in via fxp0
The format of these rules are pretty simple, the first numbers (100, 110 etc) are the unique rule number, followed by a accumulating packet count that this rule has counted and then a accumulated byte counter. I will use the byte counter as source but you can also use the packet counter if you wanted to count packets.
Once you have these setup and working we should explore some other required bits, again I am not going to show you how to setup and configure Net SNMP. There are a number of ways to integrate into snmpd, you can use the supplied perl API’s to write an AgentX plugin or you can simply execute a script and send the output back via snmp, I opted for the much simpler second option.
The snmpd.conf man page details all about the exec parameter and explains that calling a script with a line like exec MIBNUM NAME PROG ARGS will result in each line of STDOUT being returned in the MIBNUM.101 table with each line of STDOUT being one entry in this table.
The MIBNUM above should look something like .1.3.6.1.4.1.xxxx where the xxxx can be a number assigned to you by IANA, they assign them for free and you can apply here. If this is only a internal small scale thing, just choose something like 9999, I will use 9999 in my example.
A good thing about the Net SNMP exec method is that output from the script gets cached for 30 seconds before the perl script will get called again, therefore the moment your Cacti or MRTG hits it for the first query the output will get cached and all further queries in the next 30 seconds will get answered from the cache, the hit on your CPU therefore is very low.
We now know how to use ipfw to count the data, we will write a simple script to parse the output from ipfw show and we can plug it into snmpd using the exec call as above.
I have a simple script that you can use or base your own on, I placed mine in /usr/local/bin/getMultiIPFWByteCount.pl. The sample script can be plugged into snmpd.conf with the following line in the config:
exec .1.3.6.1.4.1.9999.1 ipfwCounters /usr/local/bin/getMultiIPFWByteCount.pl
A quick note about my script, it is hardcoded to only find count rules, if you want to match all rules simply change the parameters that gets called.
If you did all of this correctly and restarted your snmpd you can now make graphing queries to .1.3.6.1.4.1.9999.1.101.100 and .1.3.6.1.4.1.9999.1.101.110 for the ip address 192.168.1.10’s traffic. Similarly you can use .1.3.6.1.4.1.9999.1.101.120 and .1.3.6.1.4.1.9999.1.101.130 for 192.168.1.11.
