Extend snmpd to add detailed CPU statistics, per CPU (again)

For easier use with Cacti, it is easier to group statistics per type instead of per CPU. So you would have a parent OID for CPU time spent by user with many values (one per CPU).

Put the following in /etc/snmp/percpudetail:

case "$1" in
    grep ^cpu /proc/stat | awk '{print 2}' ;;
    grep ^cpu /proc/stat | awk '{print 3}' ;;
    grep ^cpu /proc/stat | awk '{print 4}' ;;
    grep ^cpu /proc/stat | awk '{print 5}' ;;
    grep ^cpu /proc/stat | awk '{print 6}' ;;
    grep ^cpu /proc/stat | awk '{print 7}' ;;
    grep ^cpu /proc/stat | awk '{print 8}' ;;

Make it executable:

chmod +x /etc/snmp/percpudetail

Then in your /etc/snmp/snmpd.conf file, put the following lines:

extend cpuuser /etc/snmp/percpudetail user
extend cpunice /etc/snmp/percpudetail nice
extend cpusystem /etc/snmp/percpudetail system
extend cpuidle /etc/snmp/percpudetail idle
extend cpuiowait /etc/snmp/percpudetail iowait
extend cpuirq /etc/snmp/percpudetail irq
extend cpusoftirq /etc/snmp/percpudetail softirq

Restart snmpd and you can poll those new OIDs.

Posted in Computer, Linux, Networking | Leave a comment

Extend snmpd to add detailed CPU statistics, per cpu

I needed to export detailed CPU statistics from Linux servers using SNMP. While UCD-SNMP-MIB export some detailed stats, it only does it for the whole system. I may have missed something easier though :-)

So here is a BASH script, put it in /etc/snmp/percpustats:

/bin/grep "^$1 " /proc/stat | sed -e 's/^cpu[0-9]* *//' | tr ' ' '\n'

Make it executable:

chmod +x /etc/snmp/percpustats

This simple script takes a single argument, which is the cpu id you want to look at. By “cpu id”, I mean the first word of cpu* lines in /proc/stat . So cpu, cpu0, cpu1, etc. Here is a sample content of my /proc/stat file:

# cat /proc/stat
cpu  190769457 225375 18212019 746796473 443933 1361 1240780 0 346283 0
cpu0 98979159 94876 9197600 368299173 282702 1361 1240721 0 164117 0
cpu1 91790298 130498 9014419 378497299 161231 0 58 0 182166 0

In your snmpd config file, you will need to add one extra config line per CPU you want to look at. Let’s say you have 3 CPU in your computer, you would need to add the following lines to get stats for the system in general (cpu) and details for each cpu (cpu0,cpu1,cpu2):

extend cpu /etc/snmp/percpustats cpu
extend cpu0 /etc/snmp/percpustats cpu0
extend cpu1 /etc/snmp/percpustats cpu1
extend cpu2 /etc/snmp/percpustats cpu2

Restart snmpd .

You can check the output using snmpwalk:

snmpwalk -v 2c -c yourcommunity yourip NET-SNMP-EXTEND-MIB::nsExtendObjects

You should get something like this:

NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu".1 = STRING: 52183124
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu".2 = STRING: 2487538
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu".3 = STRING: 64930047
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu".4 = STRING: 13880645861
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu".5 = STRING: 1503768
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu".6 = STRING: 5350929
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu".7 = STRING: 57971310
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu".8 = STRING: 0
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu0".1 = STRING: 3557627
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu0".2 = STRING: 42604
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu0".3 = STRING: 4803282
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu0".4 = STRING: 1749637588
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu0".5 = STRING: 27945
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu0".6 = STRING: 1
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu0".7 = STRING: 65021
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu0".8 = STRING: 0
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu1".1 = STRING: 5861096
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu1".2 = STRING: 472784
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu1".3 = STRING: 6472817
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu1".4 = STRING: 1733427227
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu1".5 = STRING: 426662
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu1".6 = STRING: 918093
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu1".7 = STRING: 10555401
NET-SNMP-EXTEND-MIB::nsExtendOutLine."cpu1".8 = STRING: 0

Now you can use these in your favourite monitoring/graph tools.

Posted in Computer, Linux, Networking | Leave a comment

Unetlab: create a QEMU image

So you have your Unified Networking Lab (UNL) server running and you want to use arbitrary QEMU images with it. Here is how to create one.

All UNL data is stored under /opt/unetlab , images are stored under /opt/unetlab/addons and QEMU images are a folder deeper under /opt/unetlab/addons/qemu .

UNL expects to find images within directories named according to some scheme. The directory names must start with some keywords followed by anything you want. For Linux images, you need to name the directory linux-something, e.g. linux-ubuntu-14.04.5-amd64. You can find the list of supported image “types” in /opt/unetlab/html/templates. The names of the files are the prefixes needed for directory names. If you want to, you can edit those files to change the default parameters of the images (make a backup before you change anything).

UNL expects the disk image file to be named hda.qcow2 . You do not have any choice here.

Now, let’s create a Debian image.

Log on UNL server using ssh (username=root password=unl).

Let’s create a new directory for our image:

cd /opt/unetlab/addons/qemu
mkdir linux-debian-8.3.0-i386
cd linux-debian-8.3.0-i386

Now you have two options, either you create an image on another computer and copy its disk image on UNL server, either you create the image on the UNL server itself. I’ll explain how to do the latter.

Download a Debian ISO file from your favorite mirror:

wget http://cdimage.debian.org/debian-cd/8.3.0/i386/iso-cd/debian-8.3.0-i386-netinst.iso

Create a disk image for the virtual machine:

/opt/qemu/bin/qemu-img create -f qcow2 hda.qcow2 4G

Start the virtual machine manually with a ISO file as virtual CDROM, booting on that ISO file and with a network interface:

/opt/qemu-2.0.2/bin/qemu-system-i386 -m 256 -smp 1 \
-cdrom debian-8.3.0-i386-netinst.iso -boot d \
-hda hda.qcow2 -monitor stdio -vnc -k fr-be \
-device e1000,netdev=net0,mac=50:02:12:34:00:00 \
-netdev tap,id=net0,ifname=debian0,script=no -S

You can now attach to the console (virtual screen) of the virtual machine using VNC (use the IP address of the UNL server and the screen number 10, e.g. You are going to get a black screen, that is normal because the last parameter of the qemu-system-i386 command is -S which means freeze CPU at startup (basically pause). To unpause the virtual machine, type cont in the QEMU console you got.

QEMU 2.0.2 monitor - type 'help' for more information
(qemu) cont

You should see the Debian install menu.

With the netinst image, the virtual machine will need Internet access. You have multiple options to achieve this: * connect the virtual machine network directly to your physical network * route the traffic between your physical network and the virtual machine * NAT the traffic of the virtual machine on the UNL server

I am going to go with the third option.

You can use one of the default bridge interfaces that UNL creates or use a new one. Let’s use a new one, we will call it brdebian and assign it the IP address

brctl addbr brdebian
ip link set brdebian up
ip address add dev brdebian

UNL does not come with iptables installed but we need it to enable NAT.

apt-get install iptables

Let’s tell the kernel we want it to NAT everything that comes from the subnet we chose:

iptables -t nat -A POSTROUTING -s -j MASQUERADE

and let’s tell the kernel it can forward IP packets:

echo 1 | tee /proc/sys/net/ipv4/ip_forward

Finally, let’s attach the virtual machine network interface to this network

brctl addif brdebian debian0
ip link set debian0 up

Assign an IP address from subnet to the virtual machine, set its gateway to and it will get Internet access.

Customize your image as you wish. When you are done, power it off, make a backup of the hda.qcow2 file (just in case).

You can now use that image in UNL.

Note that all the network changes we did are temporary and will disappear when you reboot the UNL server.

Posted in Computer, Linux, Networking | Leave a comment

Cisco IOS PPTP server : PPP: Packet throttled, Dropping packet

If you get the following message on your Cisco router when you connect a VPN client using PPTP

*Mar 17 16:43:02.371: Vi5 PPP: Control packet rate limit 10 reached
*Mar 17 16:43:02.371: Vi5 PPP: Entering block state for 30 seconds
*Mar 17 16:43:02.371: Vi5 PPP: Packet throttled, Dropping packet

Then it means you have an IOS version which greatly reduced the allowed rate of PPP control packets before it triggers a block.

Unfortunately, Windows 7 client sends packets at a rate that exceeds Cisco new default value. The result is that it takes 30 seconds (the default block duration) for the client to connect (yes, it connects successfully).

To fix this, you can change the rate threshold on the Cisco router by adding this line to your config:

ppp packet throttle 20 1 30

It tells the router to accept up to 20 packets per 1 second, and block for 30 seconds if this rate is exceeded.

Posted in Cisco, Networking | Leave a comment

Microsoft Windows 7 PPTP issues: spurious ICMP protocol-unreachable sent

I was recently confronted to a strange issue with a PPTP VPN connection to a central site. Some users could connect and some others could not. They all used Windows 7 with SP1, configured the same way, and all computers were behind NAT/PAT routers but not necessarily on the same site.

On the VPN server, the only information I could get was this log stating the GRE protocol was unreachable:
Feb 23 09:12:29 pppd[9712]: pppd 2.4.4 started by root, uid 0
Feb 23 09:12:29 zebra[2422]: interface ppp2 index 359 <POINTOPOINT,NOARP,MULTICAST> added.
Feb 23 09:12:29 pppd[9712]: Connect: ppp2 <--> /dev/pts/8
Feb 23 09:12:29 pptpd[9711]: GRE: read(fd=7,buffer=608c80,len=8260) from network failed: status = -1 error = Protocol not available
Feb 23 09:12:29 pptpd[9711]: CTRL: GRE read or PTY write failed (gre,pty)=(7,6)
Feb 23 09:12:29 pppd[9712]: Modem hangup
Feb 23 09:12:29 pppd[9712]: Connection terminated: no multilink.

I saw that I was getting ICMP protocol-unreachable packets from the WAN IP of the client. Using wireshark on the client, I saw that the packets were sent from it.

After Googling a bit, I found this post and that post.
They described exactly what was happening: some computers sent the ICMP protocol-unreachable and some others did not. The common point between computers that did send the packets was the Windows firewall was turned off.

Where applicable, I turned on the Windows firewall and it blocked those packets, and then the VPN connection was stable and working well.

To fix the issue for computers where turing the firewall on was not possible, I blocked the packets on the NAT/PAT router itself.
With a Linux router, you can do it by dropping the packets in the FORWARD rule of the FILTER table: iptables -I FORWARD -p icmp --icmp-type protocol-unreachable -d x.x.x.x -j DROP.
With a Cisco router, you need to add an ACL entry on the LAN interface: deny icmp any x.x.x.x y.y.y.y protocol-unreachable . If you had no ACL on the interface before, don’t forget to add a permit ip any any after the deny rule and before you apply the ACL on the interface, or you’ll deny all the traffic (Cisco has an implicit deny at the end of ACLs).

Stupid bugs…

Posted in Computer, Microsoft, Networking | Tagged , , , , , , , | Leave a comment

Ubuntu 14.04 and USB to Serial ch341 (chinese device from Ebay)

You can find pretty cheap USB to Serial devices on Ebay. I bought one and received a device using a chip ch341.

Unfortunately, the drive is bugged in Ubuntu 12.04 and 14.04. Fortunately, there is patch to fix it.

Here is how to recompile the module to enjoy these devices.

$ sudo apt-get source linux-source-3.13.0 linux-headers-$(uname -r)
$ cd /tmp
$ tar -xjf /usr/src/linux-source-3.13.0.tar.bz2
$ cd linux-source-3.13.0/
$ make oldconfig
$ make prepare
$ make scripts
$ cp -v /usr/src/linux-headers-$(uname -r)/Module.symvers .
$ cp /lib/modules/$(uname -r)/kernel/drivers/usb/serial/ch341.ko /lib/modules/$(uname -r)/kernel/drivers/usb/serial/ch341.ko.orig
$ cd drivers/usb/serial
$ cp ch341.c ch341.c.orig
$ wget https://github.com/karlp/ch341-linux/raw/master/0001-usb-serial-ch341-Add-parity-support.patch
$ cat 0001-usb-serial-ch341-Add-parity-support.patch | patch -p4
patching file ch341.c
Hunk #1 succeeded at 349 (offset 3 lines).
Hunk #2 succeeded at 370 (offset 3 lines).
$ make -C /lib/modules/$(uname -r)/build M=$(pwd) modules
$ sudo cp ch341.ko /lib/modules/$(uname -r)/kernel/drivers/usb/serial/ch341.ko
$ sudo rmmod ch341
$ sudo modprobe ch341

Source for instructions: askubuntu.com/questions/515407/how-recipe-to-build-only-one-kernel-module

Posted in Computer, Linux | 6 Comments

Update GNS3 server on GNS3 IOU VM

GNS3 Sourceforge account provides an OVA image ready to run gns3-server for those who want to use Cisco IOS on Unix images . However, at the time of this writing, the gns3-server version is outdated (1.3.3). The client must have the same version as the server to be able to connect to it.

Here is how you can update the gns3-server inside the image. Note that you need Internet access for the update.

  • Download the image from Sourceforge, run it with your favorite hypervisor.
  • Open a console and login, username=root and password=cisco .
  • Type pip search gns3-server to make sure you have Internet access.
  • If it succeeds, then type pip install gns3-server==1.3.10 to install latest version (at the time of this writing, it is 1.3.10).
  • Then type reboot.

Now your client running 1.3.10 should be able to connect to the server successfully.

Posted in Computer, Linux, Networking, Virtualization | Leave a comment

Migrate Debian 6+ to another server with minimal downtime

Recently I had to migrate services from a running Debian server to another one, with minimal downtime of services.
I usually do this to P2V or V2V Linux systems, as this allows me to resize the new virtual machine to meet the services requirements (adjust disk size, inodes, partitionning, etc.).
I have done this several times in the past on systems with Grub 1 but this is the first time with Grub 2, so I thought I’d share my process.

Continue reading

Posted in Computer, Linux | Leave a comment

SNOM phones and presence monitoring

If you have issues with your SNOM phones and presence monitoring (the little LEDs that display when another extension is ringing or busy), then maybe this will help.

We had this problem at work. When we checked the SIP traces, we could see our server was sending SIP NOTIFY packets to tell the phone “hey your collegue phone is ringing!”, but the phone:
– (very old firmware) either answered with a 200 OK but did not blink the LED;
– (latest firmware) or it answered a 481 Call/Transaction Does Not Exist and still did not blink the LED.

Continue reading

Posted in Computer, VoIP | Leave a comment

Nagios and LSI RAID cards

To monitor the status of a LSI RAID card, say for example a Dell PERC card, you will need to install NRPE, sudo, mpt-status and check_mpt.sh .

Install sudo and NRPE via your package manager. You can grab mpt-status via apt if you use Debian/Ubuntu or here if you use CentOS.  You can grab check_mpt.sh here.

Continue reading

Posted in Computer, Linux | Leave a comment