When the plugin was executed outside of Icinga it worked fine, online searches revealed nothing of use, I spent ages trying to adjust the config file and check command for the plugin but with no success.

Eventually I was able to track it down to the embedded Perl interpreter used by Icinga. The solution was fairly simple I just had to add nagios: -epn as a comment within the first 10 lines of the script. This told Nagios to use an external Perl interpreter when running this plugin.

This solution should apply to anything based on Nagios by the way: Icinga, Shinken, Centreon-Engine, Opsview, etc.

I’ve been doing some database work recently and I wanted to unify the character set and collation for all databases and tables. Changing the database collation was easy to do in bulk. The tables less so and I was not manually changing 13,014 tables, eventually I came up with this:

mysql --batch --skip-column-names --user=username --password=password --execute="select CONCAT('alter table ',TABLE_SCHEMA,'.',TABLE_NAME,' convert to character set utf8 collate utf8_general_ci;') from information_schema.TABLES WHERE TABLE_SCHEMA != 'information_schema' AND TABLE_SCHEMA != 'mysql' AND TABLE_SCHEMA != 'performance_schema';" | mysql --user=username --password=password

That should generate a bunch of statements that look like this:

alter table database_name.table_name convert to character set utf8 collate utf8_general_ci;

For every table in every database except the information_schema, mysql and performance_schema databases. I just pipe these right back in to MySQL but you can output them to a text file and run that later if you prefer. This snippet should also to be useful for any mass/multiple/bulk table edits or changes.

sd 5:0:0:0: [sdc] CDB: Read(10): 28 00 00 00 01 1e 00 00 01 00 00 00 end_request: critical target error, dev sdc, sector 286
sd 5:0:0:0: [sdc] Unhandled sense code
sd 5:0:0:0: [sdc]  Result: hostbyte=invalid driverbyte=DRIVER_SENSE
sd 5:0:0:0: [sdc]  Sense Key : Medium Error [current] Info fld=0x11f
sd 5:0:0:0: [sdc]  Add. Sense: Recorded entity not found

It’s most likely that these errors were caused by age but floppy disk drives also have differing alignment between the drive the data was written with and the drive used for reading the data.

There was only one thing I could do really and that was give ddrescue a try. There are two utilities called ddrescue, to be exact it was GNU ddrescue version 1.11. What I decided to do was try running ddrescue using three different read methods, on each disk in every drive (different alignments). My hope being that by writing to the same image file and using the logfile feature of ddrescue that I could amalgamate the good parts of each read into one complete image file.

Note: You’ll probably need to change the options below for each floppy device, file location, etc, if the drives are in different computers it’s probably easiest to save the files to some central location.

First we try to copy as much data as possible, without splitting sectors or retrying on failures:

sudo ddrescue --verbose --no-split --cluster-size=1 /dev/sdc floppy1.img floppy1.log

Now we retry any previous errors twice, using uncached reads:

sudo ddrescue --verbose --direct --max-retries=2 --cluster-size=1 /dev/sdc floppy1.img floppy1.log

Finally we try again but with the retrim option set, so that ddrescue will try to reread full sectors:

sudo ddrescue --verbose --direct --retrim --max-retries=2 --cluster-size=1 /dev/sdc floppy1.img floppy1.log

At this point you hopefully have a complete floppy disk image (I only had to use three of the available floppy drives) and then depending on the situation and damage you can either run the image through fsck, mount it, write it to a new disk or run it through a file carver.

In my situation I was able to use the mtools suite to copy the required files directly from the floppy disk image files.

1. No one reads anything.
2. They forget what they read.
3. They deny what they have read.
4. Users and clients will lie.
5. Fast, cheap, reliable. Pick two.
6. Always cover your ass, but come clean when it’s your fault
7. Always backup even when told not to.
8. If you don’t have a replacement it will break soon.
9. Generally no one understands what you’re talking about or doing.
10. Rebooting while not ideal will usually fix a problem.

No one reads anything, They forget what they read, They deny what they have read

All three of these are variations of the same principle, basically don’t make assumptions and expect people to have read anything. While it would be nice if they did review documentation, guides and announcements I’m not holding my breath.

Users and clients will lie

Some people will lie when you’re trying to troubleshoot, usually because they think they know what is wrong or because they’re feeling stupid or guilty and trying to cover themselves.

Fast, cheap, reliable. Pick two

Pretty self explanatory, something can be fast and cheap, fast and reliable or cheap and reliable, but not all three.

Always cover your ass, but come clean when it’s your fault

Don’t go overboard with this, I simply mean that you should get written confirmation (e-mail will do) whenever you’re asked to do something you think is illegal or whenever you want to do something that will potentially cause downtime. If you do happen to cause downtime then come clean immediately and explain what happened.

Always backup even when told not to

This is kind of like the covering your ass rule, but always backup and test those backups.

If you don’t have a replacement it will break soon

I don’t know why, call it unprecedented coincidence but every time I’ve ran out of replacements for something, even if it’s something that never needs replacing or was just replaced last week then I’ll need one.

Generally no one understands what you’re talking about or doing

This can be annoying but try to explain things in a simple and clear manner, using the terminology of the person you’re talking to while avoiding being patronising. As IT departments need to justify their existence sometimes it never hurts to keep time sheets or maintain a trouble/incident/help ticket system.

Rebooting while not ideal will usually fix a problem

We’re all aware of the ‘Have you tried turning it off and on again?’ stuff, it usually works but a lot of the time it just fixes the symptoms instead of the underlying problem and it’s good to keep that in mind.

Split a file at a word or pattern into multiple files:

awk '/Pattern to split at/{n++}{print > "split" n ".txt"}' FILE

Notes: Replace FILE with the file name you wish to run the command against.

Take a screenshot via SSH:

DISPLAY=:0.0 import -window root /path/to/directory/screenshot.png

Notes: None.

Search for something that looks like an e-mail address:

egrep -io '([[:alnum:]_.-]+@[[:alnum:]_.-]+?\.[[:alpha:].]{2,6})'

Notes: If don’t have egrep, replace egrep -io with grep -Eio.

Run a query on multiple tables in a database matching a pattern

mysql -uUSER -pPASSWORD -D DATABASE_NAME -e "show tables" -s | egrep "SEARCH_PATTERN" | xargs -I "@@" mysql -uUSER -pPASSWORD -D DATABASE_NAME -e "DROP TABLE @@;"

Notes: Replace the markers where indicated (USER, PASSWORD etc) and obviously change the example query.

Command line screencast

mkfifo /tmp/fifo;(nc -q0 -k -l -p 5000 < /tmp/fifo > /dev/null &);script -f /tmp/fifo

Notes: Run nc ADDRESS 5000 to connect and watch.

Empty all log files

for file in $(find /var/log -type f); do > $file; done;

Notes: None.

Watch MySQL queries

watch -n 1 mysqladmin --user=USER --password=PASSWORD processlist

Notes: Replace the markers where indicated (USER, PASSWORD etc).

Find potential duplicate files

find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate

Notes: None.

1. Create a file of the size you wish the filesystem to be, 1GB in this example:

   dd if=/dev/zero of=fat32.disk bs=1K count=1M

2. Format the filesystem, fat32 in this example but you can use NTFS or another case-insensitive file system:

   mkfs.vfat fat32.disk

3. Mount the filesystem and test it, you may wish to use uid= and gid= to mount it as a specific user and you’ll probably want to adjust the umask to something less permissive:

   mount /media/fat32.disk /var/www/fat32 -t vfat -o loop,owner,group,umask=000

4. Add it to /etc/fstab so that it gets mounted after a reboot:

   /media/fat32.disk /var/www/fat32 vfat loop,owner,group,umask=000 0 0

I wouldn’t use this for production systems, in that situation I’d recommend a dedicated FAT32 or NTFS partition. If you do have to run this on a production system it should be fine for 99% of setups, it just feels like a quick hack to me.

A few days later the user calls and wants to be able to print in safe mode. I look into it, do some searching, but the prevailing wisdom seemed to be that it wasn’t doable. This sounded like an MCP party line to me so I decide to explore the registry. Eventually I find the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control key which has sub-keys of Minimal and Network. Minimal being safe mode, Network being safe mode with networking. It seems to be a whitelist of services, drivers and drive groups that are allowed to start or load.

Therefore it is possible to start additional services and load additional drivers in safe mode – just add a key for the service or driver short name, then a string for type. The below entry (if in a .reg file) would allow the Print Spooler to start in safe mode with networking.

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SafeBoot\Network\Spooler]
@="Service"

If you want a list of all drivers, driver groups and services starting in normal mode and their corresponding short names check HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services.

I’d caution against whitelisting too much as it kind of defeats the purpose of safe mode, though in certain situations as a quick hack it can useful. It may also be something worth checking the next time you’re dealing with a particularly nasty malware infection. I haven’t seen anything which exploits it yet, but I imagine something does.

I had just installed the SnipMate plugin which mimics the snippet functionality of TextMate, and every time I restarted I got an error stating:

No mapping found

I did some troubleshooting and narrowed it down to the cpoptions=ces$ line in my .vimrc. This line makes the cw command and similar commands put a $ at the end instead of just deleting text and replacing it. I could see no reason for me to need this and just removed the line. That setting then defaulted back to cpoptions=aABceFs and the error was gone.

The series starts with theory and need, then a few posts on local implementation, a couple of scenarios for wider implementation, then a final article on the reticence of the IT world to IPv6 and what I think is causing it.

So starting with theory and the need for IPv6, I know they’re not glamorous, I know to some they may be mind numbing and they’d rather just get on and do it, the whole don’t talk or discuss just do mentality. I can sympathise with this opinion. That being said, sorry no dice, you need a good mix of practical and theory otherwise you get into the area of call cargo cult system administration – I’ll probably discuss that phrase more in a future post – where you follow something like a ritual with no actual understanding.

Why do we need IPv6?

The Internet currently runs on IPv4 whose addresses are 32 bits long, with an address space of 2³² resulting in 4,294,967,296 or about 4.3 billion addresses. This was considered a suitably large number when IPv4 was created, but its running out, estimates vary from the conservative one year to the liberal five years – which assumes a lot of tinkering by the IANA, such tinkering would be prohibitively expensive in places and by no means certain – It doesn’t really matter which one is right, in the long run IPv4 is running out of addresses.

IPv6 addresses are 128 bits long with an address space of 2¹²⁸ resulting in 340,282,366,920,938,463,463,374,607,431,768,211,456 – having trouble saying that, well it’s 340-undecillion, 282-decillion, 366-nonillion, 920-octillion, 938-septillion, 463-sextillion, 463-quintillion, 374-quadrillion, 607-trillion, 431-billion, 768-million, 211-thousand, 456 or about 340 trillion, trillion, trillion addresses. This is currently considered a suitably large number with a high degree of future proofing. To give it some scale:

• Its enough addresses for many trillions of them to be assigned to every person on the planet. Assuming the earth has 6.5 billion people, each person could be assigned 2⁹⁵ or 3,961,481,257,132,168,796,771,975,168 addresses
• The Earth is about 4.5 billion years old. If we had been assigning IPv6 addresses at a rate of 1 billion per second since then, we would have by now only used up less than one trillionth of the address space.

For a more extensive and always up-to-date report on the exhaustion I’d recommend Potaroo (it also has a nice countdown widget).

You’re wrong, NAT and CIDR will save me?

No they won’t. The IT community saw this exhaustion coming in the early nineties, at the time they created NAT and CIDR to delay it while they worked on IPv6. NAT and CIDR have bought us about twenty years, we can’t really expect much more from them. I realise some of the imaginary people reading this may be insulted by my assumption in asking this question, I make it because one fairly senior internal IT guy from Abertay (my former university) who talked to me during the end of year open day, tried to lecture me on how my project was crap because of NAT, after face-palming I proceeded to rip his argument apart. Since then though a friend from Dundee College IT support has conveyed similar opinions. I’m guessing that’s the preferred opinion of the alpha-geek in the area. I know it’s not a JANET member opinion, so I’m guessing it’s just Dundee College and Abertay University or perhaps the prevailing opinion of FaTMAN (Fife and Tayside Metropolitan Area Network).

The problem, the reason for this opinion is a lot of educational establishments still use public routable IPv4 addresses for clients, so they think they can just switch to NAT, sell on the addresses and that’ll be that. They fail to realise the scale of exhaustion, lets say they moved to NAT and it freed about two to ten thousand addresses per campus, that’s not going to make much of a difference; its negligible with current demand and that assumes no growth, also renumbering those addresses may not be practical or cost effective. I’ll probably cover this aversion to IPv6 in more details for the last article.

Wait what about IPv5?

There was an IPv5, also called the Internet Stream Protocol or ST. It was created in the late 1970s, IPv5 was designed to deal with streaming media – sending video, audio, and simulations over the Internet – It used IPv4 addressing, its main advantage over IPv4 UDP for media was that it offered connections and guaranteed QoS. For a while it gained a small following in places like IBM, Apple and Sun, they even revised it into ST+/ST2/ST2+. IPv5 was never accepted as a standard, the work done on it was not lost however, much of its concepts were implemented in MPLS. And That is why the next generation internet protocol is called IPv6 and not IPv5.

IPv6 addresses are too complicated?

IPv6 addresses are too complicated, the average user has trouble with IPv4, I mean 2001:41c8:0001:5a19:0000:0000:0000:0002 who’s going to remember that.

When I was first made aware of IPv6 I thought this also, since then after looking at it more closely, I’ve changed my mind. Lets explain the address hierarchy then break down that example address.

All IPv6 addresses consist of 8, 16 bit HEX blocks, separated by colons – this is known
as colon hexadecimal notation – as illustrated in the above example. IPv6 uses an address hierarchy – which incidentally also helps with routing – that looks like this:

 Internet backbone -> ISP -> Organisation (building or individual connection) -> Subnets -> Hosts

32 bits are assigned to each ISP from the Internet backbone and from this the ISP allocates 48 bit addresses to organisations. 48 bits are assigned to the organisation – combining the 32 bits from the ISP plus 16 bits for the organisation – and are globally unique, this is not assigned to the organisations entire global presence but rather to each site; to each external connection to the internet, so each will receive its own /48 address. 16 bits can then be used for organisational subnetting this gives us 65535 potential subnets. Leaving 64 bits to define unique hosts per subnet, giving us a potential 18,446,744,073,709,551,616 or eighteen quintillion, four-hundred-forty-six quadrillion, seven-hundred-forty-four trillion, seventy-three billion, seven-hundred-nine million, five-hundred-fifty-one thousand, six hundred and sixteen per each of the 65535 subnets. That last portion, which is just being assigned to hosts is twice the bit length of the entire IPv4 address pool. So a 128 bit IPv6 address can be viewed as two 64 bit addresses. The 64 bit address to the left defines the globally unique prefix or the 48 bits assigned by ISP and the 16 bits used for subnetting. The remaining 64 bits to the right are assigned to the host interface on an appropriate subnet.

Taking this back to the 2001:41c8:0001:5a19:0000:0000:0000:0002 example:

• 2001:41c8 = 32-bits, which denotes the ISP.
• 2001:41c8:0001 = 48 bits (ISP + 16 bits) which are globally unique and assigned by the ISP from its /32 address.
• 5a19 = 16 bits to be used for subnetting purposes.
• 0000:0000:0000:0002 = 64 bits to be used to identify the host or network interface.

There are perhaps some security or privacy concerns about hierarchical addressing but I’ll discuss them later.

Moving on to address complexity, in IPv6 addresses contiguous blocks of 16 bit zeros can be replaced with :: only one set of :: may be used per IPv6 address to avoid expansion confusion, leading zeros may be dropped. Notation may combine :: and dropped leading zeros to form a compressed IPv6 address, the following are all the same address and all considered valid:

• 2001:41c8:0001:5a19:0000:0000:0000:0002
• 2001:41c8:0001:5a19:0:0:0:2
• 2001:41c8:1:5a19::2

I’ll admit the addresses could still get complicated in certain situations, I think we need to move towards properly implemented DNS to solve this problem, but I have an unhealthy love and fascination for DNS so I would say that.

So what are the features of IPv6

IPv6 has a lot of features and improvements over IPv4 and while I won’t enumerate them all here, a few notable examples are listed below.

IPv6 has a new header format which is fixed length, this improves performance, lessens fragmentation and when combined with the use of link-local addresses offloads a lot of work from the router and networking layer to the protocol and transport layer.

IPv6 allows two types of configuration:

Stateless - this is like a better version of the current APIPA 169.254.x.x addresses. With this method the host configures itself via multicast without a DHCPv6 server and/or router.

Stateful - which is where the host configures itself with information provided from DHCPv6 server and/or router.

True automatic configuration is now possible, a link-local address is auto-configured for each IPv6 connected host. Each host will perform router solicitation via ICMPv6 router discovery messages, that is, it will search its default subnet for a router, which it will communicate with in order to gain further insight into the network and how it should configure itself. In addition IPv6 hosts can perform DHCPv6 solicitation, though DHCP is all but obviated in IPv6 networks not requiring advanced DHCP options. Router solicitation is preferable and has priority, both may be used in conjunction if certain DHCP specific configuration options are required. Generally routers pass along prefixes to be autoconfigured on IPv6 hosts and DHCPv6 servers pass along prefixes and/or additional settings, such as where to find NTP or WINS servers.

IPv6 supports three types of addresses:

A unicast address - unique address for a device (host/router/phone/etc).

A multicast address - groups common systems (routers/hosts/etc).

An anycast address - represents the closest address, used heavily in multihomed environments, so if a server had three interface cards on different subnets and was providing the same service to each subnet an anycast address would represent the closest address to your subnet to contact that service. These address types lower overhead and increase network performance.

IPv6 hosts uses Neighbour Discovery (ND) over multicast to find hosts, ND replaces ARP and other broadcast-based techniques and lowers network overhead.

IPv6 obviates the need for NAT because all hosts/devices, including mobile devices are assigned a Globally Unique ID (GUID) based on the prefix assigned to your service provider. Devices are still secure with the use of a firewall or properly configured network, they just now have the potential of being public routable.

The minimum MTU is 1280-bytes compared to 576-bytes for IPv4, this means larger packets can be sent assuming bandwidth is available. IPv6 supports JumboGrams – packets larger than 64k; each IPv4 packet could only have a payload of up to 64 KB, while each IPv6 packet can have a payload of up to 4GB. While such large packets are unlikely to be a regular occurrence they have uses in high-MTU networks, such as upstream or back-end providers

Conclusion

Okay we’re done here, if you have any questions on the theory or need, post a comment. Beyond that be on the look out for part 2.

Scenario: You’re a system administrator/network administrator/IT guy (whatever you want to call it) with an excellent setup, and while you sit staring at the blinking lights in the comms room pondering the quintessential meaning of things, or more realistically chatting on IRC (whatever floats your boat) you’re interrupted for the fifteenth time that week by that luser, um user^[1] you were forced to give local admin access to. Turns out now they can’t access the intranet or send e-mail.

Upon investigating you find that once again this user has changed the DNS settings on their computer, breaking Active Directory/OpenLDAP/e-mail whatever, despite repeated warnings. They’re operating under the mistaken belief that using the DNS servers provided by OpenDNS, Google DNS or any number of resolvers found here. Will make their Interweb downloads of funny cat pictures faster, you’ve tried chatting and explaining it to the guy, you tried approaching their line manager with no success. Short of beating the user with a hammer you need to find a way to resolve this situation, what do you do? You could block external DNS but that’s only half an answer. If they do it again it will break more connectivity.

Solution: What you need to do is intercept and redirect all DNS requests to a local server, so no matter what the user configures, no matter what dig or nslookup tells them they’re using, your local DNS will handle the queries. If you’re using something based on IPTables then the follow commands should do it, assuming a basic 192.168.1.0/24 network:

iptables -t nat -A PREROUTING -p udp -i br0 --dport 53 -j DNAT --to 192.168.1.1
iptables -t nat -A PREROUTING -p tcp -i br0 --dport 53 -j DNAT --to 192.168.1.1
iptables -t nat -I PREROUTING -p udp -s 192.168.1.0/24 -d ! 192.168.1.0/24 --dport 53 -j DNAT --to 192.168.1.1
iptables -t nat -I PREROUTING -p tcp -s 192.168.1.0/24 -d ! 192.168.1.0/24 --dport 53 -j DNAT --to 192.168.1.1

Of course this depends on the nuances of your setup. Both sets of commands do the same thing but in slightly different ways, I tend to prefer the former set, but again it depends on your configuration. To test simply set a client to use external DNS and try to resolve an internal host, or you could even set a client to use an IP address you know runs no DNS server if you want to be extra sure it’s working. If you happen to use a Cisco device, I’m sure this can be done on them. I haven’t got my lab setup to check the syntax but I believe a transparent proxy or DNS doctoring will handle it depending on the device.

There you have it folks some words of wisdom from your uncle Kris, try not abuse them and say redirect every third to ninth DNS request to kittenwar.com. My final word on this is: Suck in the guts, guys, we’re the Ghostbusters.

^Back – [1] – You know the guy I’m talking about. Most companies have at least one. The guy with just enough knowlege to be dangerous. He’s the wannabe, fancies himself in your job, constantly bad mouthing you, boasting about his audio-visual setup or how much he has overclocked something.