Linux blamed for leap second embarrassment that humbled Internet

The airline disruption was probably the most inconvenient problem caused by the addition of a second to atomic clocks, necessary to account for a miniscule accumulated slowing of the earth’s rotation of about 1.4 milliseconds per year.

Australian airline Qantas reported two-hour delays as Saturday turned into Sunday UTC (noon, Sydney time), which echoed wider and diffuse Internet problems that temporarily downed a clutch of well-known websites and services including Reddit, LinedkIn, FourSquare, Stumbleupon and Cisco videoconferencing.

“This incident was caused by the Linux bug triggered by the ‘leap second’ inserted into clocks worldwide on June 30th,” read an Amadeus statement on the problem that beautifully understates a complex if predictable software issue affecting some Linux-based programs while leaving others untouched.

Which programs were left exposed? Anything with a Java component plus MySQL, Firefox, Thunderbird, and Debian, the latter causing server blades to ‘go dark’.

It shouldn’t have happened as anyone who remembers the hugely expensive damp squib of the Y2K bug at the turn of the Millenium will be reminding the engineering fraternity. In some cases it appears that some techies only guessed that the problem was related to the leap second because it happened to occur precisely at the moment of midnight UTC/GMT.

Although leap seconds caused by the need to compensate for the earth’s rotation are extremely rare occurrences – the last whole second adjustment would have happened in 1820 had atomic clocks and NTP servers existed – there have in fact been 25 leap seconds for other reasons since the beginning of atomically-measured time in 1971.

Of course, Linux’s weakness was its sheer diversity rather than an inherent issue with the open source model of collaboration itself. Linux is a hugely important foundation of Internet services without that fact being obvious.

The preferred time adjustment technique is to add tiny increments of time gradually, which allows systems to add these logically when they add up to a whole second. Such an approach is already used by Google’s NTP servers.

It now appears that not everyone got the memo.