Even today, all it takes is a single second to crash popular websites. As the news broke this morning about an additional second being added to the granddaddy of all clocks (Coordinated Universal Time) last night, websites were going dark and internet dependent software was crashing.
The time adjustment, made by the International Earth Rotation and Reference Systems Service, was made to ensure atomic clocks worldwide would keep pace with the Earth’s every changing rotation speed. Thirty six seconds have been added to atomic clocks since 1972.
Sites affected included Reddit, which tweeted it’s problem early this morning. They laid the blame on their open source database, which is built with Java.
reddit status@redditstatus “We are having some Java/Cassandra issues related to the leap second at 5pm PST. We're working as quickly as we can to restore service.”
Also affected was Mozilla, the company behind the popular Firefox web browser. Mozilla blamed Hadoop, another open source platform based on Java. A blog post by site engineer Eric Ziegenhorn stated “Our system is choking on the leap second”.
Ziegenhorn also said it appeared Java was the culprit. "Servers running java apps such as Hadoop and ElasticSearch and java doesn't appear to be working. We believe this is related to the leap second happening tonight because it happened at midnight GMT.”
Gawker media sites confirmed that it experienced issues related to the bug as well on a blog post:
"We were not 100 percent offline, but the service was very unpredictable for about 30 minutes last night."
It appears StumbleUpon, Yelp, FourSquare, and LinkedIn were also affected by the leap second, according to a BuzzFeed report released this afternoon.
The idea that adjustments to time could cause issues for websites and software is not new. In a blog post in June, Marco Marongiu, an engineer at Opera Software, reported how web systems might fail by the addition of an extra second. He also described a workaround to help avoid the type of crashes that occurred today.
“What’s really frustrating is that all of this downtime could have been avoided,” said Jeremy Marken, a systems analyst based in Los Angeles. “It’s all completely predictable.”
In another blog post last year, Google explained that it gradually adds milliseconds to its systems prior to the addition of a leap second. In doing so, it “softens the blow” and allows engineers to implement fixes without downtime.
"This meant that when it became time to add an extra second at midnight, our clocks had already taken this into account, by skewing the time over the course of the day," Google said. "All of our servers were then able to continue as normal with the new year, blissfully unaware that a leap second had just occurred."