Many tech industry observers believe that we are in the midst of a true industrial revolution spawned by the astonishing growth of the Internet. And one of the outcomes from that growth is a very significant, but not as widely discussed, change in how the quantum leap in processing huge amounts of data is transforming the technology to actually make it all work.
This was the basic premise behind the NoSQL Conference held over the past two days in San Jose, California. It’s not exactly the most catchy name you’ll find in the tech world. NoSQL stands for “not only” SQL, with SQL being “structured query language,” the programming designed to manage data.
SQL has been used to manage large databases of information going back to the 1970s, back when tweets were bird calls and people actually took pictures with something called film. The changes in the world since then have forced some of the brightest minds in the tech industry to find new and creative ways to manage the “industrial revolution of data,” as famously coined by O’Reilly Media in 2008. Enter NoSQL.
Produced by Dataversity, the NoSQL Conference started out with 500 attendees and has expanded to over 1,000 attendees in just two years. In recognition of the growing urgency to change the model for managing huge amounts of data, one of the most visible participants at the conference this year is Oracle. Andrew Mendelsohn, Oracle’s Senior Vice President, announced in his keynote speech on Wednesday that his company would now offer a NoSQL database that can be downloaded for free (with no support) and what he called the “first NoSQL hardware.”
But this conference was not so much about making news as it was about education. The operative word these days is confusion. The NoSQL solution has given rise to so many new databases, that data managers are increasingly more baffled over which way to turn.
Imagine the hallways of a convention center jammed with people doing an excellent imitation of deer in the headlights. Complex technology has led to REALLY complex technology and two days of sessions with mind-bending titles like “Modeling for User Profile and Interest Graph, Ontology Relationships and Real Time Analytics.” Said Dan McCreary of Kelly-McCreary & Associates, “We started this conference out of frustration. Education is really one of our prime objectives here.”
The increasing number of companies who are building products in the NoSQL space such as 10gen, Basho, FoundationDB, and MarkLogic, are confronting a curious problem. They have customers who really need their help, but they also have to overcome enormous resistance to get there. “The same tools that were used to manage payroll for 2000 employees are being used today to manage billions of social media transactions,” said Max Schireson of 10gen. “We need to understand and communicate what our products can do better than anything else out there.”
Schireson cited the example of MetLife, one of the world’s largest providers of insurance and employee benefits programs. At one point, MetLife was using more than 70 different systems to contain their policy information and spent many fruitless years trying to put everything on one system. They finally approached 10gen (maker of mongoDB) and Schireson’s firm had all of the company systems consolidated into one in less than 90 days.
Stories like this explain why some of the tech world’s largest companies are starting to move cautiously towards noSQL database management. Google, Amazon, Netflix, Shutterfly and eBay have all put their corporate toes in the NoSQL waters.
After two days of sessions in San Jose, there’s a clear feeling among data managers that the continued emphasis on customer education over short-term profits may ultimately yield big results. As one speaker summed it up using a well-known quote from Gandhi, “First they ignore you, then they ridicule you, then they fight you, and then you win.”