|By Bill McColl||
|October 18, 2010 08:15 AM EDT||
Over the past few years, Hadoop has become something of a poster child for the NoSQL movement. Whether it's interpreted as "No SQL" or "Not Only SQL", the message has been clear, if you have big data challenges, then your programming tool of choice should be Hadoop. Sure, continue to use SQL for your ancient legacy stuff, but when you need cutting edge performance and scalability, it's time to go Hadoop.
The only problem with this story is that the people who really do have cutting edge performance and scalability requirements today have already moved on from the Hadoop model. A few have moved back to SQL, but the much more significant trend is that, having come to realize the capabilities and limitations of MapReduce and Hadoop, a whole raft of new post-Hadoop architectures are now being developed that are, in most cases, orders of magnitude faster at scale than Hadoop.
The problem with simple batch processing tools like MapReduce and Hadoop is that they are just not powerful enough in any one of the dimensions of the big data space that really matters. If you need complex joins or ACID requirements, SQL beats Hadoop easily. If you have realtime requirements, Cloudscale beats Hadoop by three or four orders of magnitude. If you have supercomputing requirements, MPI or BSP beat Hadoop easily. If you have graph computing requirements, Google's Pregel beats Hadoop by orders of magnitude. If you need interactive analysis of web-scale data sets, then Google's Dremel architecture beats Hadoop by orders of magnitude. If you need to incrementally update the analytics on a massive data set continuously, as Google now have to do on their index of the web, then an architecture like Percolator beats Hadoop easily.
The one area where MapReduce/Hadoop wins today is that it's freely available to anyone, but for those that have reasonably challenging big data requirements, that simple type of architecture is nowhere near enough.
- The Top 250 Players in the Cloud Computing Ecosystem
- Cloud People: A Who's Who of Cloud Computing
- Ulitzer Names the World's 30 Most Influential Cloud Computing Bloggers
- 4th International Cloud Expo: Photo Album
- Cloud Expo New York to Attract More Than 8,000 Delegates
- The Cloud Computing Kettle Heats Right Up
- Industry Experts Discuss the State of Cloud Computing
- 4th International Cloud Computing Conference & Expo Starts Today
- Cloud Expo and The End of Tech Recession
- SYS-CON.TV: Cloud Computing Expo Power Panel