UC Irvine Medical Center maintains millions of records for more than a million patients, including radiology images and other semi-structured reports, unstructured physicians’ notes, plus volumes of spreadsheet data. To solve the challenge the hospital faced with data storage, integration, and accessibility, the hospital created a data lake based on a Hadoop architecture, which enables distributed big data processing by using broadly accepted open software standards and massively parallel commodity hardware. Hadoop allows the hospital’s disparate records to be stored in their native formats for later parsing, rather than forcing all-or-nothing integration up front as in a data warehousing scenario. Preserving the native format also h...

by Jessica Davis Streaming and batch big data analytics technology Apache Apex has been elevated to a Top-Level Project by the Apache Software Foundation. Used by organizations including Capital One and GE, the technology can help developers more quickly create apps that leverage real-time data. The rise of interest in Apache Spark has demonstrated just how important streaming data has become in the big data ecosystem. Real-time data and the technologies that support it were perhaps the biggest stars of last month's Strata + Hadoop Worldconference in San Jose. So it's prob...