Hadoop in Healthcare, Part 1

This is the first blog in a series written by Richard Proctor, GM of Global Healthcare at Hortonworks, Inc. The series will discuss the reasons for Healthcare’s surging interest in, and rapid adoption of, Hadoop.

We’ve all heard a lot lately about Hadoop, especially within the Healthcare industry. Well, this is really important stuff! Healthcare represents the next frontier for gaining big insights from Big Data and driving innovation around:

• Remote patient monitoring
• Telehealth
• Insights into patient behavior
• Managing chronic diseases
• Improving patient experience
• Reducing the per capita cost of healthcare

Some forward-thinking healthcare providers are capturing this Big Data opportunity with Hadoop. They’re ingesting more structured and unstructured data into a data lake where caregivers and researchers can analyze the combined dataset in myriad ways at once.

These pioneers have been able to build new applications that use Big Data to improve operational efficiency and enhance patient outcomes while staying ahead of rapidly evolving compliance, regulation, and industry changes.

Data Architecture Issues in Healthcare

Healthcare of the past was plagued by data infrastructures incapable of handling the volume, velocity, and variety of data needed to derive deep clinical, financial, and operational insights of the industry.

Traditionally, data has been the result of independent business processes, which invariably led to data silos. This delayed critical patient data and forced it to be reactive, if spotted and reported at all. Meaningful data would sit in an overnight batch queue waiting to be loaded into the enterprise data warehouse (EDW) where key analytical applications could glean insights.

Healthcare entities continue to be heavily focused on regulatory and compliance reporting, yet they’ve also begun to recognize the value of the data they’re generating. Unfortunately their data silos continue to be costly and limited in terms of storage, forcing them to face hard choices about what data to retain.

The Push for Hadoop

With a modern data architecture powered by Hadoop, Healthcare organizations are seeing data as a byproduct of patient care and are looking to near real-time data interactions and insights. Hospitals such as Mercy see it as a “leap forward” and are now managing 50 terabytes or more of data. Analytics teams are now looking to analyze new data sources such as:

• Sentiment data
• Log files
• Social media
• Streaming data from sensors and wearables
• Machine-generated data
• Clickstream
• Geo-location
• Weather data

The mantra of store everything is now both financially and technically possible as Hadoop as enabled Healthcare organizations to store data in its native format, whether structured, semi-structured, or completely unstructured. Hadoop has provided inexpensive storage with a vast array of options like On-Premise, Cloud, and Hybrid deployments.

“So your IT department shouldn’t be on the fence about Hadoop because it’s a given. It’s a done deal. It’s going to happen. Hadoop has so much momentum at the moment it’s hard to see an alternate data management infrastructure emerging in the foreseeable future.  Almost anyone that wants to manage massive amounts of unstructured (or semi-structured) data will have Hadoop”.

— Steve Banker, Forbes, August 2014

The Advantage and Savings of Open-Source

It should go without saying, but open-source software helps keep costs down. That’s why Hortonworks Data Platform (HDP) is free! HDP has full life-cycle support and is backed by the largest and most experienced Hadoop team on the planet.

Open-source software is also an upgrade in quality. A vast community of coders, contributors, reviewers, and testers helps ensure Hadoop remains the master of Big Data.

Hortonworks also employs more committers to the Hadoop project than any other vendor. This helps ensure agility and the ability to react faster to the needs of the marketplace.

Hortonworks leads by example and contributes more code to open-source than any other vendor. There is no proprietary software here. This mitigates business risk and liberates customers from vendor lock-in.

For more information on Hortonworks and our 175+ member strong Healthcare Hadoop User group (HUG), please reach out to Richard Proctor.