Data Lakes: 8 Enterprise Data Management Requirements
|
 |
|
Ingest and Delivery
Data lakes need mechanisms for getting data into and out of the backend storage platform. In traditional data warehouses, data is inserted and queried using some form of SQL and a database driver, possibly via ODBC or JDBC. While compatibility drivers do exist to access Hadoop data, the variety of data formats requires more flexible tooling to accommodate the different formats. Open source tools such as Sqoop and Flume provide low-level interfaces for pulling in data from relational databases and log data, respectively. In addition, custom MapReduce programs and scripts are currently used to import data from APIs and other data sources. Commercial tools provide pre-built connectors and a wealth of data format support to mix and match data sources to data repositories in the data lake.
Given the variety of data formats for Hadoop data, a comprehensive schema management tool does not yet exist. Hive's metastore extended via HCatalog provides a relational schema manager for Hadoop data. Yet, not all data formats can be described in HCatalog. To date, quite a bit of Hadoop data is defined inside applications themselves, perhaps using JSON, AVRO, RCFile or Parquet. Just like with data endpoints and data formats, the right commercial tools can help describe the lake data and surface the schemas to the end users more readily.
2016 is the year of the data lake. It will surround and, in some cases, drown the data warehouse, and we'll see significant technology innovations, methodologies and reference architectures that turn the promise of broader data access and Big Data insights into a reality. But Big Data solutions must mature and go beyond the role of being primarily developer tools for highly skilled programmers. The enterprise data lake will allow organizations to track, manage and leverage data they've never had access to in the past. New data management strategies are already leading to more predictive and prescriptive analytics that are driving improved customer-service experiences, cost savings and an overall competitive advantage when there is the right alignment with key business initiatives.
So whether your enterprise data warehouse is on life support or moving into maintenance mode, it will most likely continue to do what it's good at for the time being: operational and historical reporting and analysis (a.k.a. rear-view mirror).
As you consider adopting an enterprise data lake strategy to manage more dynamic, poly-structured data, your data integration strategy must also evolve to handle the new requirements. Thinking that you can simply hire more developers to write code or rely on your legacy rows-and-columns-centric tools is a recipe to sink in a data swamp instead of swimming in a data lake. In this slideshow, Craig Stewart, VP product management at SnapLogic, has identified eight enterprise data management requirements that must be addressed in order to get maximum value from your Big Data technology investments.
Related Topics : APC, Resellers, Data Replication, Extract Transform and Load, Structured Data Integration
How to Find Business Value in Your Data Through Modernization
Data only becomes a meaningful and valuable asset when organizations can transform it into actionable insights. ... More >>
5 Steps to Wrangle Uncontrolled Data Flow
As the availability of data exponentially increases, unprecedented opportunities exist to do all kinds of amazing things, but these opportunities also come with data wrangling challenges. ... More >>
5 Data Warehouse Design Mistakes to Avoid
If you are designing a data warehouse, you need to map out all the areas where there is a potential for your project to fail, before you begin. ... More >>