Data lake

From HandWiki
Short description: System or repository of data stored in its natural/raw format
Example of a database that can be used by a data lake (in this case structured data)

A data lake is a system or repository of data stored in its natural/raw format,[1] usually object blobs or files. A data lake is usually a single store of data including raw copies of source system data, sensor data, social data etc.,[2] and transformed data used for tasks such as reporting, visualization, advanced analytics and machine learning. A data lake can include structured data from relational databases (rows and columns), semi-structured data (CSV, logs, XML, JSON), unstructured data (emails, documents, PDFs) and binary data (images, audio, video).[3] A data lake can be established "on premises" (within an organization's data centers) or "in the cloud" (using cloud services from vendors such as Amazon, Microsoft, Oracle Cloud, or Google).

Background

James Dixon, then chief technology officer at Pentaho, coined the term by 2011[4] to contrast it with data mart, which is a smaller repository of interesting attributes derived from raw data.[5] In promoting data lakes, he argued that data marts have several inherent problems, such as information siloing. PricewaterhouseCoopers (PwC) said that data lakes could "put an end to data silos".[6] In their study on data lakes they noted that enterprises were "starting to extract and place data for analytics into a single, Hadoop-based repository."

Examples

Many companies use cloud storage services such as Google Cloud Storage and Amazon S3 or a distributed file system such as Apache Hadoop distributed file system (HDFS).[7] There is a gradual academic interest in the concept of data lakes. For example, Personal DataLake at Cardiff University is a new type of data lake which aims at managing big data of individual users by providing a single point of collecting, organizing, and sharing personal data.[8]

An earlier data lake (Hadoop 1.0) had limited capabilities with its batch-oriented processing (Map Reduce) and was the only processing paradigm associated with it. Interacting with the data lake meant one had to have expertise in Java with map reduce and higher-level tools like Apache Pig, Apache Spark and Apache Hive (which by themselves were originally batch-oriented).

Criticism

Poorly-managed data lakes have been facetiously called data swamps.[9]

In June 2015, David Needle characterized "so-called data lakes" as "one of the more controversial ways to manage big data".[10] PwC was also careful to note in their research that not all data lake initiatives are successful. They quote Sean Martin, CTO of Cambridge Semantics:

They describe companies that build successful data lakes as gradually maturing their lake as they figure out which data and metadata are important to the organization.

Another criticism is that the term "data lake" is not useful because it is used in so many different ways. [11] It may be used to refer to, for example: any tools or data management practices that are not data warehouses; a particular technology for implementation; a raw data reservoir; a hub for ETL offload; or a central hub for self-service analytics.

While critiques of data lakes are warranted, in many cases they apply to other data projects as well.[12] For example, the definition of “data warehouse” is also changeable, and not all data warehouse efforts have been successful. In response to various critiques, McKinsey noted[13] that the data lake should be viewed as a service model for delivering business value within the enterprise, not a technology outcome.

Extensions

Data lakehouse is a proposed hybrid approach of a data lake and a data warehouse,[14] and attempts to solve some of the challenges with data lakes.[15][clarification needed] It has been described as starting with a "data lake architecture [and attempting] to add data warehouse capabilities to it".[16] According to Oracle, it combines the "flexible storage of unstructured data from a data lake and the management features and tools from data warehouses".[17]

See also

References

  1. ↑ "The growing importance of big data quality". 21 November 2016. https://blogs.sas.com/content/datamanagement/2016/11/21/growing-import-big-data-quality/. Retrieved 1 June 2020. 
  2. ↑ "What is a data lake?". https://aws.amazon.com/big-data/datalakes-and-analytics/what-is-a-data-lake/. Retrieved 12 October 2020. 
  3. ↑ Campbell, Chris. "Top Five Differences between DataWarehouses and Data Lakes". https://www.blue-granite.com/blog/bid/402596/top-five-differences-between-data-lakes-and-data-warehouses. 
  4. ↑ Woods, Dan (21 July 2011). "Big data requires a big architecture". Forbes. https://www.forbes.com/sites/ciocentral/2011/07/21/big-data-requires-a-big-new-architecture/. 
  5. ↑ Dixon, James (14 October 2010). "Pentaho, Hadoop, and Data Lakes". James Dixon. https://jamesdixon.wordpress.com/2010/10/14/pentaho-hadoop-and-data-lakes/. Retrieved 7 November 2015. "If you think of a datamart as a store of bottled water – cleansed and packaged and structured for easy consumption – the data lake is a large body of water in a more natural state. The contents of the data lake stream in from a source to fill the lake, and various users of the lake can come to examine, dive in, or take samples." 
  6. ↑ 6.0 6.1 Stein, Brian; Morrison, Alan (2014). Data lakes and the promise of unsiloed data (Report). Technology Forecast: Rethinking integration. PricewaterhouseCoopers. http://www.pwc.com/en_US/us/technology-forecast/2014/cloud-computing/assets/pdf/pwc-technology-forecast-data-lakes.pdf. 
  7. ↑ Tuulos, Ville (22 September 2015). "Petabyte-Scale Data Pipelines with Docker, Luigi and Elastic Spot Instances". http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-docker.html. 
  8. ↑ Walker, Coral; Alrehamy, Hassan (2015). "Personal Data Lake with Data Gravity Pull". 2015 IEEE Fifth International Conference on Big Data and Cloud Computing. pp. 160–167. doi:10.1109/BDCloud.2015.62. ISBN 978-1-4673-7183-4. 
  9. ↑ Olavsrud, Thor (June 8, 2017). "3 keys to keep your data lake from becoming a data swamp" (in en). CIO. https://www.cio.com/article/230163/3-keys-to-keep-your-data-lake-from-becoming-a-data-swamp.html. 
  10. ↑ Needle, David (10 June 2015). "Hadoop Summit: Wrangling Big Data Requires Novel Tools, Techniques". eWeek. http://www.eweek.com/enterprise-apps/hadoop-summit-wrangling-big-data-requires-novel-tools-techniques-2.html. "Walter Maguire, chief field technologist at HP's Big Data Business Unit, discussed one of the more controversial ways to manage big data, so-called data lakes." [yes|permanent dead link|dead link}}]
  11. ↑ "Are Data Lakes Fake News?". 2017-08-08. https://sonra.io/2017/08/08/are-data-lakes-fake-news/. 
  12. ↑ Belov, Vladimir; Kosenkov, Alexander N.; Nikulchev, Evgeny (2021). "Experimental Characteristics Study of Data Storage Formats for Data Marts Development within Data Lakes". Applied Sciences 11 (18): 8651. doi:10.3390/app11188651. 
  13. ↑ "A smarter way to jump into data lakes". 2017-08-01. https://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/a-smarter-way-to-jump-into-data-lakes. 
  14. ↑ What is a Data Lakehouse?
  15. ↑ What Is A Data Lakehouse? A Super-Simple Explanation For Anyone
  16. ↑ What is a Data Lakehouse? | Snowflake
  17. ↑ What is a Data Lakehouse? | Oracle



Retrieved from "https://handwiki.org/wiki/index.php?title=Data_lake&oldid=3433459"