Data version control

From HandWiki
Short description: Software designed for managing workflows involving analysis of large data sets


Data version control is a method of working with data sets. It is similar to the version control systems used in traditional software development, but is optimized to allow better processing of data and collaboration in the context of data analytics, research, and any other form of data analysis. Data version control may also include specific features and configurations designed to facilitate work with large data sets and data lakes.[1]

History

Background

As early as 1985, researchers recognized the need for defining timing attributes in database tables, which would be necessary for tracking changes to databases.[2] This research continued into the 1990s, and the theory was formalized into practical methods for managing data in relational databases,[3] providing some of the foundational concepts for what would later become data version control.

In the early 2010s the size of data sets was rapidly expanding, and relational databases were no longer sufficient to manage the amounts of data organizations were accumulating. The rise of the Apache Hadoop eco system, with HDFS as a storage layer, and later object storage had become dominant in big data operations.[4] Research into data management tools and data version control systems increased sharply, along with demand for such tools from both academia and the private and public sectors.[5]

Version controlled databases

The first versioned database was proposed in 2012 for the SciDB database, and demonstrated it was possible to create chains and trees of different versions of the database while decreasing both the overall storage size and access speeds associated with previous methods.[6] In 2014, a proposal was made to generalize these principles into a platform that could be used for any application.[7]

In 2016, a prototype for a data version control system was developed during a Kaggle competition. This software was later used internally at an AI firm, and eventually spun off as a startup.[8] Since then, a number of data version control systems, both open and closed source, have been developed and offered commercially,[9] with a subset dedicated specifically to machine learning.[10]

Use cases

Reproducibility

A wide range of scientific disciplines have adopted automated analysis of large quantities of data, including astrophysics, seismology, biology and medicine, social sciences and economics, and many other fields. The principle of reproducibility is an important aspect of formalizing findings in scientific disciplines, and in the context of data science presents a number of challenges. Most datasets are constantly changing, whether due to the addition of more data or changes in the structure and format of the data, and small changes can have significant effects on the outcome of experiments. Data version control allows for recording the exact state of data sets at a particular moment of time, making it easier to reproduce and understand experimental outcomes.[11] If data practitioners can only know the present state of the data, they may run into a number of challenges such as difficulties in problem debugging or complying with data audits.

Development and testing

Data version control is sometimes used in testing and development of applications that interact with large quantities of data. Some data version control tools allow users to create replicas of their production environment for testing purposes. This approach allows them to test data integration processes such as extract, transform and load (ETL) and understand the changes made to data without having a negative impact on the consumers of the production data.

Machine learning and artificial intelligence

In the context of machine learning, data version control can be used for optimizing the performance of models. It can allow automating the process of analyzing outcomes with different versions of a data set to continuously improve performance.[12] It is possible that open source data version control software could eliminate the need for proprietary AI platforms by extending tools like Git and CI/CD for use by machine learning engineers.[13] Many open-source solutions build on Git-like semantics to provide these capabilities, as Git itself was designed for small text files and doesn't support typical machine learning datasets, which are very large.

CI/CD for data

CI/CD methodologies can be applied to datasets using data version control.[14] Version control enables users to integrate with automation servers that allow establishing a CI/CD process for data. By adding testing platforms to the process, they can guarantee high quality of the data product. In this scenario, teams execute Continuous Integration (CI) tests on data and set checks in place to ensure the data is promoted to production only all the set data quality and data governance criteria are met.

Experimentation in isolated environments

To experiment on a dataset without impacting production data, one can use data version control to create replicas of the production environment where tests can be carried out. Such replicas allow testing and understanding of changes safely applied to data.

Data version control tools allow replication environments without the time- and resource-consuming maintenance. Instead, such tools allow objects to be shared using metadata.

Rollback

Continuous changes in data sets can sometimes cause functionality issues or lead to undesired outcomes, especially when applications are using the data. Data version control tools allow for the possibility to roll back a data set to an earlier state. This can be used to restore or improve functionality of an application or to correct errors or bad data which has been mistakenly included.[15]

Examples

Version controlled data sources:

Data version control for data lakes:

ML-Ops systems that implement data version control:

See also

References

  1. ↑ "A guide to open source data version control - Fuzzy Labs" (in en). https://www.fuzzylabs.ai/guides/data-version-control. 
  2. ↑ Snodgrass, Richard; Ahn, Ilsoo (1985-05-01). "A taxonomy of time databases". ACM SIGMOD Record 14 (4): 236–246. doi:10.1145/971699.318921. ISSN 0163-5808. https://doi.org/10.1145/971699.318921. 
  3. ↑ Temporal databases : theory, design, and implementation. Internet Archive. Redwood City, Calif. : Benjamin/Cummings Pub. Co.. 1993. ISBN 978-0-8053-2413-6. http://archive.org/details/temporaldatabase0000unse. 
  4. ↑ "Apache Hadoop turns 10: The Rise and Glory of Hadoop" (in en). https://www.projectpro.io/article/apache-hadoop-turns-10-the-rise-and-glory-of-hadoop/211. 
  5. ↑ Bryan, Jennifer (2018-01-02). "Excuse Me, Do You Have a Moment to Talk About Version Control?". The American Statistician 72 (1): 20–27. doi:10.1080/00031305.2017.1399928. ISSN 0003-1305. https://doi.org/10.1080/00031305.2017.1399928. 
  6. ↑ Seering, Adam; Cudre-Mauroux, Philippe; Madden, Samuel; Stonebraker, Michael (2012-04-01). "Efficient Versioning for Scientific Array Databases". 2012 IEEE 28th International Conference on Data Engineering. pp. 1013–1024. doi:10.1109/ICDE.2012.102. ISBN 978-0-7695-4747-3. https://ieeexplore.ieee.org/document/6228152. 
  7. ↑ Bhardwaj, Anant; Bhattacherjee, Souvik; Chavan, Amit; Deshpande, Amol; Elmore, Aaron J.; Madden, Samuel; Parameswaran, Aditya G. (2014-09-02). "DataHub: Collaborative Data Science & Dataset Version Management at Scale". arXiv:1409.0798 [cs.DB].
  8. ↑ "neptune.ai | About us, our story, team and Neptune in the news" (in en-US). https://neptune.ai/about-us. 
  9. ↑ StartupStash. "Top 16 Data Versioning Tools" (in en-US). https://startupstash.com/data-versioning-tools/. 
  10. ↑ Aryan Jadon (26 December 2022). "Survey of Data Versioning Tools for Machine Learning Operations" (in en-US). https://medium.com/@aryanjadon/analysis-of-data-versioning-tools-for-machine-learning-operations-1cb27146ce49. 
  11. ↑ Reproducibility and replicability in science. Engineering, and Medicine.. Washington, DC. 2019. pp. 114. ISBN 978-0-309-48617-0. OCLC 1122461743. https://www.worldcat.org/oclc/1122461743. 
  12. ↑ "Versionskontrolle für Machine-Learning-Projekte" (in de). https://www.informatik-aktuell.de/betrieb/kuenstliche-intelligenz/versionskontrolle-fuer-machine-learning-projekte.html,%20https://www.informatik-aktuell.de/betrieb/kuenstliche-intelligenz/versionskontrolle-fuer-machine-learning-projekte.html. 
  13. ↑ "Streamlining data science with open source: Data version control and continuous machine learning" (in en). https://www.zdnet.com/article/streamlining-data-science-with-open-source-data-version-control-and-continuous-machine-learning/. 
  14. ↑ "The Ultimate Guide to Database Version Control, CI/CD, and Deployment" (in en-AU). 2020-02-01. https://www.databasestar.com/database-version-control-deployment/. 
  15. ↑ "Version Control for Data — The Turing Way" (in en). https://the-turing-way.netlify.app/reproducible-research/vcs/vcs-data.html. 
  16. ↑ "Day 1: Data Versioning & Creating Datasets" (in en). https://kaggle.com/code/ostaski/day-1-data-versioning-creating-datasets. 
  17. ↑ "Quilt Data" (in en). https://quiltdata.com/. 
  18. ↑ Hall, Susan (2020-08-19). "Dolt, a Relational Database with Git-Like Cloning Features" (in en-US). https://thenewstack.io/dolt-a-relational-database-with-git-like-cloning-features/. 
  19. ↑ "X-MOL". https://en.x-mol.com/paper/article/1459234533095145472. 
  20. ↑ "Treeverse raises $23M to bring Git-like version control to data lakes" (in en-US). 2021-07-28. https://venturebeat.com/business/treeverse-raises-23m-to-bring-git-like-version-control-to-data-lakes/. 
  21. ↑ "About Nessie - Project Nessie: Transactional Catalog for Data Lakes with Git-like semantics". https://projectnessie.org/features/. 
  22. ↑ "Git Large File Storage". https://git-lfs.com/. 
  23. ↑ Lardinois, Frederic (2022-06-01). "Iterative launches MLEM, a tool to simplify ML model deployment" (in en-US). https://techcrunch.com/2022/06/01/iterative-launches-mlem-an-open-source-tool-to-simplify-ml-model-deployment/. 
  24. ↑ "Top AI startup news of the week: InstaDeep, DeepL, Pachyderm and more" (in en-US). 2023-01-13. https://venturebeat.com/ai/top-ai-startup-news-of-the-week-instadeep-deepl-pachyderm-and-more/. 
  25. ↑ Ingle, Prathamesh (2022-10-21). "Top Data Version Control Tools for Machine Learning Research in 2022" (in en-US). https://www.marktechpost.com/2022/10/21/top-data-version-control-tools-for-machine-learning-research-in-2022/. 
  26. ↑ Miller, Ron (2021-11-02). "Activeloop snags $5M seed to build streaming database for AI applications" (in en-US). https://techcrunch.com/2021/11/02/activeloop-snags-5m-seed-to-build-database-for-ai-applications/. 
  27. ↑ "Edward Cui, Founder & CEO of Graviti - Interview Series - Unite.AI". https://www.unite.ai/edward-cui-founder-ceo-of-graviti-interview-series/. 
  28. ↑ Ingle, Prathamesh (2022-10-06). "Top Tools for Machine Learning (ML) Experiment Tracking and Management" (in en-US). https://www.marktechpost.com/2022/10/06/top-tools-for-machine-learning-ml-experiment-tracking-and-management/. 
  29. ↑ "How to Set Yourself Apart from Other Applicants with Data-Centric AI" (in en-US). https://www.kdnuggets.com/how-to-set-yourself-apart-from-other-applicants-with-data-centric-ai.html. 
  30. ↑ Shields, Ronan (2023-01-05). "The Trade Desk attempts to woo advertisers at CES with 'Galileo' — a bid to chart the 'Open Internet' without cookies" (in en-US). https://digiday.com/media/the-trade-desk-attempts-to-woo-advertisers-at-ces-with-galileo-a-bid-to-chart-the-open-internet-without-cookies/. 
  31. ↑ Wiggers, Kyle (2022-09-21). "Voxel51 lands funds for its platform to manage unstructured data" (in en-US). https://techcrunch.com/2022/09/21/voxel51-lands-funds-for-its-platform-to-manage-unstructured-data/. 
  32. ↑ Cheptsov, Andrey. "Reproducible ML workflows for teams - dstack" (in en). https://docs.dstack.ai/. 
  33. ↑ Katz, William T.; Plaza, Stephen M. (2019). "DVID: Distributed Versioned Image-Oriented Dataservice". Frontiers in Neural Circuits 13: 5. doi:10.3389/fncir.2019.00005. ISSN 1662-5110. PMID 30804760. 



Retrieved from "https://handwiki.org/wiki/index.php?title=Data_version_control&oldid=3376592"