Software:Pegasus (workflow management)

From HandWiki
Jump to: navigation, search
Pegasus
Developer(s)University of Southern California, Information Sciences Institute, University of Wisconsin-Madison
Stable release
4.9.3 / January 31, 2020; 4 months ago (2020-01-31)
Written inJava, Python, C
Operating systemmacOS, Linux
Available inJava, Python, C
TypeWorkflow management system
LicenseApache License 2.0
Websitepegasus.isi.edu

Pegasus is an open-source workflow management system.[1] It provides the necessary abstractions for scientists to create workflows and allows for transparent execution of these workflows on a range of computing platforms including clusters, clouds, and national cyberinfrastructure.[2][3] In Pegasus, workflows are described abstractly as directer acyclic graphs (DAGs). Pegasus enables user to construct their workflows using Jupyter Notebooks, Python, R, and Java. During execution, Pegasus translates the abstract workflow into an executable workflow.[4][5] Workflow execution with Pegasus includes data management, monitoring, and failure handling. Pegasus relies on the DAGMan workflow engine to manage the task dependencies.[6] Individual workflow tasks are managed by a task scheduler (HTCondor), which supervises their execution on local and remote resources.[7]

Pegasus is being used in production to execute scientific workflows for dozens of high-profile applications in a number of different disciplines including astronomy, gravitational-wave physics, bioinformatics, earthquake engineering, and helioseismology. [8] Notably, the LIGO Scientific Collaboration has used it to directly detect a gravitational wave for the first time.[4]

Overview

Pegasus uses directed acyclic graphs (DAGs) to manage workflow orchestration. Tasks represent applications defined in any programming language, and dependencies are typically defined as files.

Area of applications

Application examples:[2][3]

  • Gravitational-Wave Physics
  • Earthquake Science
  • Bioinformatics
  • Workflows for Volcanic Mass Flows
  • Diffusion Image Processing and Analysis
  • Spallation Neutron Source (SNS)

See also

References

  1. E. Deelman, K. Vahi, G. Juve, M. Rynge, S. Callaghan, P. J. Maechling, R. Mayani, W. Chen, R. Ferreira da Silva, M. Livny, and K. Wenger, "Future Generation Computer Systems", Elsevier; 46, pp. 17-35 (2015)
  2. 2.0 2.1 E. Deelman, G. Singh, M. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, B. G. Berriman, J. Good, A. Laity, J. C. Jacob, and D. S. Katz, “Pegasus: a Framework for Mapping Complex Scientific Workflows onto Distributed Systems”, Scientific Programming; 13, pp. 19 (2005)
  3. 3.0 3.1 The Scientific Workflow Integrity with Pegasus (SWIP), by Center for Applied Cybersecurity Research; published 16 September 2016; retrieved 1 May 2020
  4. 4.0 4.1 "Testing LIGO's Sensitivity". September 1, 2007. https://www.research.gov/research-portal/appmanager/base/desktop?_nfpb=true&_windowLabel=researchAreas_11&_urlType=action&wlpresearchAreas_11_id=%2FresearchGov%2FAwardHighlight%2FPublicAffairs%2F2012%2F23562_TestingLIGOsSensitivity.html&wlpresearchAreas_11_action=selectAwardDetail. 
  5. Duncan Brown and Ewa Deelman, "Looking for gravitational waves: A computing perspective", at Science Node; published June 8, 2011; retrieved April 30, 2020
  6. $1M NSF award goes to IU-led data integrity project, by Indiana University; published 16 September 2016; retrieved 1 May 2020
  7. Brian Mattmiller, "High Throughput Computing helps LIGO confirm Einstein’s last unproven theory", at Morgridge Institute for Research; published March 7, 2016; retrieved May 1, 2020
  8. Sanden Totten, "Caltech Wasn’t the Only SoCal School Helping Discover Gravitational Waves", at KPCC; published 11 February 2016; retrieved May 1, 2020


Category:Workflow applications

References

https://en.wikipedia.org/wiki/Pegasus (workflow management) was the original source. Read more.