Software:Python SCOOP

Python SCOOP

Original author(s)	Marc Parizeau and Yannick Hold
Developer(s)	Yannick Hold and Olivier Gagnon
Stable release	0.7.1 / March 17, 2014 (2014-03-17)
Written in	Python
Operating system	POSIX-compliant
Platform	Cross-platform
Type	Distributed computing framework
License	LGPL
Website	www.pyscoop.org

SCOOP (Scalable Concurrent Operations in Python) is a Python software module for distributing concurrent tasks on various environments, from heterogeneous grids of workstations to supercomputers.

It uses ØMQ and the Greenlet package as building blocks to encapsulate and distribute tasks (named a Future) between processes and/or systems. Its interface is inspired from the PEP-3148 proposal.

SCOOP is targeted towards scientific applications that require execution of many loosely coupled tasks using all available hardware resources. These resources need to be accessible through SSH.

History

SCOOP was initiated by Yannick Hold and Marc Parizeau at the Computer Vision and Systems Laboratory of Université Laval. It is an iterative step over the now deprecated DTM module of the DEAP framework for developing evolutionary algorithm. While DTM used MPI for its communications, SCOOP uses instead ØMQ.

Network topology

SCOOP uses the Broker Architecture to distribute its Futures. It is based on a central element, called the Broker, that dispatches work to its workers. The main difference between this pattern and a Master/slave topology reside in the Future origin. In the Broker architecture, the Futures emanate from a worker, which is located on the periphery of the topology, instead of the master in the Master/slave architecture. This allows higher reliability in regard to worker faults and generally better performances due to the generic function of the Broker. Since he doesn't need to serialize nor deserialize any Future to route them around the grid, his workload consists of networking or interprocess I/O and almost no CPU processing time. This lowers the bottleneck of the Broker topology.

The Broker architecture won't stress the networking fabric and elements as much as a totally distributed topology since there is only one connection required on every worker.

Example

An introductory parallel "Hello, world!" example is implemented this way:

from scoop import futures

def hello_world(value) -> str:
    return "Hello World from Future #{}".format(value)

if __name__ == "__main__":
    return_values = futures.map(hello_world, range(16))
    print("\n".join(return_values))

External links

• Official documentation