asyncoro: Asynchronous, Concurrent, Distributed Programming with Python¶
asyncoro is a Python framework for asynchronous, concurrent, network, distributed programming and distributed computing, using generator functions, asynchronous completions and message passing. asyncoro can be used to create coroutines with generator functions, similar to the way threads are created with functions with Python’s threading module. Programs developed with asyncoro have same logic and structure as programs with threads, except for a few syntactic changes - mostly using yield with asynchronous completions that give control to asyncoro’s scheduler, which interleaves executions of generators, similar to the way an operating system executes multiple processes.
Unlike threads, creating processes (coroutines) with asyncoro is very efficient. Moreover, with asyncoro context switch occurs only when coroutines use yield (typically with an asychronous call), so there is no need for locking and there is no overhead of unnecessary context switches.
asyncoro features include:
- No callbacks or event loops! No need to lock critical sections either,
- Efficient polling mechanisms epoll, kqueue, /dev/poll, Windows I/O Completion Ports (IOCP) for high performance and scalability,
- Asynchronous (non-blocking) sockets and pipes, for concurrent processing of I/O,
- SSL for security,
- Asynchronous timers, including non-blocking sleep,
- Asynchronous locking primitives similar to Python threading module,
- Message passing for (local and remote) coroutines to exchange messages one-to-one with Message Queue Pattern or through broadcasting channels with Publish-Subscribe Pattern,
- Location transparency with naming and locating (local and remote) resources,
- Monitoring and restarting of (local or remote) coroutines, for fault detection and fault-tolerance,
- Distributing computation components (code and data) for execution of Distributed Communicating Processes (discoro), for wide range of use cases, covering SIMD, MISD, MIMD system architectures at the process level, and web interface to monitor cluster, nodes, servers and coroutines; in-memory processing and data streaming are supported as well,
- Remote execution of coroutines for distributed programming
with Remote Coroutine Invocation
RCIand message passing,
- Hot-swapping of coroutine functions, for dynamic system reconfiguration,
- Thread pools with asynchronous task completions, for executing synchronous tasks, e.g., external library calls, such as reading standard input.
import time import asyncoro def coro_proc(coro=None): yield coro.suspend() coros = [asyncoro.Coro(coro_proc) for i in xrange(10000)] time.sleep(5) # wait for coroutines to run (and suspend) for coro in coros: coro.resume() for line in open('/proc/self/status'): if line.startswith('VmPeak'): print('VM Peak: %d MB' % (int(line.split()) / 1024)) elif line.startswith('VmHWM'): print('VM Max: %d MB' % (int(line.split()) / 1024)) elif line.startswith('VmStk'): print('VM Stack: %d MB' % (int(line.split()) / 1024))
shows that running 10,000 processes takes about 23 MB of memory (with about 10 MB taken by modules used in asyncoro). Memory and time scale linearly: For 100,000 processes memory used is 116 MB, for 200,000 processes it is 218 MB, for 300,000 processes it is 315 MB.
asyncoro is implemented with standard modules in Python. Under Windows efficient polling notifier I/O Completion Ports is supported if pywin32 is installed; otherwise, inefficient ‘select’ notifier is used.
asyncoro works with Python 2.7+ and Python 3.1+ and tested on Linux, Mac OS X and Windows; it may work on other platforms too. asyncoro works with PyPy as well.
asyncoro package is available in Python Package Index (PyPI) so it can be installed for Python 2.7+ with:
pip install asyncoro
and/or for Python 3 with:
pip3 install asyncoro
asyncoro can also be downloaded from Sourceforge Files. ‘py2’ directory contains files to be used with Python 2.7+ and ‘py3’ contains files to be used with Python 3.1+. To use asyncoro without installing, rename ‘py2’ to ‘asyncoro’ (to use with Python 2.7+); it can then be used from parent directory of ‘asyncoro’.
Examples illustrating some of the features of asyncoro are installed in ‘examples’ directory under where asyncoro module is installed, which can be obtained with the program:
import os, asyncoro print(os.path.join(os.path.dirname(asyncoro.__file__), 'examples'))
README file in that directory for
brief description of each of the files.
- 1. Introduction
- 2. Asynchronous Concurrenct Programming (asyncoro)
- 3. Asynchronous Network Programming (asyncoro)
- 4. Asynchronous Files and Pipes (asyncfile)
- 5. Distributed Programming (disasyncoro)
- 6. Distributed Communicating Processes (discoro)
- 7. Tutorial / Examples
- 8. Contribute to / Recommend / Share asyncoro