Concurrency in the Python 3.0 World - Oh My!
Anand B Pillai (~pythonhacker) |
In Python 3.0, concurrent programming takes many forms. One can go with traditional threads using the threading module or perform process based parallelism using the multiprocessing module.
However Python 3.x also comes with concurrent futures a way to parallelize your code using multiple threads or processes and receive the results in a future object - combining parallel processing with asynchronous execution - Pretty elegant!
Wait - there is also the asyncio module with its support for Asynchronous execution - for executing tasks truly asynchronously but with zero parallelism. It is an exciting way of execution typically in one thread but covering multiple tasks bit by bit - in a truly concurrent fashion. This module introduces new ways of creating co-routines using the exciting async and await keywords.
The talk will also explore a few libraries outside core Python - especially we will look at an example of parallelizing execution of a Fractal generation program using multiple cores using PyMP - which is inspired by OpenMP.
We will also briefly explore asynchronous programming with Twisted and also look at some examples using Gevent/Eventlet towards end of the workshop.
The aim of the workshop is to introduce concurrency and parallel programming to it via Python 3.x. We will be touching upon the basics such as GIL, concurrency vs parallelism etc with real world examples and analogies, so even if you are new to the entire topic, the workshop should suit you.
Here is the approximate outline.
- What is Concurrency
- Concurrency vs Parellelism
- The GIL weakness - Multithreading vs Multiprocessing
- Prime number example with multiple threads & multiple processes
- Illustrating how GIL forces computing to 1 core
- Generating Mandelbrot Fractals
- Speeding up using multiprocessing
- Speeding up using PyMP
- Simple generator based task scheduler
- Asynchronous Execution using asyncio
- Co-routines using async def
- Co-routines using Generators
- Tasks & Futures
- Examples - HTTP fetcher, simple web crawler
- Parallelizing using Concurrent futures
- ThreadPool & ProcessPool Executors
- Mapping tasks to inputs
- Submitting and processing asynchronously
- Examples - Image thumbnail generation
- Look at other libraries
- The Reactor pattern
- Gevent & Eventlet with Examples
- When to use - threads vs processes
- When to use - asyncio vs true parallelism
- When to use - concurrent futures vs multiprocessing
- When to use - Twisted vs Python asyncio
- Conclusion & Discussion
- Fundamentals of concurrent computing.
- Previous experience in writing multithreaded or multiprocess applications is good.
- Knowledge of Python internals - GIL etc is useful.
Please find the code content here:
Request participants who plan to attend the workshop to do the following to prepare for workshop.
- Clone the above repository
- Create a virtualenv in Python3
- Install the requirements via the requirements.txt in the repository
NOTE that the examples for twisted/gevent and eventlet dont work in Python3 so those should be installed in a Python2 virtualenv or to your system Python2.
Slides (landslide) at https://github.com/pythonhacker/pydelhi2017concurrency/blob/master/slides.html
Suggest to go through slides once.
Anand has been a long time advocate and community leader of the Python programming language in India and Bangalore.
Anand founded the BangPypers community in Feb 2005 as the result of a meeting of Bangalore Pythonistas. Discussions in this community went on to build other communities and laid the foundation of PyCon India and similar conferences.
He has 17+ years of software development experience having worked in a variety of technical roles in a number of software companies. He is currently working as Director Engineering, Synup at Bangalore.
His interests are high performance computing architectures, scalability & security. Recently he has been wetting his hands in data visualization and machine learning using Python.
CPython Contributions (Patches/Bugs)