Python Multiprocessing Jump-Start

Super Fast Python
2 ratings

Python Code Does Not Have to Be Slow!
(or run on just one CPU)

With "Python Multiprocessing Jump-Start" you will learn how to develop parallel Python programs in just 7 lessons.

How Much Faster Could Your Code Run
...if it used all CPU cores?

Has this happened to you...

You develop your Python program to perform some routine task many times, such as in a loop.

You run the script and are frustrated at how long it takes.

Yet, you have 2, 4, 8 or more CPU cores sitting idle.
Using electricity.
Waiting for work.

What a waste!

  • What if you could develop Python programs that were parallel from the start?
  • What if your parallel Programs used all of the CPU cores in your system the first time?

This is possible right now with a little-used Python module that offers process-based concurrency (and is already installed on your system).

Concurrency is the Path to Faster Code

Python is a joy to use, but getting Python code to run fast is challenging.

Concurrency is a standard approach to running multiple functions simultaneously.

Python concurrency has a bad reputation. So bad, that many developers believe Python does not support true concurrency.

I'm happy to say that these misconceptions are dead wrong.

Python supports real concurrency with first-class native support for threads and processes.

It always has.

  • ...on all recent Python versions, like Python v2.6+ and v3.0+.
  • ...on all major platforms, like Windows, MacOS, and Linux.
  • ...with all major hardware, like Intel, AMD, ARM, and Apple Silicon.

And most importantly, Python concurrency is easy and fun to use.

The trick is to use the multiprocessing module...

Multiprocessing: Module for Parallelism
....that you've been missing

The multiprocessing module provides easy-to-use process-based concurrency.

This is not some random third-party library that is hard to install, this is the Python standard library (already installed on your system).

This is the module you need to use to make your code run faster.

It is specifically designed for you to develop parallel Python programs and make use of all of the CPU cores available in your system.

It does this by side-stepping the infamous Global Interpreter Lock (GIL).

Unlike processes, Python threads only provide limited parallelism given the presence of the GIL. This is widely known and leads to opinions like:

"Python doesn't support concurrency because of the GIL."

But this is only true for threads, not processed-based concurrency.

The answer is the multiprocessing module.

This module was developed specifically to offer parallelism by side-stepping the GIL completely.

There's just one problem.

Few developers know about it (or how to use it well).

  • ...the API documentation is thin at best, providing very little guidance on how to use its features.
  • ...the API docs are buried deep within the Python documentation website, making it impossible to find.
  • ...the few examples out there on the web are terse and 10 years out of date.

This is madness!

The perfect module for faster parallel Python programs is right there in the standard library, and few Python developers know it exists or can locate it, let alone decipher the API documentation.

One group of developers know about this module and how to use it well.

Python machine learning developers...

My Story: Who Is Jason Brownlee?

(from Python ML engineer to Python concurrency expert and evangelist)

Hi, I'm Jason Brownlee, Ph.D.

I'm a Python developer, husband, and father to two boys.

(a photo of me in the backyard, taken by my wife)

I want to share something with you.

I am obsessed with Python concurrency, but I wasn't always this way.

My background is in Artificial Intelligence and I have a few fancy degrees and past job titles to prove it.

You can see my LinkedIn profile here:

So what?

Well, AI and machine learning has been hot for the last decade. I have spent that time as a Python machine learning developer:

  • ... working on a range of predictive modeling projects.
  • ... writing more than 1,000+ tutorials.
  • ... and authoring over 20+ books.

There's one thing about machine learning in Python, your code must be fast.

Really fast.

Modeling code is already generally fast, built on top of C and Fortran code libraries.

But you know how it is on real projects...

You always have to glue bits together, wrap the fast code and run it many times, and so on.

Making code run fast requires Python concurrency and I have spent most of the last decade using all the different types of Python concurrency available.

Including threading, multiprocessing, asyncio, and the suite of popular libraries.

I know my way around Python concurrency and I am deeply frustrated at the bad wrap it has.

This is why I started SuperFastPython.com where you can find hundreds of free tutorials on Python concurrency.

And this is why I wrote this book...

Introducing:
"Python Multiprocessing Jump-Start"

...develop parallel programs, side-step the GIL, use all CPU cores

(cover of the book)

You don't want to read a book.

You want faster Python code, and you want it yesterday.

I hear you.

  • This is not a textbook on Python Concurrency
  • It's not a technical thesis on the internals of process-based concurrency in Python
  • Instead, it's what you actually need...

It's a fast-paced, 7-part ebook to get you started and get you good at using the multiprocessing module, super fast.

Each of the 7 lessons was carefully designed to teach one critical aspect of the multiprocessing module, with explanations, code snippets and worked examples.

Each lesson ends with an exercise for you to complete to confirm you understood the topic, a summary of what was learned, and links for further reading if you want to go deeper.

Next, let's take a closer look at the 7-lessons in the book....

So What Are The Lessons?
...7 lessons to be completed over 7 days

This book is designed to bring you up-to-speed with how to use the multiprocessing module as fast as possible.

As such, it is not exhaustive.

There are many topics that are interesting or helpful, that are not on the critical path to getting you productive.

This book is divided into 7 lessons, they are:

  • Lesson 01: Process-Based Concurrency
  • Lesson 02: Create and Start Child Processes
  • Lesson 03: Configuring and Interacting with Processes
  • Lesson 04: Synchronize and Coordinate Processes
  • Lesson 05: Share Data Between Processes
  • Lesson 06: Run Tasks with Reusable Workers in Pools
  • Lesson 07: Share Centralized Objects with Managers

Next, let's look at the structure of each lesson...

So What is the Structure of Each Lesson?
...tutorials followed by exercises

Each lesson has two main parts, they are:

  1. The body of the lesson (code tutorials)
  2. The lesson overview (exercises and refs)

The body of the lesson will introduce a topic with code examples, whereas the lesson overview will review what was learned with exercises and links for further information.

Each lesson has a specific learning outcome and is designed to be completed in 10-to-20 minutes.

Each lesson is also designed to be self-contained so that you can read the lessons out of order if you choose, such as dipping into topics in the future to solve specific programming problems.

We Python developers learn best from real and working code examples. So each lesson has multiple large worked examples with sample output.

So, after completing the book, what will you know?

What You Will Know After Reading

...become a "Dev Who Can Write Programs to Run on All Cores"

This book will transform you from a Python developer into a Python developer that can confidently bring concurrency to your projects with the multiprocessing module.

After working through all of the lessons in this book, you will know:

  • The difference between thread-based and process-based concurrency and the types of tasks that are well suited to the capabilities of the multiprocessing module.
  • How to execute your own ad hoc functions in parallel using the Process class.
  • How to identify the main process, about parent and child processes, and the life-cycle of processes in Python.
  • How to configure a new child process and access Process instance for running processes, kill them, and query their status such as their exit code and whether they are still running.
  • How to coordinate and synchronize Python processes using mutex locks, semaphores, condition variables and the full suite of concurrency primitives.
  • How to inherit and use global variables from parent processes when using the fork start method.
  • How to share data between processes using shared ctypes, how to send and receive data using pipes and how to create producer and consumer processes using queues.
  • How to create and configure multiprocessing pools to execute ad hoc tasks using reusable child worker processes.
  • How to process results, handle errors, and query the status of asynchronous tasks executed in multiprocessing pools.
  • How to create centralized Python objects that can be accessed and used in a process-safe manner using proxy objects.

How Long Will It Take?
...a week of one lesson per day

About 20-to-30 minutes per lesson, so a few hours total.

But that's too fast.

Instead, I recommend 1 lesson per day, over 7 days (1 week).

Work at your own pace.

There's no rush and I recommend that you take your time.

The book is designed to be read linearly from start to finish, guiding you from being a Python developer at the start of the book to being a Python developer that can confidently use the multiprocessing module in your projects by the end of the book.

In order to avoid overload, I recommend completing one or two lessons per day, such as in the evening or during your lunch break. This will allow you to complete the transformation in about one week.

I recommend you maintain a directory with all of the code you type from the lessons in the book. This will allow you to use the directory as your own private code library, allowing you to copy-paste code into your projects in the future.

I recommend trying to adapt and extend the examples in the lessons. Play with them. Break them. This will help you learn more about how the API works and why we follow specific usage patterns.

What Format Is the Ebook?

The ebook is provide in 2 formats:

  • PDF (.pdf): perfect for reading on the screen or tablet.
  • EPub (.epub): perfect for reading on a tablet with a Kindle or iBooks app.

Who Is This Book For?

Before you pull the trigger, let's make sure it is a good fit for you.

This book is designed for Python developers who want to discover how to use and get the most out of the multiprocessing module to write fast programs.

Specifically, this book is for:

  • ...developers that can write simple Python programs.
  • ...developers that need better performance from current or future Python programs.
  • ...developers that are working with CPU-bound tasks.

This book does not require that you are an expert in the Python programming language or concurrency.

Specifically:

  • ...you do not need to be an expert Python developer.
  • ...you do not need to be an expert in concurrency.

What Version of Python is Used?

All code examples use Python 3.

Python 3.9+ to be exact.

Python 2.7 is not supported because it reached end of life in 2020.

Are There Code Examples?

Yes.

There are 24 .py code files.

Each lesson has one or more complete, standalone, and fully-working code examples.

The book is provided in a .zip file that includes a src/ directory containing all source code files used in the book.

How Many Pages Is The Book?

The PDF is 102 US letter sized pages.

Can I Print The Book?

Yes.

Although, I think it's better working through it on the screen.

  • You can search, skip, and jump around really fast.
  • You can copy-and-paste code examples.
  • You can compare code output directly.

Is There Digital Rights Management?

No.

The ebooks have no DRM.

What if I Need Help?

The lessons were designed to be easy to read and follow.

Nevertheless, sometimes we need a little extra help.

A list of further reading resources is provided at the end of each lesson. These can be helpful if you are interested in learning more about the topic covered, such as fine grained details of the standard library and API functions used.

The conclusions at the end of the book provide a complete list of websites and books that can help if you want to learn more about Python concurrency and the relevant parts of the Python standard library. It also lists places where you can go online and ask questions about Python concurrency.

Finally, if you ever have questions about the lessons or code in this book, you can contact me any time and I will do my best to help. My contact details are provided at the end of the book.

Do I Get Free Updates?

Yes.

Each time I release an updated version, I will send you an email with a link so that you can download the latest version for free.

Happiness Guarantee

I want you to be happy, and I stand behind my materials.

If you decide that Python concurrency is not for you, or whatever reason, I'll understand.

I offer a 100% money back guarantee, no questions asked.

Get in touch at:

Can I Buy The Book Elsewhere?

Yes!

You can get a kindle or paperback version from Amazon.

In fact, the book was the #1 new release in its category shortly after its release.

#1 new release in Parallel Computer Programming

Is There a Paperback Version?

Yes!

You can get a paperback version from Amazon.

Can I Read a Sample?

Yes.

You can read a book sample via google books "preview" or via the amazon "look inside" feature:

Generally, if you like my writing style on SuperFastPython, then you will like the books.

Can I Download the Source Code Now?

The source code (.py) files are included in the .zip with the book.

Nevertheless, you can also download all of the code from the dedicated GitHub Project:

Does It Work on My Operating System?

Yes.

Python parallelism is built into the Python programming language and works equally well on:

  • Windows
  • MacOS
  • Linux

Does It Work on My Hardware?

Yes.

Python parallelism is agnostic to the underlying CPU hardware.

If you are running Python on a modern computer, then you will have support for parallelism, e.g. Intel, AMD, ARM, and Apple Silicon CPUs are supported.

How do the Jump-Start Books Compare to the Guides?

The SuperFastPython Jump-Start books are laser-focused on making you productive with a Python concurrency module or class as fast as possible.

This means that many broader topics are not covered because they are not on the critical path.

The guides on SuperFastPython.com are broader in scope and compare the class or module to related modules, describing best practices, common errors, common usage questions, and common objects.

This material may be interesting but is a distraction when you are focused on getting productive as fast as possible.

Another important difference is that the jump-start books are provided in book form, whereas the guides are very long web pages.

This makes the books easy to read on a kindle, tablet, or paperback, as well as the screen, whereas the guides must be read in the browser.

Any Questions?

Contact me directly, any time about this book or Python concurrency generally.

I'm here to help as best I can.

You can send an email directly to my inbox via:

Praise for Super Fast Python

What Are You Waiting For?

Stop reading out-dated StackOverflow answers.

Learn Python concurrency correctly, step-by-step.

Start today.

Buy now and get your copy in seconds!

I want this!

You'll get a .zip download containing both ebook formats (pdf and epub) and all .py code files.

Formats
PDF and EPUB
Code
24 .py files
Lessons
7
Pages
102

Ratings

5.0
(2 ratings)
5 stars
100%
4 stars
0%
3 stars
0%
2 stars
0%
1 star
0%
Powered by