Python parallel for loop can simplify the producer-consumer model, and we provide a simple Python example that uses the async thread module, asyncio, for demonstration.
The example simulates the behaviors of restaurant’s cookers and their customers. Briefly, Python language provides a quick method to execute for loop in parallel that meets the feature of such a problem.
All codes here are not complicated, so you can easily understand even though you are still students in school. To benefit your learning, we will provide you download link to a zip file thus you can get all source codes for future usage.
Estimated reading time: 5 minutes
EXPLORE THIS ARTICLE
TABLE OF CONTENTS
BONUS
Source Code Download
We have released it under the MIT license, so feel free to use it in your own project or your school homework.
Download Guideline
- Install Python on Windows by clicking Python Downloads, or search a Python setup pack for Linux.
- The installation package for Windows also contains pip install, which allow you to obtain more Python libraries in the future.
SECTION 1
Producer-Consumer Model
In the section, the Producer-Consumer problem is simulated using async threads in Python that execute for loop in parallel. The model helps researches about the balance of supplies and consumption.
Suggested environment is Python 3.7+, because beyond that, there are some differences for async syntax.
Number of Participants
How many producers does the model need to satisfy all consumers? How many consumers is the constraint upon available producers? Researchers can use Python’s asyncio module to create as many async threads for producers and customers as they like, so that they will evaluate the number of participants. Let’s walk through an example for restaurants.
Restaurant – Cooker and Customer
In the restaurant, managers have to arrange enough cookers so as to provide services for customers. For example, we have one cooker which produces a meal per 3 seconds, and two customers which consume a meal per 3 to 10 seconds. Let’s take a look at the simulation.
- READY : The cooker make a meal ready.
- TAKEN AWAY : The customer take a meal away.
- Food Queue Empty : The food queue space is empty.
- Food Queue Full : The food queue space is full.
C:\>python parallel.py
20:28:37 producer - **Air Fryer Steak** READY
20:28:37 consumer_1 - **Air Fryer Steak** TAKEN AWAY
20:28:40 consumer_2 - ! Food Queue Empty !!!
20:28:40 producer - **Beef Goulash** READY
20:28:42 consumer_1 - **Beef Goulash** TAKEN AWAY
20:28:43 producer - **Apple Pie** READY
20:28:46 consumer_2 - **Apple Pie** TAKEN AWAY
20:28:46 consumer_1 - ! Food Queue Empty !!!
20:28:46 producer - **Air Fryer Steak** READY
20:28:49 producer - **Fried Rice** READY
20:28:50 consumer_1 - **Air Fryer Steak** TAKEN AWAY
20:28:52 producer - **Pumpkin Soup** READY
20:28:55 producer - **Fried Rice** READY
20:28:56 consumer_2 - **Fried Rice** TAKEN AWAY
20:28:57 consumer_1 - **Pumpkin Soup** TAKEN AWAY
20:28:58 producer - **Air Fryer Steak** READY
20:29:00 consumer_2 - **Fried Rice** TAKEN AWAY
20:29:01 consumer_1 - **Air Fryer Steak** TAKEN AWAY
20:29:01 producer - **Favourite Lasagne** READY
20:29:04 producer - **Favourite Lasagne** READY
20:29:07 producer - **Air Fryer Steak** READY
20:29:08 consumer_2 - **Favourite Lasagne** TAKEN AWAY
20:29:10 producer - **Favourite Lasagne** READY
20:29:11 consumer_1 - **Favourite Lasagne** TAKEN AWAY
20:29:13 producer - **Fried Rice** READY
20:29:14 consumer_2 - **Air Fryer Steak** TAKEN AWAY
20:29:16 producer - **Thai Beef Salad** READY
20:29:19 producer - ! Food Queue Full !!!
20:29:21 consumer_1 - **Favourite Lasagne** TAKEN AWAY
20:29:22 consumer_2 - **Fried Rice** TAKEN AWAY
20:29:22 producer - **Air Fryer Steak** READY
Limit to Queue
Python’s asyncio module also provides a queue structure that contains data items with a configured limitation. Therefore, the queue will be full if producer threads make too many meals, and be empty if consumer threads take away meals too frequently.
SECTION 2
Python Parallel for Loop
Traditionally, programs are running in sequential steps. However, solving the producer-consumer problem needs Python’s async threading technology for both sides to perform in parallel so that researchers can observe and discover which side should increase capability to keep balanced.
Threads in Parallel
First, asyncio.run(main()) start the parallel for loop in Python. The asynchronous function async def main() creates a queue as food container with limitation of 3. Then it creates 1 async thread for the only producer and 2 async threads for two consumers. That is the core structure for concurrent loop in Python.
The producer, function async def producer(queue), always make a random meal from 8 templates per 3 seconds. However, each consumer takes a meal from FIFO queue per random seconds from 3 to 10. The program will raise except for FULL and EMPTY conditions.
'''
Simulate Producer-Consumer Model
'''
import random
import asyncio
import json
from datetime import datetime
all_foods = ['Beef Goulash', 'Chicken Parmesan Pasta', 'Air Fryer Steak', 'Pumpkin Soup', 'Fried Rice', 'Apple Pie', 'Favourite Lasagne', 'Thai Beef Salad']
def do_queue(queue, role, item="") :
try:
ts = datetime.now().strftime("%H:%M:%S")
if item == "" :
item = queue.get_nowait() # raise exception if empty
print(f"\n{ts} {role} - **{item}** TAKEN AWAY")
else :
queue.put_nowait(item) # raise exception if full
print(f"\n{ts} {role} - **{item}** READY")
except asyncio.QueueEmpty:
print(f"\n{ts} {role} - ! Food Queue Empty !!!")
except asyncio.QueueFull:
print(f"\n{ts} {role} - ! Food Queue Full !!!")
''' Asynchronous routines '''
async def producer(queue):
while True:
do_queue(queue, "producer", random.choice(all_foods))
await asyncio.sleep(3)
async def consumer_1(queue):
while True:
do_queue(queue, "consumer_1")
await asyncio.sleep(random.randint(3,10))
async def consumer_2(queue):
while True:
do_queue(queue, "consumer_2")
await asyncio.sleep(random.randint(3,10))
async def main():
queue = asyncio.Queue(3) # max size
task0 = asyncio.create_task(producer(queue))
task1 = asyncio.create_task(consumer_1(queue))
task2 = asyncio.create_task(consumer_2(queue))
await task0
await task1
await task2
asyncio.run(main())
The diagram depicts the producer-consumer model using Python async threads. The three orange colored Python threads are running for loop in parallel. Like assembly lines in factories, producers push meals one by one until container gets full with alerts. And consumers pull meals one by one until container becomes empty also with alarms.
FINAL
Conclusion
Python async thread provides a concise approach to parallel for loop, but other languages probably need trivial steps. If interested, you can modify time interval in loop or change the number of producers and consumers in the Python example, so that the best solution for allocating resources would be generated.
Thank you for reading, and we have suggested more helpful articles here. If you want to share anything, please feel free to comment below. Good luck and happy coding!
Suggested Reading
- Solving Post JSON to Python CGI FieldStorage Error
- Googletrans Python Example for A French Chatbot
- Auto Refresh HTML Data in Web Page by 4 Methods