multiprocessing.Process()
The multiprocessing module is a standard library for running tasks in parallel using multiple processes. Each process has its own independent memory space and is not subject to Python's GIL (Global Interpreter Lock), allowing CPU-intensive computations to be distributed across multiple CPU cores for faster execution. Because memory is not shared between processes, data must be exchanged through mechanisms such as Queue or Pipe.
Syntax
from multiprocessing import Process, Pool, Queue
# Create and run a process
p = Process(target=function, args=(argument,))
p.start()
p.join()
# Process pool (parallel execution with multiple processes)
with Pool(processes=num_processes) as pool:
results = pool.map(function, iterable)
# Inter-process communication (Queue)
q = Queue()
q.put(data)
data = q.get()
Classes and Methods
| Class / Method | Description |
|---|---|
| Process(target, args) | Creates a child process object. |
| p.start() | Starts the child process. |
| p.join() | Waits for the child process to finish. |
| p.terminate() | Forcibly terminates the process. |
| Pool(processes=N) | Creates a pool of N worker processes. |
| pool.map(func, iter) | Applies func to each element of iter in parallel and returns the results as a list. |
| pool.starmap(func, iter) | Unpacks each element and passes it to func, supporting multiple arguments. |
| pool.apply_async(func, args) | Executes func asynchronously and returns an AsyncResult. |
| Queue() | Creates a queue for passing data between processes. |
| q.put(item) | Adds an item to the queue. |
| q.get() | Retrieves an item from the queue, blocking if empty. |
Sample Code
from multiprocessing import Process, Pool, Queue
import os
import time
# Basic usage of Process
def worker(name, duration):
print(f"[{name}] Process ID: {os.getpid()}, starting")
time.sleep(duration)
print(f"[{name}] Done")
if __name__ == '__main__':
processes = []
for i in range(3):
p = Process(target=worker, args=(f"Worker-{i}", 1))
processes.append(p)
p.start()
for p in processes:
p.join()
print("All processes finished")
# Pool.map: parallelizing CPU-bound tasks
def cpu_task(n):
"""Check if n is prime (simulates a heavy computation)"""
if n < 2:
return False
for i in range(2, int(n ** 0.5) + 1):
if n % i == 0:
return False
return True
if __name__ == '__main__':
numbers = list(range(10000, 10100))
# Sequential processing
start = time.time()
result1 = [cpu_task(n) for n in numbers]
print(f"Sequential: {time.time() - start:.3f}s")
# Parallel processing (4 processes)
start = time.time()
with Pool(processes=4) as pool:
result2 = pool.map(cpu_task, numbers)
print(f"Parallel: {time.time() - start:.3f}s")
# Queue: passing data between processes
def producer(q):
for i in range(5):
q.put(i * i)
time.sleep(0.1)
q.put(None) # Termination signal
def consumer(q):
while True:
item = q.get()
if item is None:
break
print(f"Received: {item}")
if __name__ == '__main__':
q = Queue()
p1 = Process(target=producer, args=(q,))
p2 = Process(target=consumer, args=(q,))
p1.start()
p2.start()
p1.join()
p2.join()
Notes
When using multiprocessing, you must always place the main script code inside an if __name__ == \'__main__\': guard. On Windows, processes are created using the spawn method rather than fork, so this guard prevents child processes from re-importing the script and entering an infinite loop.
Because processes do not share memory, any changes made to a global variable in a child process are not reflected in the parent process. Use Queue, Pipe, or Manager to exchange data between processes.
Spawning processes carries more overhead than spawning threads. For small amounts of data or short-running tasks, parallel processing may actually be slower than sequential processing — always benchmark to verify the benefit.
If you find any errors or copyright issues, please contact us.
シェアする