is there any reason to use ipyparallel for common python script (not ipython notebook) over multiprocessing module?

As of IPython Parallel 7, this will include installing/enabling an extension for both the classic Jupyter Notebook and JupyterLab ≥ 3.0., Using IPython for parallel computing Installing IPython Parallel Quickstart Contents ,Reference Using MPI with IPython IPython’s Task Database Security details of IPython Parallel DAG Dependencies Details of Parallel Computing with IPython Messaging for Parallel Computing Connection Diagrams of The IPython ZMQ Cluster Launchers ,Tutorial Overview and getting started Starting the IPython controller and engines IPython’s Direct interface Parallel Magic Commands The IPython task interface The AsyncResult object Parallel examples

pip install ipyparallel

conda install ipyparallel

import time
import ipyparallel as ipp

task_durations = [1] * 25
# request a cluster
with ipp.Cluster() as rc:
   # get a view on the cluster
view = rc.load_balanced_view()
# submit the tasks
asyncresult = view.map_async(time.sleep, task_durations)
# wait interactively
for results
asyncresult.wait_interactive()
# retrieve actual results
result = asyncresult.get()
# at this point, the cluster processes have been shutdown

import ipyparallel as ipp

def mpi_example():
   from mpi4py
import MPI
comm = MPI.COMM_WORLD
return f "Hello World from rank {comm.Get_rank()}. total ranks={comm.Get_size()}"

# request an MPI cluster with 4 engines
with ipp.Cluster(engines = 'mpi', n = 4) as rc:
   # get a broadcast_view on the cluster which is best
# suited
for MPI style computation
view = rc.broadcast_view()
# run the mpi_example
function on all engines in parallel
r = view.apply_sync(mpi_example)
# Retrieve and print the result from the engines
print("\n".join(r))
# at this point, the cluster processes have been shutdown

Parallel function mapping to a list of arguments (multiprocessing module),Uses the multiprocessing module that comes by default with python, i.e. method independent of IPython.,Idea: you have a function $f(\mathbf{x},\mathbf{y})$ of two parameters (e.g., $f$ may represent your model) stored in the arrays $(\mathbf{x},\mathbf{y})$. Given the arrays $\mathbf{x}$ and $\mathbf{y}$, you want to compute the values of $f(\mathbf{x},\mathbf{y})$. Let's assume for simplicity that there is no dependence on the neighbours. This is an embarassingly parallel problem.,Time wasting function that depends on two parameters. Here, I generate 1E5 random numbers based on the normal distribution and then sum them. The two parameters are $\mu$ and $\sigma$.

In[5]:

   %
   pylab inline

% pylab inline

In [5]:

    

%pylab inline


    

Populating the interactive namespace from numpy and matplotlib

% pylab inline

Populating the interactive namespace from numpy and matplotlib

In[6]:

   import multiprocessing

import multiprocessing

import multiprocessing

In[7]:

   import scipy

def f(z):
   x = z[1] * scipy.random.standard_normal(100000) + z[0]
return x.sum()

import scipy

def f(z):
   x = z[1] * scipy.random.standard_normal(100000) + z[0]
return x.sum()

Using the web interface provided in the Jupyter Notebook's main page by clicking on the IPython Clusters tab and launching four engines, This is one of the 100+ free recipes of the IPython Cookbook, Second Edition, by Cyrille Rossant, a guide to numerical computing and data science in the Jupyter Notebook. The ebook and printed book are available for purchase at Packt Publishing.,In this recipe, we used the direct interface; we addressed individual engines explicitly by specifying their identifiers in the %px magics.,1.  Launching several IPython engines (there is typically one process per core). 2.  Creating a Client that acts as a proxy to these engines. 3.  Using the client to launch tasks on the engines and retrieve the results.

from ipyparallel
import Client
rc = Client()

rc.ids

[0, 1, 2, 3]

% % px
import os
print(f "Process {os.getpid():d}.")

[stdout: 0] Process 10784.
   [stdout: 1] Process 10785.[stdout: 2] Process 10787.[stdout: 3] Process 10791.

% % px - t 1, 2
# The os module has already been imported in
   # the previous cell.
print(f "Process {os.getpid():d}.")