logo
Splunk Team

python multiprocessing daemon vs non-daemon vs main

  • Last Update :
  • Techknowledgy :

When you see

[INFO / MainProcess] calling join() for process p1

INPUT

        import multiprocessing
        import time
        import logging

        def daemon():
           p = multiprocessing.current_process()
        print('Starting:', p.name, p.pid, flush = True)
        print('---' * 15)
        time.sleep(2)
        print('===' * 15 + ' < ' + f 'Exiting={p.name, p.pid}' + ' > ' + '===' * 15, flush = True)

        def non_daemon():
           p = multiprocessing.current_process()
        print('Starting:', p.name, p.pid, flush = True)
        print('---' * 15)
        print('===' * 15 + ' < ' + f 'Exiting={p.name, p.pid}' + ' > ' + '===' * 15, flush = True)

        if __name__ == '__main__':
           print('===' * 15 + ' < ' + 'MAIN PROCESS' + ' > ' + '===' * 15)
        logger = multiprocessing.log_to_stderr(logging.INFO)
        # DAEMON
        daemon_proc = multiprocessing.Process(name = 'MyDaemon', target = daemon)
        daemon_proc.daemon = True

        # NO - DAEMON
        normal_proc = multiprocessing.Process(name = 'non-daemon', target = non_daemon)
        normal_proc.daemon = False

        daemon_proc.start()
        time.sleep(2)
        normal_proc.start()

        # daemon_proc.join()

OUTPUT NO JOIN METHOD##

=== === === === === === === === === === === === === === === < MAIN PROCESS > === === === === === === === === === === === === === === ===
   Starting: MyDaemon 8448
   -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - [INFO / MyDaemon] child process calling self.run()[INFO / MainProcess] process shutting down[INFO / MainProcess] calling terminate() for daemon MyDaemon[INFO / MainProcess] calling join() for process MyDaemon[INFO / MainProcess] calling join() for process non - daemon
Starting: non - daemon 6688
   -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -
   ===
   === === === === === === === === === === === === === === < Exiting = ('non-daemon', 6688) > === === === === === === === === === === === === === === === [INFO / non - daemon] child process calling self.run()[INFO / non - daemon] process shutting down[INFO / non - daemon] process exiting with exitcode 0

Process finished with exit code 0

OUTPUT + JOIN METHOD

=== === === === === === === === === === === === === === === < MAIN PROCESS > === === === === === === === === === === === === === === ===
   Starting: MyDaemon 13588
   -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - [INFO / MyDaemon] child process calling self.run() ===
   === === === === === === === === === === === === === === < Exiting = ('MyDaemon', 13588) > === === === === === === === === === === === === === === === [INFO / MyDaemon] process shutting down[INFO / MyDaemon] process exiting with exitcode 0[INFO / MainProcess] process shutting down[INFO / MainProcess] calling join() for process non - daemon
Starting: non - daemon 13608
   -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -
   ===
   === === === === === === === === === === === === === === < Exiting = ('non-daemon', 13608) > === === === === === === === === === === === === === === === [INFO / non - daemon] child process calling self.run()[INFO / non - daemon] process shutting down[INFO / non - daemon] process exiting with exitcode 0

Process finished with exit code 0

Suggestion : 2

When a main process exits, it terminates all of its child daemon processes. (Which can be seen here),A main process can not exit before all of its child non-daemon processes exit.,The daemon process is terminated automatically before the main program exits, to avoid leaving orphaned processes running but is not terminated with the main program. So any outstanding processes left by the daemoned function won't run! ,I'm learning multiprocessing in Python while anycodings_python I found this odd behaviour between daemon anycodings_python and non-daemon process with respect to the anycodings_python main process. My Code:

I'm learning multiprocessing in Python while anycodings_python I found this odd behaviour between daemon anycodings_python and non-daemon process with respect to the anycodings_python main process. My Code:

import multiprocessing
import time

def worker(name, num):
   print name, 'Starting'
time.sleep(num)
print name, 'Exiting'

if __name__ == '__main__':
   print 'main starting'
p1 = multiprocessing.Process(target = worker, args = ("p1", 7, ))
p2 = multiprocessing.Process(target = worker, args = ("p2", 5, ))
p2.daemon = True
p1.start()
p2.start()
time.sleep(4)
print 'main exiting'

The output I'm getting is:

main starting
p1 Starting
p2 Starting
main exiting
p1 Exiting

Expected Output:

main starting
p1 Starting
p2 Starting
main exiting
p2 Exiting
p1 Exiting

And the output I got is,

main starting
   [INFO / p1] child process calling self.run()
p1 Starting
   [INFO / p2] child process calling self.run()
p2 Starting
main exiting
   [INFO / MainProcess] process shutting down[INFO / MainProcess] calling terminate() for daemon p2[INFO / MainProcess] calling join() for process p2[INFO / MainProcess] calling join() for process p1
p1 Exiting
   [INFO / p1] process shutting down[INFO / p1] process exiting with exitcode 0

After adding the line p1.join() at the end anycodings_python of the code, I'm getting the following anycodings_python output:

main starting
   [INFO / Process - 1] child process calling self.run()
p1 Starting
   [INFO / Process - 2] child process calling self.run()
p2 Starting
main exiting
p2 Exiting
   [INFO / Process - 2] process shutting down[INFO / Process - 2] process exiting with exitcode 0
p1 Exiting
   [INFO / Process - 1] process shutting down[INFO / Process - 1] process exiting with exitcode 0[INFO / MainProcess] process shutting down

When you see

[INFO / MainProcess] calling join() for process p1

INPUT

        import multiprocessing
        import time
        import logging

        def daemon():
           p = multiprocessing.current_process()
        print('Starting:', p.name, p.pid, flush = True)
        print('---' * 15)
        time.sleep(2)
        print('===' * 15 + ' < ' + f 'Exiting={p.name, p.pid}' + ' > ' + '===' * 15, flush = True)

        def non_daemon():
           p = multiprocessing.current_process()
        print('Starting:', p.name, p.pid, flush = True)
        print('---' * 15)
        print('===' * 15 + ' < ' + f 'Exiting={p.name, p.pid}' + ' > ' + '===' * 15, flush = True)

        if __name__ == '__main__':
           print('===' * 15 + ' < ' + 'MAIN PROCESS' + ' > ' + '===' * 15)
        logger = multiprocessing.log_to_stderr(logging.INFO)
        # DAEMON
        daemon_proc = multiprocessing.Process(name = 'MyDaemon', target = daemon)
        daemon_proc.daemon = True

        # NO - DAEMON
        normal_proc = multiprocessing.Process(name = 'non-daemon', target = non_daemon)
        normal_proc.daemon = False

        daemon_proc.start()
        time.sleep(2)
        normal_proc.start()

        # daemon_proc.join()

OUTPUT NO JOIN METHOD##

=== === === === === === === === === === === === === === === < MAIN PROCESS > === === === === === === === === === === === === === === ===
   Starting: MyDaemon 8448
   -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - [INFO / MyDaemon] child process calling self.run()[INFO / MainProcess] process shutting down[INFO / MainProcess] calling terminate() for daemon MyDaemon[INFO / MainProcess] calling join() for process MyDaemon[INFO / MainProcess] calling join() for process non - daemon
Starting: non - daemon 6688
   -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -
   ===
   === === === === === === === === === === === === === === < Exiting = ('non-daemon', 6688) > === === === === === === === === === === === === === === === [INFO / non - daemon] child process calling self.run()[INFO / non - daemon] process shutting down[INFO / non - daemon] process exiting with exitcode 0

Process finished with exit code 0

OUTPUT + JOIN METHOD

=== === === === === === === === === === === === === === === < MAIN PROCESS > === === === === === === === === === === === === === === ===
   Starting: MyDaemon 13588
   -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - [INFO / MyDaemon] child process calling self.run() ===
   === === === === === === === === === === === === === === < Exiting = ('MyDaemon', 13588) > === === === === === === === === === === === === === === === [INFO / MyDaemon] process shutting down[INFO / MyDaemon] process exiting with exitcode 0[INFO / MainProcess] process shutting down[INFO / MainProcess] calling join() for process non - daemon
Starting: non - daemon 13608
   -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -
   ===
   === === === === === === === === === === === === === === < Exiting = ('non-daemon', 13608) > === === === === === === === === === === === === === === === [INFO / non - daemon] child process calling self.run()[INFO / non - daemon] process shutting down[INFO / non - daemon] process exiting with exitcode 0

Process finished with exit code 0

Suggestion : 3

Last Updated : 17 Aug, 2021,GATE CS 2021 Syllabus

Output: 

this is non - daemon thread
this is non - daemon thread
this is non - daemon thread
main Thread execution
this is non - daemon thread
this is non - daemon thread

Output:

this is thread T
this is thread T
this is Main Thread

Suggestion : 4

Daemons are only useful when the main program is running, and it's okay to kill them off once the other non-daemon threads have exited. Without daemon threads, we have to keep track of them, and tell them to exit, before our program can completely quit. By setting them as daemon threads, we can let them run and forget about them, and when our program quits, any daemon threads are killed automatically.,As we can see from the output, it does not have "Exiting" message from the daemon thread, since all of the non-daemon threads (including the main thread) exit before the daemon thread wakes up from its five second sleep.,In other words, since the timeout passed is less than the amount of time the daemon thread sleeps, the thread is still "alive" after join() returns.,After 3 seconds, the join was timed out, and the daemon thread is still alive and sleep. The main thread and t exited before the daemon thread wakes up from its five second sleep.

Usually our main program implicitly waits until all other threads have completed their work. However, sometimes programs spawn a thread as a daemon that runs without blocking the main program from exiting. Using daemon threads is useful for services where there may not be an easy way to interrupt the thread or where letting the thread die in the middle of its work without losing or corrupting data. To designate a thread as a daemon, we call its setDaemon() method with a boolean argument. The default setting for a thread is non-daemon. So, passing True turns the daemon mode on.

import threading
import time
import logging

logging.basicConfig(level = logging.DEBUG,
   format = '(%(threadName)-9s) %(message)s', )

def n():
   logging.debug('Starting')
logging.debug('Exiting')

def d():
   logging.debug('Starting')
time.sleep(5)
logging.debug('Exiting')

if __name__ == '__main__':

   t = threading.Thread(name = 'non-daemon', target = n)

d = threading.Thread(name = 'daemon', target = d)
d.setDaemon(True)

d.start()
t.start()

Output:

(daemon) Starting
   (non - daemon) Starting(non - daemon) Exiting

Note that if we do not have the time.sleep(5) in the thread function d(), the daemon also exits as well:

(daemon) Starting
   (daemon) Exiting(non - daemon) Starting(non - daemon) Exiting

The following code is using timeout argument (3 seconds) which is shorter than the sleep (5 seconds).

import threading
import time
import logging

logging.basicConfig(level = logging.DEBUG,
   format = '(%(threadName)-9s) %(message)s', )

def n():
   logging.debug('Starting')
logging.debug('Exiting')

def d():
   logging.debug('Starting')
time.sleep(5)
logging.debug('Exiting')

if __name__ == '__main__':

   t = threading.Thread(name = 'non-daemon', target = n)
d = threading.Thread(name = 'daemon', target = d)
d.setDaemon(True)

d.start()
t.start()

d.join(3.0)
print 'd.isAlive()', d.isAlive()
t.join()

However, if we set the timeout 7 seconds:

d.join(7.0)

Suggestion : 5

The output is different when compared to the one generated by daemon threads, because the process in no daemon mode have an output. Hence, the daemonic process ends automatically after the main programs end to avoid the persistence of running processes.,Here, we are using the same example as used in the daemon threads. The only difference is the change of module from multithreading to multiprocessing and setting the daemonic flag to true. However, there would be a change in output as shown below −,When we need parallel execution of our tasks then we need to use theapply_async()method to submit tasks to the pool. It is an asynchronous operation that will not lock the main thread until all the child processes are executed.,Due to this, there is no restriction of executing the bytecode of one thread within our programs at any one time.

The following Python script example will help you understabd how to create a new child process and get the PIDs of child and parent processes −

import os

def child():
   n = os.fork()

if n > 0:
   print("PID of Parent process is : ", os.getpid())

else:
   print("PID of Child process is : ", os.getpid())
child()

Output

PID of Parent process is: 25989
PID of Child process is: 25990

The following example of Python script helps in spawning three processes

import multiprocessing

def spawn_process(i):
   print('This is process: %s' % i)
return

if __name__ == '__main__':
   Process_jobs = []
for i in range(3):
   p = multiprocessing.Process(target = spawn_process, args = (i, ))
Process_jobs.append(p)
p.start()
p.join()

Example

import multiprocessing
import time
def Child_process():
   print('Starting function')
time.sleep(5)
print('Finished function')
P = multiprocessing.Process(target = Child_process)
P.start()
print("My Process has terminated, terminating main thread")
print("Terminating Child Process")
P.terminate()
print("Child Process successfully terminated")

Every process in the operating system is having process identity known as PID. In Python, we can find out the PID of current process with the help of the following command −

import multiprocessing
print(multiprocessing.current_process().pid)

Suggestion : 6

When multiprocessing is initialized the main process is assigned a random string using os.urandom().,Generally synchronization primitives are not as necessary in a multiprocess program as they are in a multithreaded program. See the documentation for threading module.,Manager processes will be shutdown as soon as they are garbage collected or their parent process exits. The manager classes are defined in the multiprocessing.managers module:,multiprocessing uses the usual queue.Empty and queue.Full exceptions to signal a timeout. They are not available in the multiprocessing namespace so you need to import them from queue.

from multiprocessing
import Pool

def f(x):
   return x * x

if __name__ == '__main__':
   with Pool(5) as p:
   print(p.map(f, [1, 2, 3]))
[1, 4, 9]
from multiprocessing
import Process

def f(name):
   print('hello', name)

if __name__ == '__main__':
   p = Process(target = f, args = ('bob', ))
p.start()
p.join()
from multiprocessing
import Process
import os

def info(title):
   print(title)
print('module name:', __name__)
print('parent process:', os.getppid())
print('process id:', os.getpid())

def f(name):
   info('function f')
print('hello', name)

if __name__ == '__main__':
   info('main line')
p = Process(target = f, args = ('bob', ))
p.start()
p.join()
import multiprocessing as mp

def foo(q):
   q.put('hello')

if __name__ == '__main__':
   mp.set_start_method('spawn')
q = mp.Queue()
p = mp.Process(target = foo, args = (q, ))
p.start()
print(q.get())
p.join()
import multiprocessing as mp

def foo(q):
   q.put('hello')

if __name__ == '__main__':
   ctx = mp.get_context('spawn')
q = ctx.Queue()
p = ctx.Process(target = foo, args = (q, ))
p.start()
print(q.get())
p.join()

Suggestion : 7

To mark a process as a daemon, set its daemon attribute with a boolean value. The default is for processes to not be daemons, so passing True turns the daemon mode on.,Passing arguments to identify or name the process is cumbersome, and unnecessary. Each Process instance has a name with a default value that can be changed as the process is created. Naming processes is useful for keeping track of them, especially in applications with multiple types of processes running simultaneously.,Since the timeout passed is less than the amount of time the daemon sleeps, the process is still “alive” after join() returns.,By default, join() blocks indefinitely. It is also possible to pass a timeout argument (a float representing the number of seconds to wait for the process to become inactive). If the process does not complete within the timeout period, join() returns anyway.

import multiprocessing

def worker():
   ""
"worker function"
""
print 'Worker'
return

if __name__ == '__main__':
   jobs = []
for i in range(5):
   p = multiprocessing.Process(target = worker)
jobs.append(p)
p.start()
$ python multiprocessing_simple.py

Worker
Worker
Worker
Worker
Worker
import multiprocessing

def worker(num):
   ""
"thread worker function"
""
print 'Worker:', num
return

if __name__ == '__main__':
   jobs = []
for i in range(5):
   p = multiprocessing.Process(target = worker, args = (i, ))
jobs.append(p)
p.start()
$ python multiprocessing_simpleargs.py

Worker: 0
Worker: 1
Worker: 2
Worker: 3
Worker: 4
import multiprocessing
import multiprocessing_import_worker

if __name__ == '__main__':
   jobs = []
for i in range(5):
   p = multiprocessing.Process(target = multiprocessing_import_worker.worker)
jobs.append(p)
p.start()
def worker():
   ""
"worker function"
""
print 'Worker'
return


Trending Technology


Most popular in python