why python logging writes to stderr by default?

For me it isn't really obvious why stderr is the default. I noticed that something is wrong when I run python script.py | tee out.log and ended up with empty log file. Now I know that It can be solved by either python script.py 2>&1 | tee out.log or using stream parameter:

logging.basicConfig(stream = sys.stdout)

Suggestion : 2

The default level is WARNING, which means that only events of this level and above will be tracked, unless the logging package is configured to do otherwise.,Loggers have a concept of effective level. If a level is not explicitly set on a logger, the level of its parent is used instead as its effective level. If the parent has no explicit level set, its parent is examined, and so on - all ancestors are searched until an explicitly set level is found. The root logger always has an explicit level set (WARNING by default). When deciding whether to process an event, the effective level of the logger is used to determine whether the event is passed to the logger’s handlers.,The logging functions are named after the level or severity of the events they are used to track. The standard levels and their applicability are described below (in increasing order of severity):,The following Python module creates a logger, handler, and formatter nearly identical to those in the example listed above, with the only difference being the names of the objects:

import logging
logging.warning('Watch out!') # will print a message to the console
logging.info('I told you so') # will not print anything

WARNING: root: Watch out!

import logging
logging.basicConfig(filename = 'example.log', encoding = 'utf-8', level = logging.DEBUG)
logging.debug('This message should go to the log file')
logging.info('So should this')
logging.warning('And this, too')
logging.error('And non-ASCII stuff, too, like Øresund and Malmö')

DEBUG: root: This message should go to the log file
INFO: root: So should this
WARNING: root: And this, too
ERROR: root: And non - ASCII stuff, too, like Øresund and Malmö

--log = INFO

getattr(logging, loglevel.upper())

Suggestion : 3

Logging is a module in the Python standard library that provides a richly-formatted log with a flexible filter and the possibility to redirect logs to other sources such as syslog or email.,This Python logging tutorial is not meant to be a complete document on the logging module but rather a “getting started” guide that introduces some logging concepts as well as some “gotchas” to watch out for. The post will end with best practices and contain some pointers to more advanced logging topics.,This section gives an overview on some concepts that are often encountered in the logging module.,The logging module is indeed very handy, but it contains some quirks that can cause long hours of headache for even the best Python developers. Here are the best practices for using this module in my opinion:

Please note that all code snippets in the post suppose that you have already imported the logging module:

import logging

For example, when a log “hello world” is sent through a log formatter:

"%(asctime)s — %(name)s — %(levelname)s — %(funcName)s:%(lineno)d — %(message)s"

it will become

2018-02-07 19:47:41,864 - a.b.c - WARNING - <module>:1 - hello world

Logger is probably the one that will be used directly the most often in the code and which is also the most complicated. A new logger can be obtained by:

toto_logger = logging.getLogger("toto")

A logger is unique by name, meaning that if a logger with the name “toto” has been created, the consequent calls of logging.getLogger("toto") will return the same object:

assert id(logging.getLogger("toto")) == id(logging.getLogger("toto"))

Suggestion : 4

Not all log messages are created equal. Logging levels are listed here in the Python documentation; we’ll include them here for reference. When you set a logging level in Python using the standard module, you’re telling the library you want to handle all events from that level on up. If you set the log level to INFO, it will include INFO, WARNING, ERROR, and CRITICAL messages. NOTSET and DEBUG messages will not be included here.,This article covers the basics of using the standard logging module that ships with all Python distributions. After reading this, you should be able to easily integrate logging into your Python application.,Logging in Python is simple and well-standardized, thanks to a powerful logging framework right in the standard library.,Python comes with a logging module in the standard library that provides a flexible framework for emitting log messages from Python programs. This module is widely used by libraries and is the first go-to point for most developers when it comes to logging.

To emit a log message, a caller first requests a named logger. The name can be used by the application to configure different rules for different loggers. This logger then can be used to emit simply-formatted messages at different log levels (DEBUG, INFO, ERROR, etc.), which again can be used by the application to handle messages of higher priority different than those of a lower priority. While it might sound complicated, it can be as simple as this:

import logging
log = logging.getLogger("my-logger")
log.info("Hello, world")

The only responsibility modules have is to make it easy for the application to route their log messages. For this reason, it is a convention for each module to simply use a logger named like the module itself. This makes it easy for the application to route different modules differently, while also keeping log code in the module simple. The module just needs two lines to set up logging, and then use the named logger:

import logging

log = logging.getLogger(__name__)

def do_something():
   log.debug("Doing something!")

In the case of running Python in containers like Docker, logging to standard output is also often the easiest move as this output can be directly and easily managed by the container itself.

import logging
import os

logging.basicConfig(level = os.environ.get("LOGLEVEL", "INFO"))

exit(main())

The alternative is to send it directly to syslog. This is great for older operating systems that don’t have systemd. In an ideal world this should be simple, but sadly, Python requires a bit more elaborate configuration to be able to send unicode log messages. Here is a sample implementation.

import logging
import logging.handlers
import os

class SyslogBOMFormatter(logging.Formatter):
   def format(self, record):
   result = super().format(record)
return "ufeff" + result

handler = logging.handlers.SysLogHandler('/dev/log')
formatter = SyslogBOMFormatter(logging.BASIC_FORMAT)
handler.setFormatter(formatter)
root = logging.getLogger()
root.setLevel(os.environ.get("LOGLEVEL", "INFO"))
root.addHandler(handler)

try:
exit(main())
except Exception:
   logging.exception("Exception in main()")
exit(1)

Here’s a sample implementation.

import logging
import logging.handlers
import os

handler = logging.handlers.WatchedFileHandler(
   os.environ.get("LOGFILE", "/var/log/yourapp.log"))
formatter = logging.Formatter(logging.BASIC_FORMAT)
handler.setFormatter(formatter)
root = logging.getLogger()
root.setLevel(os.environ.get("LOGLEVEL", "INFO"))
root.addHandler(handler)

try:
exit(main())
except Exception:
   logging.exception("Exception in main()")
exit(1)