logo
Splunk Team

amazon lambda does not show python logs

  • Last Update :
  • Techknowledgy :

As AWS documents themselves, to correctly use the logging library in the AWS Lambda context, you only need to set the log-level for the root-logger:

import logging
logging.getLogger().setLevel(logging.INFO)

If you want your Python-script to be both executable on AWS Lambda, but also with your local Python interpreter, you can check whether a handler is configured or not, and fall back to basicConfig (which creates the default stderr-handler) otherwise:

if len(logging.getLogger().handlers) > 0:
   # The Lambda environment pre - configures a handler logging to stderr.If a handler is already configured,
   # `.basicConfig`
does not execute.Thus we set the level directly.
logging.getLogger().setLevel(logging.INFO)
else:
   logging.basicConfig(level = logging.INFO)

Copied straight from the top answer in the question @StevenBohrer's answer links to (this did the trick for me, replacing the last line with my own config):

root = logging.getLogger()
if root.handlers:
   for handler in root.handlers:
   root.removeHandler(handler)
logging.basicConfig(format = '%(asctime)s %(message)s', level = logging.DEBUG)

I've struggled with this exact problem. The solution that works both locally and on AWS CloudWatch is to setup your logging like this:

import logging

# Initialize you log configuration using the base class
logging.basicConfig(level = logging.INFO)

# Retrieve the logger instance
logger = logging.getLogger()

# Log your output to the retrieved logger instance
logger.info("Python for the win!")

If python version 3.8 and above

import os
import logging

default_log_args = {
   "level": logging.DEBUG
   if os.environ.get("DEBUG", False)
   else logging.INFO,
   "format": "%(asctime)s [%(levelname)s] %(name)s - %(message)s",
   "datefmt": "%d-%b-%y %H:%M",
   "force": True,
}

logging.basicConfig( ** default_log_args)
log = logging.getLogger("Run-Lambda")

log.info("I m here too)

If python version 3.7 and below

import os
import logging

root = logging.getLogger()
if root.handlers:
   for handler in root.handlers:
   root.removeHandler(handler)

default_log_args = {
   "level": logging.DEBUG
   if os.environ.get("DEBUG", False)
   else logging.INFO,
   "format": "%(asctime)s [%(levelname)s] %(name)s - %(message)s",
   "datefmt": "%d-%b-%y %H:%M"
}

logging.basicConfig( ** default_log_args)
log = logging.getLogger("Run-Lambda")

log.info("Iam here")

I would suggest use aws python lambda powertools. The logging doc is here. Code example:

from aws_lambda_powertools
import Logger
logger = Logger() # Sets service via env
var
   # OR logger = Logger(service = "example")
    LOGGER = logging.getLogger()
    HANDLER = LOGGER.handlers[0]
    HANDLER.setFormatter(
       logging.Formatter(“[ % (asctime) s] % (levelname) s: % (name) s: % (message) s”, “ % Y - % m - % d % H: % M: % S”)
    )

Suggestion : 2

Each log stream corresponds to an instance of your function. A log stream appears when you update your Lambda function, and when additional instances are created to handle multiple concurrent invocations. To find logs for a specific invocation, we recommend instrumenting your function with AWS X-Ray. X-Ray records details about the request and the log stream in the trace.,Log groups aren't deleted automatically when you delete a function. To avoid storing logs indefinitely, delete the log group, or configure a retention period after which logs are deleted automatically.,AWS Lambda automatically monitors Lambda functions on your behalf and sends function metrics to Amazon CloudWatch. Your Lambda function comes with a CloudWatch Logs log group and a log stream for each instance of your function. The Lambda runtime environment sends details about each invocation to the log stream, and relays logs and other output from your function's code. ,You can use the Amazon CloudWatch console to view logs for all Lambda function invocations.

import os

def lambda_handler(event, context):
   print('## ENVIRONMENT VARIABLES')
print(os.environ)
print('## EVENT')
print(event)
START RequestId: 8 f507cfc - xmpl - 4697 - b07a - ac58fc914c95 Version: $LATEST
# # ENVIRONMENT VARIABLES
environ({
   'AWS_LAMBDA_LOG_GROUP_NAME': '/aws/lambda/my-function',
   'AWS_LAMBDA_LOG_STREAM_NAME': '2020/01/31/[$LATEST]3893xmpl7fac4485b47bb75b671a283c',
   'AWS_LAMBDA_FUNCTION_NAME': 'my-function',
   ...
})
# # EVENT {
   'key': 'value'
}
END RequestId: 8 f507cfc - xmpl - 4697 - b07a - ac58fc914c95
REPORT RequestId: 8 f507cfc - xmpl - 4697 - b07a - ac58fc914c95 Duration: 15.74 ms Billed Duration: 16 ms Memory Size: 128 MB Max Memory Used: 56 MB Init Duration: 130.49 ms
XRAY TraceId: 1 - 5e34 a614 - 10 bdxmplf1fb44f07bc535a1 SegmentId: 07 f5xmpl2d1f6f85 Sampled: true

The following example shows how to retrieve a log ID from the LogResult field for a function named my-function.

aws lambda invoke-- function -name my - function out--log - type Tail

You should see the following output:

{
   "StatusCode": 200,
   "LogResult": "U1RBUlQgUmVxdWVzdElkOiA4N2QwNDRiOC1mMTU0LTExZTgtOGNkYS0yOTc0YzVlNGZiMjEgVmVyc2lvb...",
   "ExecutedVersion": "$LATEST"
}

In the same command prompt, use the base64 utility to decode the logs. The following example shows how to retrieve base64-encoded logs for my-function.

aws lambda invoke-- function -name my - function out--log - type Tail\
   --query 'LogResult'--output text | base64 - d

You should see the following output:

START RequestId: 57 f231fb - 1730 - 4395 - 85 cb - 4 f71bd2b87b8 Version: $LATEST "AWS_SESSION_TOKEN": "AgoJb3JpZ2luX2VjELj...", "_X_AMZN_TRACE_ID": "Root=1-5d02e5ca-f5792818b6fe8368e5b51d50;Parent=191db58857df8395;Sampled=0"
",ask/lib:/opt/lib",
END RequestId: 57 f231fb - 1730 - 4395 - 85 cb - 4 f71bd2b87b8
REPORT RequestId: 57 f231fb - 1730 - 4395 - 85 cb - 4 f71bd2b87b8 Duration: 79.67 ms Billed Duration: 80 ms Memory Size: 128 MB Max Memory Used: 73 MB

The cli-binary-format option is required if you are using AWS CLI version 2. You can also configure this option in your AWS CLI config file.

#!/bin/bash

aws lambda invoke-- function -name my - function --cli - binary - format raw - in - base64 - out--payload '{"key": "value"}'
out
sed - i '' - e 's/"//g'
out
sleep 15
aws logs get - log - events--log - group - name / aws / lambda / my - function --log - stream - name stream1--limit 5

In the same command prompt, macOS and Linux users may need to run the following command to ensure the script is executable.

chmod - R 755 get - logs.sh

Suggestion : 3

A Lambda function inherently comes with a CloudWatch Logs log group and each instance of your function has a log stream. When a function is invoked, the runtime (Python, Java, etc..) sends details about each invocation to the log stream. It relays logs and other output from the function’s code.,There are various ways you can view logs for an AWS Lambda Python function – in the Lambda console, in the CloudWatch Logs console, or from the command line.,Init Duration – This provides information for the first request served. The amount of time the runtime took to load the function and run code outside of the handler method.,To use the formatter, import it and initiate an instance in each of your Lambda functions, and use it to log events, like this:

Below is a format for a log example:

START RequestId: 7 f231cfc - xmpl - 2647 - b01a - bc68ec115c16 Version: 1.2
# # ENVIRONMENT VARIABLES
environ({
   'AWS_LAMBDA_LOG_GROUP_NAME': '/aws/lambda/log-function',
   'AWS_LAMBDA_LOG_STREAM_NAME': '2020/09/25/[1.2]3893xmpl7fac4513b41bb42b621a213c',
   'AWS_LAMBDA_FUNCTION_NAME': 'log-function',
   ...
})
# # EVENT {
   'key': 'value'
}
END RequestId: 7 f231cfc - xmpl - 2647 - b01a - bc68ec115c16
REPORT RequestId: 87 f231cfc - xmpl - 2647 - b01a - bc68ec115c16 Duration: 10.14 ms Billed Duration: 90 ms Memory Size: 128 MB Max Memory Used: 66 MB Init Duration: 120.19 ms
XRAY TraceId: 1 - 3e54 a324 - 10 brweplf1fb22f01bc555a1 SegmentId: 01 f1xmpl2d1f3f63 Sampled: true

Below is a basic example of usage of logging library in Python:

import os
import logging
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)

def lambda_handler(event, context):
   logger.info('## ENVIRONMENT VARIABLES')
logger.info(os.environ)
logger.debug('## EVENT')
logger.debug(event)

To use this library, you need to apply this code:

import aws_lambda_logging

def handler(event, context):
   aws_lambda_logging.setup(level = 'DEBUG')

Let’s take a simple example of AWS Lambda logging and see what will be the output:

import aws_lambda_logging

def handler(event, context):
   aws_lambda_logging.setup(level = 'DEBUG', aws_request_id = context.get('aws_request_id'))
log.debug('first example')
log.debug('{"Inputs": [“param1”,”param2”,”param3”]}')

The output will be in a JSON formatted message:

{
   "level": "DEBUG",
   "timestamp": "2020-09-25 11:37:22,428",
   "apigw_request_id": "513fde81-536d-12e1-c7ed-1d374ea70152",
   "location": "root.handler:6",
   "message": {
      "first example",
      "Inputs": [
         "param1",
         "param2",
         "param3"
      ]
   }
}

Suggestion : 4

Logger propagates a few formatting configurations to the built-in LambdaPowertoolsFormatter logging formatter.,Prefer using datetime string formats? Set use_datetime_directive at Logger constructor or at Lambda Powertools Formatter.,By default, Logger uses LambdaPowertoolsFormatter that persists its custom structure between non-cold start invocations. There could be scenarios where the existing feature set isn't sufficient to your formatting needs.,The Python standard library log records contains a large set of atttributes, however only a few are included in Powertools Logger log record by default.

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless - 2016 - 10 - 31
Description: AWS Lambda Powertools Tracer doc examples

Globals:
   Function:
   Timeout: 5
Runtime: python3 .9
Tracing: Active
Environment:
   Variables:
   POWERTOOLS_SERVICE_NAME: payment
LOG_LEVEL: INFO
Layers:
   # Find the latest Layer version in the official documentation
# https: //awslabs.github.io/aws-lambda-powertools-python/latest/#lambda-layer
   -!Sub arn: aws: lambda: $ {
      AWS::Region
   }: 017000801446: layer: AWSLambdaPowertoolsPython: 21

Resources:
   LoggerLambdaHandlerExample:
   Type: AWS::Serverless::Function
Properties:
   CodeUri: .. / src
Handler: inject_lambda_context.handler
 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
from aws_lambda_powertools
import Logger
from aws_lambda_powertools.utilities.typing
import LambdaContext

logger = Logger()

@logger.inject_lambda_context
def handler(event: dict, context: LambdaContext) - > str:
   logger.info("Collecting payment")

# You can log entire objects too
logger.info({
   "operation": "collect_payment",
   "charge_id": event["charge_id"]
})
return "hello world"
 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
[{
      "level": "INFO",
      "location": "collect.handler:9",
      "message": "Collecting payment",
      "timestamp": "2021-05-03 11:47:12,494+0200",
      "service": "payment",
      "cold_start": true,
      "lambda_function_name": "test",
      "lambda_function_memory_size": 128,
      "lambda_function_arn": "arn:aws:lambda:eu-west-1:12345678910:function:test",
      "lambda_request_id": "52fdfc07-2182-154f-163f-5f0f9a621d72"
   },
   {
      "level": "INFO",
      "location": "collect.handler:12",
      "message": {
         "operation": "collect_payment",
         "charge_id": "ch_AZFlk2345C0"
      },
      "timestamp": "2021-05-03 11:47:12,494+0200",
      "service": "payment",
      "cold_start": true,
      "lambda_function_name": "test",
      "lambda_function_memory_size": 128,
      "lambda_function_arn": "arn:aws:lambda:eu-west-1:12345678910:function:test",
      "lambda_request_id": "52fdfc07-2182-154f-163f-5f0f9a621d72"
   }
]

Suggestion : 5

Still works. I updated the gist to make it a self-contained Python 3.6 function that you can copy & paste in to the AWS Lambda Console in a web browser. Expected output is shown at the bottom. The string "This is a test log statement" should be present in the output. All other output details will differ.,This would prevent anything being logged using logger from propagating up to parent loggers. In the Lambda case the logger that Lambda adds.,Obviously this comes with the risk of assuming AWS do not change the runtime bootstrapping code (tested on python3.8 runtime).,Or... use the aws-lambda-powertools logger which has a lot of this, structured logs, and you get things like setting correlation ids.

In order to preserve the original behavior and keep tracebacks grouped together in one message, you can just update the format of the existing handler like:

FORMAT = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
logger = logging.getLogger()
for h in logger.handlers:
   h.setFormatter(logging.Formatter(FORMAT))
logger.setLevel(logging.INFO)
"[%(levelname)s]\t%(asctime)s.%(msecs)dZ\t%(aws_request_id)s\t%(message)s\n"
"%Y-%m-%dT%H:%M:%S"
logger = logging.getLogger()
return [{
      'fmt': handler.formatter._fmt,
      'datefmt': handler.formatter.datefmt
   }
   for handler in logger.handlers
]

It's accessible through second handler parameter usually named as the context:

def handler(event, context):
   aws_request_id = context['aws_request_id']

A bit inefficient but this worked for me:

log = logging.getLogger()
for h in log.handlers:
   h.setFormatter(logging.Formatter("%(aws_request_id)s [%(levelname)s] %(message)s"))

log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)

Suggestion : 6

2022-01-22T22:41:29-05:00,2022-01-22T00:00:00-05:00, Posted on January 22, 2022 3 minute read

1._
1
2
3
4
5
6
logging.basicConfig(
   format = "%(message)s",
   stream = sys.stdout,
   level = logging.DEBUG,
   force = True,
)
2._
1
2
3
4
5
6
3._
    logging.basicConfig(
       format = "%(message)s",
       stream = sys.stdout,
       level = logging.DEBUG,
       force = True,
    )
1
2
3
4
5
6
    logging.basicConfig(
       format = "%(message)s",
       stream = sys.stdout,
       level = logging.DEBUG,
       force = True,
    )
1._
1
2
3
4
def handler(event, context):
   setup_logging(context)
log = structlog.get_logger()
log.info("STARTED", httpapi_event = event)
2._
1
2
3
4
3._
def handler(event, context):
   setup_logging(context)
log = structlog.get_logger()
log.info("STARTED", httpapi_event = event)
1
2
3
4
def handler(event, context):
   setup_logging(context)
log = structlog.get_logger()
log.info("STARTED", httpapi_event = event)

Suggestion : 7

Amazon Lambda Does Not Show Python Logs,  › Python how do classes work ,  › Why does subclassing in python slow things down so much , Tags: amazon-web-services , python , aws-lambda , serverless , aws-serverless Answers: 1 | Viewed 4,640 times 


START RequestId: 62341 bgd - 6231 - 11e8 - 8 c5b - 25793532 a32u Version: $LATEST END RequestId: 62341 b0d - 6231 - 1128 - 8 r5b - 2 b793032a3ed REPORT RequestId: 6234 te0b - 6 rte - aaa8 - au5a - 21 t93132r3rt Duration: 0.46 ms

START RequestId: 62341 bgd - 6231 - 11e8 - 8 c5b - 25793532 a32u Version: $LATEST END RequestId: 62341 b0d - 6231 - 1128 - 8 r5b - 2 b793032a3ed REPORT RequestId: 6234 te0b - 6 rte - aaa8 - au5a - 21 t93132r3rt Duration: 0.46 ms


Trending Technology


Most popular in python