Why doesn't AWS Lambda Python logging show in CloudWatch?

General Discussions
, ,
1

Understanding AWS Lambda Python Logging Behavior

I’m trying to use AWS Lambda Python logging based on the AWS documentation, which shows the following snippet:

import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

def my_logging_handler(event, context):
    logger.info('got event{}'.format(event))
    logger.error('something went wrong')

This works well and logs to CloudWatch, but when I try a similar approach with the code below, it doesn’t show anything in CloudWatch:

import logging
logging.basicConfig(level=logging.INFO)
logging.info("Hello World!")

Both snippets use the root logger, but the second one doesn’t seem to log anything in the CloudWatch console. What could be the reason behind this difference?

2

So, based on my experience with AWS Lambda Python logging, one reason your logs might not be appearing in CloudWatch is that the Lambda runtime itself pre-configures a logging handler, which modifies the log message format and may even add extra metadata. The catch here is that this handler doesn’t set the log level, so without an explicit configuration, logs won’t show up unless you manually set the log level.

To fix this, you can set the log level for the root logger in your AWS Lambda Python logging configuration. Here’s how you do it:

import logging
logging.getLogger().setLevel(logging.INFO)

Also, if you want to ensure compatibility both in AWS Lambda and locally, I suggest checking if the logging handler is already configured. If it’s not, you can use basicConfig to set up the default handler like this:

if len(logging.getLogger().handlers) > 0:
    logging.getLogger().setLevel(logging.INFO)
else:
    logging.basicConfig(level=logging.INFO)

That way, your aws lambda python logging will work both in the cloud and locally without any issues.

3

Yeah, that makes sense. And to build on what @madhurima_sil mentioned, another approach I like is creating a custom logging handler for AWS Lambda Python logging. With this, you can control exactly how the log messages are formatted and ensure they get sent to CloudWatch properly.

Here’s a quick example of how to create a custom handler:

import logging

class LambdaLoggingHandler(logging.Handler):
    def emit(self, record):
        log_message = self.format(record)
        print(log_message)  # AWS Lambda automatically logs this to CloudWatch

# Set up the custom logging handler
logger = logging.getLogger()
logger.setLevel(logging.INFO)
handler = LambdaLoggingHandler()
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)

logger.info("Custom handler test")

This ensures your aws lambda python logging behaves exactly as you need, and it will be sent to CloudWatch just like you’d expect.

4

To add to what @netra.agarwal said, if you’re looking for something even more streamlined, you could use the loguru library. It simplifies the entire process for AWS Lambda Python logging and works perfectly both in Lambda and locally. The best part is that it automatically handles the setup for you, so you don’t have to worry about configuring everything manually.

Here’s how you can use loguru:

from loguru import logger
import sys

# Simple log setup
logger.remove()  # Removes the default logger
logger.add(sys.stderr, level="INFO")

logger.info("Hello, world!")

Loguru outputs to stderr, which is automatically captured by AWS Lambda and sent to CloudWatch. So with minimal configuration, your aws lambda python logging will work effortlessly.