AWS Lambda CPU and Memory Profiling (Python)

Profiling cloud applications

Performance profiling is essential for optimizing and fixing application’s resource consumption, response time and failures. A performance issue without an execution profile is like an error without a stack trace. It will lead to a lot of manual work to get to the root cause.

Profiling cloud applications and functions such as AWS Lambda require special profiling tools designed for the cloud production environments, since it is quite unrealistic to simulate cloud environment with all its data, traffic and configuration locally.

Adding StackImpact profiler agent to the Lambda function

StackImpact cloud profiler was specifically designed for production environments. Unlike traditional profilers, which usually only run locally, StackImpact profiler runs inside of the cloud applications and completely automates the burdensome process of profiling of CPU, memory allocations, and other aspects of the application. Additionally it reports various health metrics and errors.

The following simple AWS Lambda function simulates some CPU work and a memory leak. Adding the StackImpact agent is only one statement. Make sure you install the StackImpact Python package with pip install stackimpact locally before bundling the lambda package.

import stackimpact
import random

stackimpact.start(
    agent_key = 'your agent key',
    app_name = 'LambdaDemoPython',
    app_environment = 'prod',
    block_profiler_disabled = True)

def simulate_cpu_work():
    for j in range(0, 100000):
        random.randint(1, 1000000)

mem1 = []
def simulate_mem_leak():
   for i in range(0, 1000):
        obj1 = {'v': random.randint(0, 1000000)}
        mem1.append(obj1)

def handler(event, context):
    simulate_cpu_work()
    simulate_mem_leak()
    
    response = {
        "statusCode": 200,
        "body": 'Done'
    }
    return response

You can get an agent key by signing up for a free account. The agent can also be started inside of the handler if necessary, it will only start once per process. Blocking call profiler needs to be disabled for now because of some signal conflicts with AWS Lambda environments.

Locating CPU hot spots and memory leaks

When constantly generating requests against this lambda function, the CPU hot spots can be located in the reported profiles.

The memory allocation rate for function calls can be found in the Hot spots section as well. Using these rates we can locate where exactly is the most of the memory allocated, which is not immediately released. Please note that allocation profiling is only possible since Python 3.

Continuous performance profiling

One of the important benefits of continuously profiling application is that profiles can be historically analysed and compared. Unlike one-time call graphs, a call graph history per process and application allow for a much deeper understanding of application execution. For example, a root cause of the performance regression in a new version of the function can be easily identified.

Since there can be many instances of Lambda containers, not every one of them will have active profiling agents, but only a small subset (adjustable from Dashboard).

See full documentation for more details.