How do I cache multiple AWS Parameter Store values in an AWS Lambda?

Question:

I want to limit the number of times that AWS Parameter Store is called in my AWS Lambda. Using a global variable, I’m caching a Parameter Store value on the first call to Parameter Store.

main.py

import os

import boto3


redis_password = None

def get_redis_password():
    global redis_password
    if not redis_password:
        client = boto3.client("ssm")
        redis_password = client.get_parameter(
            Name=f"{os.environ["ENV"]}.redis-cache.password",
            WithDecryption=True
        )
    return redis_password["Parameter"]["Value"]

def lambda_handler(event, context):
    get_redis_password()

However, if I want to cache multiple Parameter Store values, I must create multiple global variables and if not [INSERT_GLOBAL_VARIABLE] checks. For example:

main.py

import os

import boto3


redis_password = None
another_parameter_store_value = None

def get_redis_password():
    global redis_password
    if not redis_password:
        client = boto3.client("ssm")
        redis_password = client.get_parameter(
            Name=f"{os.environ["ENV"]}.redis-cache.password",
            WithDecryption=True
        )
    return redis_password["Parameter"]["Value"]

def get_another_parameter_store_value():
    global another_parameter_store_value
    if not another_parameter_store_value:
        client = boto3.client("ssm")
        another_parameter_store_value = client.get_parameter(
            Name=f"{os.environ["ENV"]}.another.parameter.store.key",
            WithDecryption=True
        )
    return redis_password["Parameter"]["Value"]

def lambda_handler(event, context):
    get_redis_password()
    get_another_parameter_store_value()

Attempted Solution

In an attempt to solve this issue, I’ve created a Parameter Store utility.

parameter_util.py

import os
import boto3


class ParameterUtil:
    def __init__(self):
        self.boto_client = boto3.client("ssm")

    def get_parameter(self, parameter_path):
        response = self.boto_client.get_parameter(
            Name=f"{os.environ['ENV']}.{parameter_path}", WithDecryption=True
        )

        return response["Parameter"]["Value"]

My theory is that by instantiating the AWS Boto client as an instance variable, it will cache the entire Boto client object. Then get_parameter will be called using the cached Boto client. For example:

main.py

import os

import boto3

from parameter_util import ParameterUtil


redis_password = None

def get_redis_password():
    global redis_password
    if not redis_password:
        client = boto3.client("ssm")
        redis_password = client.get_parameter(
            Name=f"{os.environ["ENV"]}.redis-cache.password",
            WithDecryption=True
        )
    return redis_password["Parameter"]["Value"]

def lambda_handler(event, context):
    param_util = ParameterUtil()
    param_util.get_parameter(".redis-cache.password")
    param_util.get_parameter(".another.parameter.store.key")

However, I’m not really sure if this solves the issue.

Questions

Does caching the Boto client result in only one call per parameter to the Parameter Store when get_parameter is called? Or am I optimizing in the wrong place?

Asked By: Ryan Payne

||
Source

Answers:

I like this approach. I might suggest abstracting it a little bit to something like this:

main.py

parameter_store_values = {}
client = boto3.client("ssm")

def lookup_function(key):
    global parameter_store_values
    global client
    if parameter_store_values.get(key) is None:
        value = client.get_parameter(
            Name=key,
            WithDecryption=True)["Parameter"]["Value"]
        parameter_store_values[key] = value
    return value

def lambda_handler(event, context):
    redis_password = lookup_function(f"{os.environ["ENV"]}.redis-cache.password")
    another_parameter_store_key = lookup_function(f"{os.environ["ENV"]}.another.parameter.store.key")
Answered By: williamsbdev

Your original code won’t work because param_util is a local variable that will go out of scope for every Lambda call.

You can use the built-in @functools.lru_cache to create a simple function that handles any parameter. It will cache the return values for you based on the input of the function (Python 3.2+).

Decorator to wrap a function with a memoizing callable that saves up to the maxsize most recent calls. It can save time when an expensive or I/O bound function is periodically called with the same arguments.

Example:

ssm_client = boto3.client("ssm")

@lru_cache(maxsize=None)
def get_param(name):
    return ssm_client.get_parameter(
        Name=f"{os.environ['ENV']}.{name}",
        WithDecryption=True
    )["Parameter"]["Value"]

def lambda_handler(event, context):
  redis_password = get_param("redis-cache.password")
  another_parameter_store_key = get_param("another.parameter.store.key")
Answered By: kichik