[Enhancement] SSM get parameter taking too long to retrieve a parameter.

[luismospinam]

Hello, is it normal that a simple code to get a simple SSM parameter takes over 10 seconds?

  @Override
  public String handleRequest(Object o, Context context) {
    LambdaLogger logger = context.getLogger();

    Instant start = Instant.now();
    SSMProvider ssmProvider = ParamManager.getSsmProvider();
    String ssmParameter = ssmProvider.withDecryption().get("test-parameter");
    logger.log("ssm parameter: " + ssmParameter);
    Instant finish = Instant.now();
    logger.log("Total duration SSM: " + Duration.between(start, finish).toMillis());

    return ssmParameter;
 }

Similar code to pull a parameter from SSM takes less than a second in other languages using AWS SDK, for instance, NodeJs.

Is this a normal java behavior?

Notes:

• This issue is not related to the withDecryption() functionality, even without it the code takes the same amount of time.
• This is not related to Java cold start (using Lambda), a simple "Hello World" application takes less than a second to execute.

[dgomesbr]

Hello Luis, can you initialize the block bellow outside the handleRequest, and retest?

  public static SSMProvider ssmProvider = ParamManager.getSsmProvider();

  @Override
  public String handleRequest(Object o, Context context) {
    LambdaLogger logger = context.getLogger();

    Instant start = Instant.now();
    String ssmParameter = ssmProvider.withDecryption().get("test-parameter");
    logger.log("ssm parameter: " + ssmParameter);
    Instant finish = Instant.now();
    logger.log("Total duration SSM: " + Duration.between(start, finish).toMillis());

    return ssmParameter;
 }

[luismospinam]

Hello @dgomesbr , thanks for your quick reply.

Declaring the variable as static improved the performance, now it takes 4-5 seconds to fetch the SSM parameter (cold start, further requests takes less than a second).

thanks!

[dgomesbr]

@msailes Let's review the docs to push this practices on the docs

[heitorlessa]

Isn’t 4-5 seconds too high? Any other optimisation that can be shared with customers?

…

On Tue, 17 Nov 2020 at 17:24, Diego Magalhães ***@***.***> wrote: @msailes <https://github.com/msailes> Let's review the docs to push this practices on the docs — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#181 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBGMF75FS2WX7XY76OTSQKPUDANCNFSM4TU3IMDQ> .

[pankajagrawal16]

@msailes Let's review the docs to push this practices on the docs

I will have a look at this later tonight

[pankajagrawal16]

Isn’t 4-5 seconds too high?

Any other optimisation that can be shared with customers?

Indeed, I will have a look at this, validate and suggest if this can be optimised in some way.

[pankajagrawal16]

It turns out during initial fetch, most of the time taken is by SSMClient itself. Next remote invocations of SSMClient is quite fast.

Total time fetching via SSM Client is the time spent internally by SSMClient in AWS SDK.
Total duration powertools SSM is time which is taken by powertools client.

1st call to SDK client is surprisingly slow.

So nothing much is happening inside powertools, most of it is from SDK client itself.

[pankajagrawal16]

Diving a bit more into the observation above and following this discussion which is on the same lines.

There are few things which can be done:

• Just increasing functions memory from 512 MB to 2048 MB, Significantly reduces the initial fetch time. From ~3 seconds to less than 1 seconds. @luismospinam This can be a thing to try out.
• There are few other other options that you can take to further optimise the initial fetch time.

Library provides you a way to pass the customer SSM/Secret manager client. You can for now configure the customer client like this:

  private static final SSMProvider ssmProvider = ParamManager.getSsmProvider(SsmClient.builder()
            .httpClientBuilder(UrlConnectionHttpClient.builder())
            .credentialsProvider(EnvironmentVariableCredentialsProvider.create())
            .build());

I will create a release tomorrow to make this the default configuration and also update docs.

[pankajagrawal16]

@msailes Let's review the docs to push this practices on the docs

I will have a look at this later tonight

Just looked at the docs, and the example kind of already guides to initialize ssm/secret provider outside the handler method. Can we make this more explicit somehow ? 🤔

[arjunpunnam]

I think declaring clients static outside the handler reducing the init time applies to almost every SDK client. I have experienced this with SSM, Dynamo, SQS, and SNS clients.IMO this behavior needs to be more publicized and even better if it is enforced at the SDK level. At work, we kind of followed a factory approach where all expensive clients are initialized above the handler function and the approach has yielded good results. One important thing to remember is to make sure that the total initialization time doesn't exceed more than 10 seconds.

[pankajagrawal16]

@luismospinam Just wondering if you got chance to try out some of the options mentioned above ?