API Gateway Timeout error

[MounicaRamini]

Implementations: Spring Boot / AWS Lambda Function/API Gateway

Expected behavior
Should process the requests and give response in 30 seconds when lambda is invoked from API Gateway

Actual behavior
Timeout Error for the first call made from API Gateway and gives response for the calls after

How to avoid timeout error on API Gateway and bring response in <10 seconds when using java framework ? To match time frame of API Gateway, I have increased memory_size of my lambda function to 1472 MB, out of which only 368 MB is used when executed.

@sapessi Could you please give some input on this

[sapessi]

Hi @MounicaRamini, are you using Async initialization? Also, looking at the logs, do you have any indication of how long the spring boot application actually takes to start?

[MounicaRamini]

Yes, am using Async initialization, and from logs I see what's below. It took more than 10000 ms

16:13:27.941 [main] INFO com.amazonaws.serverless.proxy.AsyncInitializationWrapper - Async initialization will wait for 8863ms
16:13:27.943 [Thread-0] INFO com.amazonaws.serverless.proxy.AsyncInitializationWrapper - Starting async initializer
.
.
2019-10-16 16:13:36.805 INFO 1 --- [ main] c.a.s.proxy.AsyncInitializationWrapper : Initialization took longer than 10000, setting new CountDownLatch and continuing in event handler
.
.

2019-10-16 16:13:46.889 ERROR 1 --- [ main] c.a.s.p.internal.LambdaContainerHandler : Error while handling request

com.amazonaws.serverless.exceptions.ContainerInitializationException: Could not initialize framework within the 10000ms timeout
.
.
2019-10-16 16:13:46.889 ERROR 1 --- [ main] c.a.s.proxy.AwsProxyExceptionHandler : Called exception handler for:

com.amazonaws.serverless.exceptions.ContainerInitializationException: Could not initialize framework within the 10000ms timeout
.
.
at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:114)
com.amazonaws.serverless.exceptions.ContainerInitializationException: Could not initialize framework within the 10000ms timeout
.
.
 Duration: 10132.39 ms Billed Duration: 10200 ms Memory Size: 1728 MB Max Memory Used: 344 MB Init Duration: 9786.61 ms

[sapessi]

Looks like you are getting into this branch of the code. By default, the library first waits for 10 seconds for the framework to initialize during Lambda's init time. Then, if the framework initialization took longer than 10 seconds, it returns control to Lambda and waits in the request handling logic for a configurable timeout (config.getInitializationTimeout()). By default, this timeout is an additional 10 seconds. Sounds like your Spring Boot application is taking longer than 20 seconds to start.

You could work around the issue by increasing the initialization timeout. In your handler, before starting the async initialization, set the configuration property

public class LambdaHandler implements RequestStreamHandler {
    private SpringBootLambdaContainerHandler<AwsProxyRequest, AwsProxyResponse> handler;
    
    public LambdaHandler() {
        try {
            long startTime = Instant.now().toEpochMilli();
            // by setting this value to 20_000 the framework will wait for an additional 20 seconds
            // on top of the default 10 initialization time.
            LambdaContainerHandler.getContainerConfig().setInitializationTimeout(20_000);

            handler = new SpringBootProxyHandlerBuilder()
                    .defaultProxy()
                    .asyncInit(startTime)
                    .springBootApplication(SlowTestApplication.class)
                    .buildAndInitialize();
        } catch (ContainerInitializationException e) {
            e.printStackTrace();
        }
    }

    @Override
    public void handleRequest(InputStream input, OutputStream output, Context context) {
        return handler.proxyStream(input, output, context);
    }
}

This may solve your issue. However, I would recommend investigating why the application is taking so long to start and try to reduce that time. The Spring Boot guide includes some optimization tips.

[MounicaRamini]

Thanks, that resolved API Gateway timeout issue

[sapessi]

I'll keep this issue open. We should update the async initializer documentation to explain this process.

[MounicaRamini]

Looks like you are getting into this branch of the code. By default, the library first waits for 10 seconds for the framework to initialize during Lambda's init time. Then, if the framework initialization took longer than 10 seconds, it returns control to Lambda and waits in the request handling logic for a configurable timeout (config.getInitializationTimeout()). By default, this timeout is an additional 10 seconds. Sounds like your Spring Boot application is taking longer than 20 seconds to start.

You could work around the issue by increasing the initialization timeout. In your handler, before starting the async initialization, set the configuration property

public class LambdaHandler implements RequestStreamHandler {
    private SpringBootLambdaContainerHandler<AwsProxyRequest, AwsProxyResponse> handler;
    
    public LambdaHandler() {
        try {
            long startTime = Instant.now().toEpochMilli();
            // by setting this value to 20_000 the framework will wait for an additional 20 seconds
            // on top of the default 10 initialization time.
            LambdaContainerHandler.getContainerConfig().setInitializationTimeout(20_000);

            handler = new SpringBootProxyHandlerBuilder()
                    .defaultProxy()
                    .asyncInit(startTime)
                    .springBootApplication(SlowTestApplication.class)
                    .buildAndInitialize();
        } catch (ContainerInitializationException e) {
            e.printStackTrace();
        }
    }

    @Override
    public void handleRequest(InputStream input, OutputStream output, Context context) {
        return handler.proxyStream(input, output, context);
    }
}

This may solve your issue. However, I would recommend investigating why the application is taking so long to start and try to reduce that time. The Spring Boot guide includes some optimization tips.

Hello @sapessi , I have followed optimization tips and have removed Asynchronuos Initialization Lambda Handler code. I have implemented the below LambdaHandler code with the optimaization tips mentioned in optimization tips I still see that JVM takes 22-24 seconds or mostly 30 seconds for initialization and time outs when a request is sent from API Gateway or GUI.

Could you please help me with what can I do here to resolve this

LambdaHandler

public class LambdaHandler implements RequestStreamHandler {
	
	private static SpringBootLambdaContainerHandler<AwsProxyRequest, AwsProxyResponse> handler;
	
	@Override
	public void handleRequest(InputStream inputStream, OutputStream outputStream, Context context)
			throws IOException {
		if (handler == null) {
			try {
				handler = SpringBootLambdaContainerHandler.getAwsProxyHandler(Application.class);
			} catch (ContainerInitializationException e) {
				// if we fail here. We re-throw the exception to force another cold start
				e.printStackTrace();
				throw new RuntimeException("Could not initialize Spring Boot application", e);
			}
		}
		handler.proxyStream(inputStream, outputStream, context);
	}
}

[sapessi]

You should definitely used the asynchronous initialization handler from the static context or constructor (as per my initial example). Initializing the library from the handler method will cause it to be much slower.

[MounicaRamini]

You should definitely used the asynchronous initialization handler from the static context or constructor (as per my initial example). Initializing the library from the handler method will cause it to be much slower.

@sapessi Even if I use asynchronous initialization with timeout set to 20000 , I still get timeout error for the very first request. For second, third requests, responses are pretty quick.
If I increase that 20000 to 30000 , I still get same issues as mentioned above

Here are the logs :

19:23:00.241 [main] INFO com.amazonaws.serverless.proxy.AsyncInitializationWrapper - Async initialization will wait for 8747ms
19:23:00.243 [Thread-0] INFO com.amazonaws.serverless.proxy.AsyncInitializationWrapper - Starting async initializer
.
.
2019-11-07 19:23:08.997  INFO 1 --- [           main] c.a.s.proxy.AsyncInitializationWrapper   : Initialization took longer than 10000, setting new CountDownLatch and continuing in event handler
.
.
com.amazonaws.serverless.exceptions.ContainerInitializationException: Could not initialize framework within the 20000ms timeout
.
.
com.amazonaws.serverless.exceptions.ContainerInitializationException: Could not initialize framework within the 20000ms timeout
.
.
com.amazonaws.serverless.exceptions.ContainerInitializationException: Could not initialize framework within the 20000ms timeout

Second request process fast after the first one