Python – Memory limit exceeded during Google App Engine deployment

Question:

I am creating a Python project and deploying it to Google App Engine.

When I use the deployed link in another project, I get the following error message in Google Cloud Logging:

Exceeded hard memory limit of 256 MB with 667 MB after servicing 0 requests total. Consider setting a larger instance class in app.yaml.

So, I looked at this and this link and here are the main points:

  • Instance Class Memory Limit CPU Limit Supported Scaling Types
    F1 (default) 256 MB 600 MHz automatic
    F2 512 MB 1.2 GHz automatic
    F4 1024 MB 2.4 GHz automatic
    F4_1G 2048 MB 2.4 GHz automatic
  • instance_class: F2

The error says the limit is 256 MB, but 667 MB is recorded. The memory limit for F1 and the memory limit for F2 are less than 667 MB. So I added instance_class: F2 to app.yaml and changed F2 to F4.

When I do the above, I get the following error in Google Cloud Logging:

Exceeded hard memory limit of 1024 MB with 1358 MB after servicing 0 requests total. Consider setting a larger instance class in app.yaml.

This is a bit strange since the recorded memory is from 667 MB to 1358 MB.

The memory limit of F4_1G is over 1358 MB, so I changed instance_class: F4 to instance_class: F4_1G. But it shows me the following error in Google Cloud Logging:

Exceeded hard memory limit of 2048 MB with 2194 MB after servicing 0 requests total. Consider setting a larger instance class in app.yaml.

This is very strange since the recorded memory goes from 667 MB to 1358 MB to 2194 MB.

Update:

I have reproduced this problem without additional instance class.

Please refer error log below:

0: {
logMessage: "Exceeded soft memory limit of 256 MB with 924 MB after servicing 0 requests total. Consider setting a larger instance class in app.yaml."
severity: "CRITICAL"
time: "2022-10-19T06:00:39.747954Z"
}
1: {
logMessage: "This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application."
severity: "INFO"
time: "2022-10-19T06:00:39.748029Z"
}
2: {
logMessage: "While handling this request, the process that handled this request was found to be using too much memory and was terminated. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may have a memory leak in your application or may be using an instance with insufficient memory. Consider setting a larger instance class in app.yaml."
severity: "WARNING"
time: "2022-10-19T06:00:39.748031Z"
}

Another finding:

When the app is running in local terminal, it consumes 1 GB – 3 GB memory to running the app fully loaded which takes around 30 seconds. Meanwhile, the memory usage is 700 MB – 750 MB during idle state, and 750 MB – 800 MB to serve single request.

Can anyone explain to me why this is happening? How can I fix this error and use the deployed link successfully? I would appreciate if someone could help me with this. Thank you in advance!

Asked By: My Car

||

Answers:

First, I store larger files from local to cloud. Then I successfully deployed the code to Google App Engine without any errors.

Answered By: My Car