AppEngine warning – OpenBLAS WARNING – could not determine the L2 cache size on this system
Question:
I try to deploy application on the GC AppEngine. There are no errors during deploy process but application doesn’t work (Just show loading page).
The only one strange raw in logs
OpenBLAS WARNING - could not determine the L2 cache size on this system
By the way – it works well on my local machine.
This is python web app based on Dash framework
My app.yaml:
runtime: python37
service: service-name
instance_class: F2
entrypoint: gunicorn -b :$PORT main:app.server
Requirements.txt:
Flask==1.0.2
dash==0.34.0
dash-html-components==0.13.4
dash-core-components==0.41.0
dash-table==3.1.11
gunicorn==19.9.0
google-cloud-pubsub==0.37.2
requests==2.21.0
pandas==0.23.4
Answers:
I just had your same problem with pandas and Dash and found your question (hoping it would give me some light). After being stuck for several hours, I found the answer, and came back to share 🙂
If the only error that you’re seeing is the OpenBLAS warning, most likely the app is working well. After debugging this problem for several hours, I found that as Dash and Pandas consume a lot of memory, the F2 instance is not able to handle the web app properly and fails due to lack of RAM memory. Please try changing in your YAML/JSON configuration file your instance to the highest possible automatic unit with more RAM memory, and then it will probably work:
instance_class: F4_HIGHMEM
EDIT: Google App Engine now supports more instance types. Check the docs of instance types: standard instances
In addition, please keep in mind that the first time you run this web app, it will take considerably more time to execute. If you check the logs you’ll have several prompts like the one below. Just wait a little bit more
This request caused a new process to be started for your application,
and thus caused your application code to be loaded for the first time.
This request may thus take longer and use more CPU than a typical
request for your application.
I personally solved it by adding a timeout to gunicorn, as the default timeout is only 30 sec
entrypoint: gunicorn -b :$PORT main:app.server --timeout 120
I found this solution as I tried the following:
- switched from a F1 to a F4_1G instance: still had the same warning
- switched from App Engine Standard to App Engine Flexible environment (which I highly DO NOT recommend, as app engine flexible instances not properly shut down (deleted) can cost you a lot of money: please see here for reference Pricing of Google App Engine Flexible env, a $500 lesson), with 16gb of ram and 4 CPU -> eventually got a different warning "[CRITICAL] WORKER TIMEOUT" which pointed me to this post: Gunicorn worker timeout error , which is where I found this solution.
Now my app works well, even with a F1 instance
Don’t forget to change the number of workers !
there is a default number of workers in AppEngine instances, so your application is multiplied by this number on the same instance
https://cloud.google.com/appengine/docs/standard/python3/runtime
I experienced the same error log using a AWS Lambda function. My function was processing a decent number of records so I needed to increase two settings in my lambda function configuration:
- Increase the size of Memory.
- Increase the Timeout in seconds.
Hope that helps.
I try to deploy application on the GC AppEngine. There are no errors during deploy process but application doesn’t work (Just show loading page).
The only one strange raw in logs
OpenBLAS WARNING - could not determine the L2 cache size on this system
By the way – it works well on my local machine.
This is python web app based on Dash framework
My app.yaml:
runtime: python37
service: service-name
instance_class: F2
entrypoint: gunicorn -b :$PORT main:app.server
Requirements.txt:
Flask==1.0.2
dash==0.34.0
dash-html-components==0.13.4
dash-core-components==0.41.0
dash-table==3.1.11
gunicorn==19.9.0
google-cloud-pubsub==0.37.2
requests==2.21.0
pandas==0.23.4
I just had your same problem with pandas and Dash and found your question (hoping it would give me some light). After being stuck for several hours, I found the answer, and came back to share 🙂
If the only error that you’re seeing is the OpenBLAS warning, most likely the app is working well. After debugging this problem for several hours, I found that as Dash and Pandas consume a lot of memory, the F2 instance is not able to handle the web app properly and fails due to lack of RAM memory. Please try changing in your YAML/JSON configuration file your instance to the highest possible automatic unit with more RAM memory, and then it will probably work:
instance_class: F4_HIGHMEM
EDIT: Google App Engine now supports more instance types. Check the docs of instance types: standard instances
In addition, please keep in mind that the first time you run this web app, it will take considerably more time to execute. If you check the logs you’ll have several prompts like the one below. Just wait a little bit more
This request caused a new process to be started for your application,
and thus caused your application code to be loaded for the first time.
This request may thus take longer and use more CPU than a typical
request for your application.
I personally solved it by adding a timeout to gunicorn, as the default timeout is only 30 sec
entrypoint: gunicorn -b :$PORT main:app.server --timeout 120
I found this solution as I tried the following:
- switched from a F1 to a F4_1G instance: still had the same warning
- switched from App Engine Standard to App Engine Flexible environment (which I highly DO NOT recommend, as app engine flexible instances not properly shut down (deleted) can cost you a lot of money: please see here for reference Pricing of Google App Engine Flexible env, a $500 lesson), with 16gb of ram and 4 CPU -> eventually got a different warning "[CRITICAL] WORKER TIMEOUT" which pointed me to this post: Gunicorn worker timeout error , which is where I found this solution.
Now my app works well, even with a F1 instance
Don’t forget to change the number of workers !
there is a default number of workers in AppEngine instances, so your application is multiplied by this number on the same instance
https://cloud.google.com/appengine/docs/standard/python3/runtime
I experienced the same error log using a AWS Lambda function. My function was processing a decent number of records so I needed to increase two settings in my lambda function configuration:
- Increase the size of Memory.
- Increase the Timeout in seconds.
Hope that helps.