gcloud app deploy : This deployment has too many files
Question:
I got the below error, when I tried to deploy my GAE app through gcloud.
Updating service [default]...failed.
ERROR: (gcloud.app.deploy) Error Response: [400] This deployment has too many files. New versions are limited to 10000 files for this app.
Details: [
[
{
"@type": "type.googleapis.com/google.rpc.BadRequest",
"fieldViolations": [
{
"description": "This deployment has too many files. New versions are limited to 10000 files for this app.",
"field": "version.deployment.files[...]"
}
]
}
]
]
Is there any way to tackle this problem?
Answers:
If you really have more than the 10000 files quota in the service you’re trying to deploy then you might have to reduce the number accordingly.
Other things to try:
- you might be able to get a quota increase, see Getting error on GAE: Max number of files and blobs is 10000
- delete whatever files are not actually needed, or just skip them during deployment see skip_files or, for the more recent cloud SDK versions, the
.gcloudignore
file.
- if you have a lot of static files consider moving (some of) them to GCS instead, see Approaches for overcoming 10000 file limit on Google App Engine?
- split the service into multiple smaller services – each with its own 10000 files limit.
Assuming you do not actually hit the files quota then the error usually indicates you have looping/circular referencing symlinks in your app directory. Which could also explain a path like the one you mentioned in a comment to this post: https://stackoverflow.com/a/42425048/4495081. You just have to fix the offending symlink(s). Again, a simple/consistent directory structure could help prevent such issues.
gcloud app deploy
writes a log file, and tells you where that log is early in its output. Examine that log. It’ll tell you what’s being uploaded.
Two common ways I’ve seen people get into trouble are
- Using virtualenv, but not adding
venv
(or .venv
, if that’s the name you picked) to skip_files
.
- Using git, but forgetting to add
.git
to skip_files
For python runtime 3.7, this error is solved by adding these files to a .gcloudingore not to skip_files in the app.yaml
I was able to reduce my uploaded files by adding the google cloud sdk folder to .gcloudignore in the project root folder.
Few things:
- Sometimes the
static
folder can get a bit messy. Try deleting it and rerunning python manage.py collectstatic
, this cut down about 2000 files for me.
- Make sure your
.gcloudignore
file ignores the assets
folder, given they’ve just been moved to static
.
- Ignore virtualenv folders, they’re big
Maybe your need ignore some files in file .gcloudignore
/vendor/
/node_modules/
/.git/
This work for me!
As mentioned in some comment, virtualenv folder can have a lot of files.
I just added venv/
as a new line into .gcloudignore
file.
I made a request to Google Cloud support, and this is their official response to this issue:
Thank you for contacting Google Cloud Support. I understand you’re
unable to deploy your app since you’re hitting a 10,000 file limit
upload as indicated in the error message, let me know if I
misunderstood.
GAE is unable to perform deployments with over 10,000 files. Please
run the following command to see the number of files being deployed:
find . -type f | wc -l
. In case you see that the number is above
10,000; make sure to prevent unnecessary files from being deployed by
using the app.yaml
option skip_files
[1]. This element helps you to
indicate which files should not be uploaded [1].
If the error persists, there are two other ways to solve this problem:
-
If the files that you are using are static content, use Cloud Storage bucket to store your files. You can store them in a GCS bucket
and serve them through App Engine [2]. When you create an app, App
Engine creates a default bucket. To use it you will have to list the
bucket in your project [3] and then declare Cloud Storage as a
dependency by adding it to the app’s dependency files [4]. This will
allow you to upload or download data from the buckets [5][6]. Please
consider that using this alternative may impact your monthly expenses.
-
Increase the number of uploaded files by submitting a quota increase. To do so, I will require the following information which
will be submitted to the product team so they can review it and make a
decision on it: A business justification for this quota increase The
new desired limit.
- https://cloud.google.com/appengine/docs/standard/python/config/appref#skip_files
- https://cloud.google.com/appengine/docs/standard/java/using-cloud-storage
- https://cloud.google.com/storage/docs/listing-buckets
- https://cloud.google.com/storage/docs/reference/libraries#installing_the_client_library
- https://cloud.google.com/storage/docs/uploading-objects
- https://cloud.google.com/storage/docs/downloading-objects
I had the same issue. Using .gcouldignore helped reduce from 15000 plus to 30.
node_modules/
.gitignore
.git
Like other answers, my problem was that gcloud app deploy
does not ignore virtual envs or node modules. The following helped me figure it out:
-
gcloud meta list-files-for-upload
will show you the list of files that will be uploaded, without having to go through the deploy process.
-
Add the line:
#!include:.gitignore
to your .gcloudignore
file, unless there are git-ignored files you want to upload. You can’t add .gitignore
files from other directories, but this can get you a lot of the way there.
-
Add additional entries to .gcloudignore
as needed until the list of files for upload is manageable.
I got the below error, when I tried to deploy my GAE app through gcloud.
Updating service [default]...failed.
ERROR: (gcloud.app.deploy) Error Response: [400] This deployment has too many files. New versions are limited to 10000 files for this app.
Details: [
[
{
"@type": "type.googleapis.com/google.rpc.BadRequest",
"fieldViolations": [
{
"description": "This deployment has too many files. New versions are limited to 10000 files for this app.",
"field": "version.deployment.files[...]"
}
]
}
]
]
Is there any way to tackle this problem?
If you really have more than the 10000 files quota in the service you’re trying to deploy then you might have to reduce the number accordingly.
Other things to try:
- you might be able to get a quota increase, see Getting error on GAE: Max number of files and blobs is 10000
- delete whatever files are not actually needed, or just skip them during deployment see skip_files or, for the more recent cloud SDK versions, the
.gcloudignore
file. - if you have a lot of static files consider moving (some of) them to GCS instead, see Approaches for overcoming 10000 file limit on Google App Engine?
- split the service into multiple smaller services – each with its own 10000 files limit.
Assuming you do not actually hit the files quota then the error usually indicates you have looping/circular referencing symlinks in your app directory. Which could also explain a path like the one you mentioned in a comment to this post: https://stackoverflow.com/a/42425048/4495081. You just have to fix the offending symlink(s). Again, a simple/consistent directory structure could help prevent such issues.
gcloud app deploy
writes a log file, and tells you where that log is early in its output. Examine that log. It’ll tell you what’s being uploaded.
Two common ways I’ve seen people get into trouble are
- Using virtualenv, but not adding
venv
(or.venv
, if that’s the name you picked) toskip_files
. - Using git, but forgetting to add
.git
toskip_files
For python runtime 3.7, this error is solved by adding these files to a .gcloudingore not to skip_files in the app.yaml
I was able to reduce my uploaded files by adding the google cloud sdk folder to .gcloudignore in the project root folder.
Few things:
- Sometimes the
static
folder can get a bit messy. Try deleting it and rerunningpython manage.py collectstatic
, this cut down about 2000 files for me. - Make sure your
.gcloudignore
file ignores theassets
folder, given they’ve just been moved tostatic
. - Ignore virtualenv folders, they’re big
Maybe your need ignore some files in file .gcloudignore
/vendor/
/node_modules/
/.git/
This work for me!
As mentioned in some comment, virtualenv folder can have a lot of files.
I just added venv/
as a new line into .gcloudignore
file.
I made a request to Google Cloud support, and this is their official response to this issue:
Thank you for contacting Google Cloud Support. I understand you’re
unable to deploy your app since you’re hitting a 10,000 file limit
upload as indicated in the error message, let me know if I
misunderstood.GAE is unable to perform deployments with over 10,000 files. Please
run the following command to see the number of files being deployed:
find . -type f | wc -l
. In case you see that the number is above
10,000; make sure to prevent unnecessary files from being deployed by
using theapp.yaml
optionskip_files
[1]. This element helps you to
indicate which files should not be uploaded [1].If the error persists, there are two other ways to solve this problem:
If the files that you are using are static content, use Cloud Storage bucket to store your files. You can store them in a GCS bucket
and serve them through App Engine [2]. When you create an app, App
Engine creates a default bucket. To use it you will have to list the
bucket in your project [3] and then declare Cloud Storage as a
dependency by adding it to the app’s dependency files [4]. This will
allow you to upload or download data from the buckets [5][6]. Please
consider that using this alternative may impact your monthly expenses.Increase the number of uploaded files by submitting a quota increase. To do so, I will require the following information which
will be submitted to the product team so they can review it and make a
decision on it: A business justification for this quota increase The
new desired limit.
- https://cloud.google.com/appengine/docs/standard/python/config/appref#skip_files
- https://cloud.google.com/appengine/docs/standard/java/using-cloud-storage
- https://cloud.google.com/storage/docs/listing-buckets
- https://cloud.google.com/storage/docs/reference/libraries#installing_the_client_library
- https://cloud.google.com/storage/docs/uploading-objects
- https://cloud.google.com/storage/docs/downloading-objects
I had the same issue. Using .gcouldignore helped reduce from 15000 plus to 30.
node_modules/
.gitignore
.git
Like other answers, my problem was that gcloud app deploy
does not ignore virtual envs or node modules. The following helped me figure it out:
-
gcloud meta list-files-for-upload
will show you the list of files that will be uploaded, without having to go through the deploy process. -
Add the line:
#!include:.gitignore
to your
.gcloudignore
file, unless there are git-ignored files you want to upload. You can’t add.gitignore
files from other directories, but this can get you a lot of the way there. -
Add additional entries to
.gcloudignore
as needed until the list of files for upload is manageable.