Can't find kaggle.json file in google colab

Question

I’m trying to download the kaggle imagenet object localization challenge data into google colab so that I can use it to train my model. Kaggle uses an API for easy and fast access to their datasets. (https://github.com/Kaggle/kaggle-api) However, when calling the command “kaggle competitions download -c imagenet-object-localization-challenge” in google colab, it can’t find the kaggle.json file which contains my username and api-key.

I haven’t had this problem on my mac when running a jupyter notebook, but since I want to use google’s gpu for my model, I started using google colab. Because the kaggle API expects the username and api-key to be in a kaggle.json file located in a .kaggle directory, I first created the directory .kaggle and then the file kaggle.json, into which I wrote my username and api-key (The example below doesn’t display my username and api-key). I then tried to configure the path to my json file for kaggle to use when calling the kaggle download command.

!pip install kaggle

!mkdir .kaggle
!touch .kaggle/kaggle.json

api_token = {"username":"username","key":"api-key"}

import json
import zipfile
import os
with open('/content/.kaggle/kaggle.json', 'w') as file:
    json.dump(api_token, file)

!chmod 600 /content/.kaggle/kaggle.json
!kaggle config path -p /content

However, when running the last command, I got the error:

IOError: Could not find kaggle.json. Make sure it's located in /root/.kaggle. Or use the environment method.

My goal was to use the following commands to get the dataset from kaggle:

!kaggle competitions download -c imagenet-object-localization-challenge
os.chdir('/content/competitions/imagenet-object-localization-challenge')
for file in os.listdir():
    zip_ref = zipfile.ZipFile(file, 'r')
    zip_ref.extractall()
    zip_ref.close()

I don’t understand why the kaggle API can’t find my json file. How can I use the API in google colab?

Asked By: Diego Domenig

||

Source

Answer 1

As the error said, you need to put kaggle.json in the right place.

Try:

!mv .kaggle /root/

Then run your code again.

Answered By: korakot

Answer 2

According to kaggle api documentation the location where credentials json is looking for is ~/.kaggle/kaggle.json as google colab environment is Linux based.
In your snippet you try to config path parameter, but it is not used to looking for credential json:

– path: Folder where file(s) will be downloaded, defaults to current working directory

So the full working snippet for google colab environment would be:

!mkdir ~/.kaggle
!touch ~/.kaggle/kaggle.json

api_token = {"username":"username","key":"api-key"}

import json

with open('/root/.kaggle/kaggle.json', 'w') as file:
    json.dump(api_token, file)

!chmod 600 ~/.kaggle/kaggle.json

And then some api call like

!kaggle datasets download -d datamunge/sign-language-mnist

Answered By: Egor B Eremeev

Answer 3

In case you are a windows user, move your kaggle.json into .kaggle folder, that you are able to find it in your system root (which is in your username folder).

Answered By: pari

Answer 4

You can find the JSON file that you need to place form Kaggle’s portal itself. Just log into kaggle, go to your account page and then click ‘Create New API Token’ and it will give you a JSON file that needs to be placed.

Answered By: stonelazy

Answer 5

Run this code in cell to config the environment

import os
os.environ["KAGGLE_CONFIG_DIR"] = "/path_to_your_kaggle.json_file"

They’ve written their GitHub repo that you can config the environment of kaggle.json error appears in colab or windows too:

You can define a shell environment variable KAGGLE_CONFIG_DIR to change this location to $KAGGLE_CONFIG_DIR/kaggle.json (on Windows it will be %KAGGLE_CONFIG_DIR%kaggle.json).

—
Refer this

Answered By: PSP2912

Answer 6

My own method, and if you like to minimise ‘!’ addition to Egor B Eremeev answer above.

This may be beneficial depending on your use case and in the long run lead to cleaner code if you are scripting.

import json 
import os
from pathlib import Path

# your api key
api_key = {
'username':"username" ,
'key':"some_api_key"}

# uses pathlib Path
kaggle_path = Path('/root/.kaggle')
os.makedirs(kaggle_path, exist_ok=True)

# opens file and dumps python dict to json object 
with open (kaggle_path/'kaggle.json', 'w') as handl:
    json.dump(api_key,handl)

os.chmod(kaggle_path/'kaggle.json', 600)

Answered By: fdsig

Can't find kaggle.json file in google colab

Question:

Answers: