Writing json to file in s3 bucket

Question

This code writes json to a file in s3,
what i wanted to achieve is instead of opening data.json file and writing to s3 (sample.json) file,

how do i pass the json directly and write to a file in s3 ?

import boto3

s3 = boto3.resource('s3', aws_access_key_id='aws_key', aws_secret_access_key='aws_sec_key')
s3.Object('mybucket', 'sample.json').put(Body=open('data.json', 'rb'))

Asked By: Learning

||

Source

Answer 1

Amazon S3 is an object store (File store in reality). The primary operations are PUT and GET. You can not add data into an existing object in S3. You can only replace the entire object itself.

For a list of available operations you can perform on s3 see this link
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectOps.html

Answered By: Usman Mutawakil

Answer 2

I’m not sure, if I get the question right. You just want to write JSON data to a file using Boto3? The following code writes a python dictionary to a JSON file.

import json
import boto3    
s3 = boto3.resource('s3')
s3object = s3.Object('your-bucket-name', 'your_file.json')

s3object.put(
    Body=(bytes(json.dumps(json_data).encode('UTF-8')))
)

Answered By: Uwe Bretschneider

Answer 3

I don’t know if anyone is still attempting to use this thread, but I was attempting to upload a JSON to s3 trying to use the method above, which didnt quite work for me. Boto and s3 might have changed since 2018, but this achieved the results for me:

import json
import boto3

s3 = boto3.client('s3')
json_object = 'your_json_object here'
s3.put_object(
     Body=json.dumps(json_object),
     Bucket='your_bucket_name',
     Key='your_key_here'
)

Answered By: Joseph McCombs

Answer 4

An alternative to Joseph McCombs answer can be achieved using s3fs.

from s3fs import S3FileSystem

json_object = {'test': 3.14}
path_to_s3_object = 's3://your-s3-bucket/your_json_filename.json'

s3 = S3FileSystem()
with s3.open(path_to_s3_object, 'w') as file:
    json.dump(json_object, file)

Answered By: Manuel Montoya

Writing json to file in s3 bucket

Question:

Answers: