Upload data to S3 bucket without saving it to a disk

Question

Ther is the boto3 method upload_file(Filename='path') that uses the Filename parameter to read a file from a disk and upload it to a bucket. Is it possible to upload data without saving it to a disk?

Asked By: Mykola Zotko

||

Source

Answer 1

Save text file:

obj = 'some string'
bucket = 'my-bucket'
key = 'prefix/filename.txt'

boto3.client('s3').put_object(Body=obj, Bucket=bucket, Key=key)

Save csv file from pandas dataframe:

df = my-dataframe
bucket = 'my-bucket'
key = 'prefix/filename.csv'

csv_buffer = io.StringIO()
df.to_csv(csv_buffer)
boto3.client('s3').put_object(Body=csv_buffer.getvalue(), Bucket=bucket, Key=key)

Answered By: Jonathan Leon

Answer 2

S3 connection with credential

import boto3
import pandas as pd
import io

s3 = boto3.resource('s3', endpoint_url='',
  aws_access_key_id = '',
  aws_secret_access_key = '')

Create dataframes and upload

df = pd.DataFrame({'x': [1, 2, 3], 'y': ['a', 'b', 'c']})

csv_buffer = io.StringIO()
df.to_csv(csv_buffer)
s3.Bucket('BUCKET_NAME').put_object(Body=csv_buffer.getvalue(),  Key='df.csv')

Answered By: Triloki Gupta

Upload data to S3 bucket without saving it to a disk

Question:

Answers: