How to upload s3 using boto3

Question

I want to upload my logs to my bucket
I never been used python and boto3
This is my code

import os
import datetime as dt
import boto3

x = dt.datetime.now()
date = x.strftime("%Y%m%d")
bucket = 'mybucket'
dir_path = "/log"
s3 = boto3.client('s3')

def log():
    global dir_path
    for (dir_path, dir, files) in os.walk(dir_path):
        for file in files:
            if date in file:
                file_path = os.path.join(dir_path, file)
                print file_path

file_name = (log())
key = (log())
res = s3.upoad_file(file_name, bucket, key)

and this is result

log1
log2
log3
log4
Traceback *most recent call last):
File "test2.py", line 21, in <module>
  res = s3.upload_file(file_name, bucket, key)
File "home/user/.local/lib/python2.7/site-packages/boto3/s3/tranfer.py", line 273, in upload_file extra_args=ExtraArgs, callback=Callback)
File "home/user/.local/lib/python2.7/site-packages/boto3/s3/tranfer.py", line 273, in upload_file raise ValueError('Filename must be a string')
ValueError: Filename must be a string

I have 4 log files

please help me
how to fix it?

Asked By: neeson.lee

||

Source

Answer 1

Since you need to upload more than one file, and you
stated that the upload one log works, you could
do the following, which basically goes through the
directory list as per your original intention, and
then for each file item that satisfies that criteria
(date in file), it returns the filepath to the
calling loop.

import os
import datetime as dt
import boto3

x = dt.datetime.now()
date = x.strftime("%Y%m%d")
bucket = 'mybucket'
dir_path = "/log"
s3 = boto3.client('s3')

def log(in_path):
    for (dir_path, dir, files) in os.walk(in_path):
        for file in files:
            if date in file:
                yield os.path.join(dir_path, file)


for file_name in log(dir_path):
    res = s3.upload_file(file_name, bucket, file_name)

Please note that if you need to keep track of the results,
then you could make a change like so:

.
.
.

results = {}
for file_name in log(dir_path):
    results[file_name] = s3.upload_file(file_name, bucket, file_name)

Answered By: ewokx

Answer 2

It was simple. this is my final code thanks.

import os
import datetime as dt
import boto3
import socket

x = dt.datetime.now()
date = x.strftime("%Y%m%d")
bucket = 'mybucket'
dir_path = "/log"
s3 = boto3.client('s3')

def log(in_path):
    for (dir_path, dir, files) in os.walk(in_path):
        for file in files:
            if date in file:
                yield os.path.join(dir_path, file)


for file_name in log(dir_path):
    key = socket.gethostname() + '/' + file_name
    res = s3.upload_file(file_name, bucket, key)

Answered By: neeson.lee

How to upload s3 using boto3

Question:

Answers: