Error No file found while converting file present in s3 using boto3

Question

I have a csv files present in s3
I am able to print bucket name, key and file content
But while converting to json I am getting Error no file found
There is only one csv file which is of 1 KB

code is below

import boto3
import csv
import json

# set up S3 connection
s3 = boto3.resource('s3')

my_bucket = s3.Bucket('csvfolder3')
# bucket_name = 'csvfolder3'

def lambda_handler(event, context):

    for file in my_bucket.objects.all():
        print(file.key)
        body = file.get()['Body'].read()
        print(body)
        with open(file.key, 'r') as csvfile: #####[ERROR] FileNotFoundError: [Errno 2] No such file or directory: 'data.csv'

            data = [row for row in csv.DictReader(csvfile)]
            json_data = json.dumps(data)
        s3.Object(my_bucket, 'file.json').put(Body=json_data)

Asked By: sim

||

Source

Answer 1

When you have:

body = file.get()['Body'].read()

This is already downloading the file (or object) contents in memory, within the AWS Lambda function.

N.B: keep an eye on the memory (MB) consumed. Increase memory size for the Lambda function as needed.

Further, I am assuming – based on the docs – that ['Body'].read() returns a bytes object.

So if you have bytes, you can then turn it into a string with:

bytes_data.decode()

Then create a file-like object as csv.DictReader expects, by wrapping the above with:

StringIO(...)

Full example:

from __future__ import annotations

import csv
import json
from io import StringIO

byte_content = b"""
id,name,age,height,weight
1,Alice,20,62,120.6
2,Freddie,21,74,190.6
3,Bob,17,68,120.0
""".strip()

# set encoding if known
encoding = 'utf-8'

# decode bytes -> str
content = byte_content.decode(encoding)

# create file-like object
file = StringIO(content)

data: list[dict] = [row for row in csv.DictReader(file, delimiter=",")]

json_data: str = json.dumps(data, indent=2)
print(json_data)

Result:

[
  {
    "id": "1",
    "name": "Alice",
    "age": "20",
    "height": "62",
    "weight": "120.6"
  },
  {
    "id": "2",
    "name": "Freddie",
    "age": "21",
    "height": "74",
    "weight": "190.6"
  },
  {
    "id": "3",
    "name": "Bob",
    "age": "17",
    "height": "68",
    "weight": "120.0"
  }
]

Answered By: rv.kvetch

Answer 2

like @rv.kvetch mentioned there are two way to convert file,

first method is already done by him

adding the second method

import boto3
import csv
import json
import os

# set up S3 connection
s3 = boto3.client('s3')

def lambda_handler(event, context):
    # get bucket name and file name from event
    bucket_name = event['Records'][0]['s3']['bucket']['name']
    file_name = event['Records'][0]['s3']['object']['key']
    
    # download CSV file from S3
    csv_file_path = f'/tmp/{file_name}'
    s3.download_file(bucket_name, file_name, csv_file_path)
    
    # convert CSV data to JSON format
    with open(csv_file_path, 'r') as csvfile:
        data = [row for row in csv.DictReader(csvfile)]
        json_data = json.dumps(data)

    # upload JSON data to S3
    json_file_path = f'{os.path.splitext(file_name)[0]}.json'
    s3.put_object(Body=json_data, Bucket=bucket_name, Key=json_file_path)

    return {
        'statusCode': 200,
        'body': json.dumps('CSV to JSON conversion successful!')
    }

Answered By: sim

Error No file found while converting file present in s3 using boto3

Question:

Answers: