MIDIutil write MIDI File to boto3 API server via Flask [Python]

Question

struggling with trying to write a MIDIUtil file in my Flask App connecting to an s3 server.

In a local instance, it’s no sweat:

LOCAL_UPLOAD_FOLDER = './_static/uploads/MIDI_files/'
file_name = "NAME.mid"
file_path = f'{LOCAL_UPLOAD_FOLDER}{file_name}'
MyMIDI = MIDIFile(1)

with open(file_path, "wb") as output_file:
     MyMIDI.writeFile(output_file)

However, I’m not sure how to apply this to an s3 resource, here’s my instantiations…

def get_upload_folder(UPLOAD_FOLDER=None, UPLOAD_FOLDER_KEY=None,
                      client_resource=None, client=None):
    """ Determines How to Upload / Send File for Download """
    # Flask Cloud Settings - Upload Folder
    if os.getenv('CONTEXT') == 'CLOUD':

        # Client Side
        UPLOAD_FOLDER_TYPE = 'CLOUD'
        session = boto3.session.Session()
        client = session.client(
            's3', endpoint_url=os.getenv('ENDPOINT_URL'),
            config=botocore.config.Config(s3={'addressing_style': 'virtual'}),
            region_name=os.getenv('REGION_NAME'), aws_access_key_id=os.getenv('SECRET_ID'),
            aws_secret_access_key=os.getenv('SECRET_KEY')
        )

        # Resource Side
        client_resource = boto3.resource(
            's3', endpoint_url='https://nyc3.digitaloceanspaces.com',
            config=botocore.config.Config(s3={'addressing_style': 'virtual'}),
            region_name='nyc3', aws_access_key_id=os.getenv('SECRET_ID'),
            aws_secret_access_key=os.getenv('SECRET_KEY')
        )

    UPLOAD_FOLDER, UPLOAD_FOLDER_KEY = 'MY_BUCKET', 'uploads/MIDI_files/'

   return UPLOAD_FOLDER_TYPE, UPLOAD_FOLDER, UPLOAD_FOLDER_KEY, client_resource, client

Thus far, I’ve tried:

with open(file_path, 'wb') as output_file:
    MyMIDI.writeFile(output_file)
    client.download_fileobj(UPLOAD_FOLDER, 'OBJECT_NAME', output_file)

and a wealth of other .put_object combinations with client and client_resource boto3 objects…

I’m thinking that my problem lies within:

The writeFile(filehandler) of the MIDIUtil.Midifile

Perhaps this function is closing the MIDI binary stream DATA before I could put_object into a s3 BODY=? Maybe I need to parse the binary data through a Bytes(IO) / stream object..?

OR

Trying to achieve a writeable directory using my s3 object.

Perhaps I could assign the s3 UPLOAD_FOLDER better… I’m just not sure how I would make this connection in FLASK…

app.config['UPLOAD_FOLDER'] = client.Object(
    Bucket=UPLOAD_FOLDER, Key=UPLOAD_FOLDER_KEY,
    ACL='private'
)

Any help is appreciated! Feel like I may have gotten closer with this method…
It does actually write to the s3 Bucket, so I might ditch worrying about grabbing a usable URL, but the MIDI file is corrupted and blank =(

file_path = f'{UPLOAD_FOLDER_KEY}{file_name}'
            response = client.generate_presigned_post(UPLOAD_FOLDER,
                                                      file_name,
                                                      ExpiresIn=3600)
            post_url = response['url']
            data = response['fields']
            key = data['key']
            with open(file_name, 'wb') as f:
                http_response = requests.post(url=post_url, data=data,
                                              files={file_name: MyMIDI.writeFile(f)})

print(response) produces:

{'url': 'ENDPOINT_URL', 'fields': {'key': 'files(from above)', 'x-amz-algorithm': 'STUFF', 'x-amz-credential': 'STUFF', 'x-amz-date': 'STUFF', 'policy': 'STUFF', 'x-amz-signature': 'STUFF'}}```

Just not positive if I can pull a URL from this to redirect to…
Trying to dissolve this Article on S3 File Uploads for an answer.

Asked By: Christopher Burrows

||

Source

Answer 1

My Initial Solution:
I can’t EXACTLY explain why; but I think the issue came along with the interaction between MyMIDI.writefile(filehandler) and boto3 functions. I think it has something to do within the .write() and .close() being nested with the .writefile() of the MIDIUtil package, WHILE having to simultaneously generate the byte data for s3's Body parameter. So here’s my workaround…

# Working Version on S3 Deployment
# Assign generate_presigned_post to variable 
response = client.generate_presigned_post(UPLOAD_FOLDER,
                                          file_name,
                                          ExpiresIn=3600)

# Have MIDIUtil write / close the file within writefile       
with open(file_name, 'wb') as file:
    MIDI_FILE = MyMIDI.writeFile(file)

# Read the written binary contents for the s3 Body; assign to a variable (content)
f = open(file_name, 'rb')  
content = f.read()
       
# Stage the object with its file_name and s3 Bucket(UPLOAD_FOLDER)
MIDI_Object = client_resource.Object(UPLOAD_FOLDER, file_name)

# Write to the s3 Bucket using put
MIDI_Object.put(Body=content)

VOILA! It’s no longer blank in my S3 Bucket and available for download!! 😀

Better Solution: (thanks to @jarmod) Way less code lines and does the same thing 🙂

with open(MIDI_file_name, 'wb') as file:
    MIDI_FILE = MIDI_DATA.writeFile(file)

object_name = os.path.basename(file_name)
client.upload_file(MIDI_file_name, UPLOAD_FOLDER, object_name)

Answered By: Christopher Burrows

Answer 2

This code will not work as intended:

with open(file_path, 'wb') as output_file:
    MyMIDI.writeFile(output_file)
    client.upload_file(UPLOAD_FOLDER, 'OBJECT_NAME', output_file)

The problem with this code is that you are using a context manager (with) and the context manager does not close the output file until you exit the context manager. Consequently, the file contents are not flushed to disk at the point you attempt to upload the file to S3.

It needs to be written like this:

with open(file_path, 'wb') as output_file:
    MyMIDI.writeFile(output_file)

client.upload_file(UPLOAD_FOLDER, 'OBJECT_NAME', output_file)

Answered By: jarmod

MIDIutil write MIDI File to boto3 API server via Flask [Python]

Question:

Answers: