How to mount S3 bucket as local FileSystem? Question: I have a python app running on a Jupiter-notebook on AWS. I loaded a C-library into my python code which expects a path to a file. I would like to access this file from the S3 bucket. I tried to use s3fs: s3 = s3fs.S3FileSystem(anon=False) using …
How to read partitioned parquet files from S3 using pyarrow in python Question: I looking for ways to read data from multiple partitioned directories from s3 using python. data_folder/serial_number=1/cur_date=20-12-2012/abcdsd0324324.snappy.parquet data_folder/serial_number=2/cur_date=27-12-2012/asdsdfsd0324324.snappy.parquet pyarrow’s ParquetDataset module has the capabilty to read from partitions. So I have tried the following code : >>> import pandas as pd >>> import …
