gzipped jsonlines file read and write in python

Question

While this code reads and writes a jsonlines file. How to compress it? I tried directly using gzip.open but I am getting various errors.

import json
    
def dump_jsonl(data, output_path, append=False):
    """
    Write list of objects to a JSON lines file.
    """
    mode = 'a+' if append else 'w'
    with open(output_path, mode, encoding='utf-8') as f:
        for line in data:
            json_record = json.dumps(line, ensure_ascii=False)
            f.write(json_record + 'n')
    print('Wrote {} records to {}'.format(len(data), output_path))

def load_jsonl(input_path) -> list:
    """
    Read list of objects from a JSON lines file.
    """
    data = []
    with open(input_path, 'r', encoding='utf-8') as f:
        for line in f:
            data.append(json.loads(line.rstrip('n|r')))
    print('Loaded {} records from {}'.format(len(data), input_path))
    return data

This is what I am doing to compress but I am unable to read it.

def dump_jsonl(data, output_path, append=False):
    with gzip.open(output_path, "a+") as f:
        for line in data:
            json_record = json.dumps(line, ensure_ascii = False)
            encoded = json_record.encode("utf-8") + ("n").encode("utf-8")
            compressed = gzip.compress(encoded)
            f.write(compressed)

Asked By: coding zombie

||

Source

Answer 1

Use the gzip module‘s compress function.

import gzip
with open('file.jsonl') as f_in:
    with gzip.open('file.jsonl.gz', 'wb') as f_out:
        f_out.writelines(f_in)

gzip.open() is for opening gzipped files, not jsonl.

Read:

gzip a file in Python

Python support for Gzip

Answered By: Datajack

Answer 2

you can use filesff and create json lines formatter with gzip file handle

for example:

accessor = GzippedFileHandle.of_str('fileName.gz').access(JsonLinesFormatter())

and open it by:

# dump
accessor.dump(list_of_dicts)
# load
list_of_dicts = accessor.load()

Answered By: netanelrevah

gzipped jsonlines file read and write in python

Question:

Answers: