Cannot load pickled object

Question

The problem I am having is when I try to load the pickled object. I have tried using both pickle.loads and pickle.load Here are the results:

pickle.loads:

TypeError: ‘str’ does not support the buffer interface

pickle.load:

TypeError: file must have ‘read’ and ‘readline’ attributes

Can someone please tell me what I am doing wrong in this process?

elif str(parser) == "SwissWithdrawn_Parser":
    # swissprot name changes
    print("Gathering SwissProt update info...")
    cache_hits = 0
    cache_misses = 0
    files = set()

    for f in os.listdir("out/cache/"):
        if os.path.isfile("out/cache/" + f):
            files.add(f)

    for name in sp_lost_names:

        cached = False
        url = (
            "http://www.uniprot.org/uniprot/?query=mnemonic%3a"
            + name
            + "+active%3ayes&format=tab&columns=entry%20name"
        )
        hashed_url = str(hash(url))

        ################### For Testing Only - use cache ##################
        if hashed_url in files:
            cached = True
            cache_hits += 1
            content = pickle.loads("out/cache/" + hashed_url)  # <-- problematic line
        else:
            cache_misses += 1
            content = urllib.request.urlopen(url)

        # get the contents returned from the HTTPResponse object
        content_list = [x.decode().strip() for x in content.readlines()]
        if not cached:
            with open("out/cache/" + hashed_url, "wb") as fp:
                pickle.dump(content_list, fp)
        ####################################################################

        # no replacement
        if len(content_list) is 0:
            change_log["swiss-names"] = {name: "withdrawn"}
        # get the new name
        else:
            new_name = content_list[1]
            change_log["swiss-names"] = {name: new_name}

Asked By: Houdini

||

Source

Answer 1

You need to either read the file first (as binary bytes) and use pickle.loads(), or pass an open file object to the pickle.load() command. The latter is preferable:

with open('out/cache/' +hashed_url, 'rb') as pickle_file:
    content = pickle.load(pickle_file)

Neither method supports loading a pickle from a filename.

Answered By: Martijn Pieters

Answer 2

If you happen to be porting python2 to 3 and run into this error, python2 and 3 handle bytes different leading to the requirement to open your file handle with the ‘b’ option. For instance in python2 open(file, 'r') as f: my_list = pickle.load(f) works , but not in python3. Instead you must open with open(file, 'rb') as f: my_list = pickle.load(f)

Answered By: Kyle Meador

Cannot load pickled object

Question:

Answers: