pickle and dill can't load objects with overridden __hash__ function (AttributeError) Question: In the next few lines of code I’ll replicate on a smaller scale what’s happening with my program. Class A must store a dictionary with keys that have type A (values can be any type to replicate the error). class A: def __init__(self, …
AttributeError: module 'dill._dill' has no attribute 'log' Question: I am using a python nlp module to train a dataset and ran into the following error: File "/usr/local/lib/python3.9/site-packages/nlp/utils/py_utils.py", line 297, in save_code dill._dill.log.info("Co: %s" % obj) AttributeError: module ‘dill._dill’ has no attribute ‘log’ I noticed similar posts where no attribute ‘extend’ and no attribute ‘stack’ where …
How to serialize custom classes in Python? Question: I have a custom class and I want to serialize it for multiprocessing, but pickle and dill doesn’t work fine and loses important data. How can I fix this? My class: import pickle import dill import pandas as pd import numpy as np class C(pd.Series): def __init__(self, …
Python error [WinError 6] while using multiprocessing.Process with dill instead of standard Pickle Question: I have this Process class which overrides standard multiprocessing.Process class in order to catch exceptions occurred in child process. Also I use dill and override multiprocessing standard Pickler because I need to. import dill import traceback import multiprocessing dill.Pickler.dumps, dill.Pickler.loads = …
Unable to load pickled custom estimator sklearn pipeline Question: I have a sklearn pipeline that uses custom column transformer, estimator and different lambda functions. Because Pickle cannot serialize the lambda functions, I am using dill. Here is the custom estimator I have: class customOLS(BaseEstimator): def __init__(self, ols): self.estimator_ols = ols def fit(self, X, y): X …
ERROR:root:can't pickle fasttext_pybind.fasttext objects Question: I am using gunicorn with multiple workers for my machine learning project. But the problem is when I send a train request only the worker getting the training request gets updated with the latest model after training is done. Here it is worth to mention that, to make the inference …
TypeError: can't pickle PyCapsule objects Question: I use dill to save ML model to file. When I run my tests with python -m unittest it works. But if I try run tests with python setup.py test it getting error TypeError: can’t pickle PyCapsule objects in raw where I try to save model. My settings in …
How can I recover a corrupted, partially pickled file? Question: My program was killed while serializing data (a dict) to disk with dill. I cannot open the partially-written file now. Is it possible to partially or fully recover the data? If so, how? Here’s what I’ve tried: >>> dill.load(open(filename, ‘rb’)) Traceback (most recent call last): …
What are the pitfalls of using Dill to serialise scikit-learn/statsmodels models? Question: I need to serialise scikit-learn/statsmodels models such that all the dependencies (code + data) are packaged in an artefact and this artefact can be used to initialise the model and make predictions. Using the pickle module is not an option because this will …
What can multiprocessing and dill do together? Question: I would like to use the multiprocessing library in Python. Sadly multiprocessing uses pickle which doesn’t support functions with closures, lambdas, or functions in __main__. All three of these are important to me In [1]: import pickle In [2]: pickle.dumps(lambda x: x) PicklingError: Can’t pickle <function <lambda> …
