Picking function to set:get values (and how to pass **kwargs)

Question

I have a program that runs daily where there are a number of objects that a) take a while to create, b) are static for the day once they’re built, so are good candidates to get pickled.

To avoid writing this a bunch of times:

if object_pickle_exists:
    obj = load_pickle_object
else:
    obj = run_some_function_to_build_object()
    pickle(obj) for later

I am instead trying to build a pickle-based function that I can generically use to "Get/Set" variables.

Loads Pickle, otherwise runs function and pickles it

def _dill(name, function_as_an_object, date = dt.date.today()):                                  
     try:
        with open('Pickles/'+name+'_'+date.strftime('%y-%b-%d'),'rb') as f:
            obj = pickle.load(f)
        return obj
    except FileNotFoundError:
        obj = function_as_an_object()
        with open('Pickles/'+name+'_'+date.strftime('%y-%b-%d'),'wb') as f:
            pickle.dump(obj, f)
        return obj

Question 1) Is there a better way to do this / some existing package out there/

Question 2) I would like to add **kwargs to _dill(), but I’m not sure how to pass those kwargs down to the function_as_an_object:

    def _dill(name, function_as_an_object, date = dt.date.today(), **kwargs):                                  
        try:
            with open('Pickles/'+name+'_'+date.strftime('%y-%b-%d'),'rb') as f:
                obj = pickle.load(f)
            return obj
        except FileNotFoundError:
            obj = function_as_an_object(**kwargs) ??
            with open('Pickles/'+name+'_'+date.strftime('%y-%b-%d'),'wb') as f:
                pickle.dump(obj, f)
            return obj

Asked By: keynesiancross

||

Source

Answer 1

Answer to Q1:

I am not a pickle or Clean Code expert myself, but I think that your way of handling an exception is a quite nice solution to this necessary check. Try/except blocks are ugly when used too extensively, but deliver a safety layer to your function. If you would like to check for a file without a try/except block, see this thread on checking for files with the os module. I hope this helps.

Short Answer to Q2:

Your function signature is correct as is:

def _dill(name, function_as_an_object, date = dt.date.today(), **kwargs):

Moreover, your call to the function_as_an_object is technically correct. But depending on how the function is defined, you could cause an error, by passing too many, as little arguments or wrong data types.

Longer Answer to Q2:

Let me make a simple example:

def function_as_an_object(x, y):
    print("x =", x)
    print("y =", y)

def foo(function_as_an_object, **kwargs):
    function_as_an_object(**kwargs)

foo(function_as_an_object, x=1, y=2, z=3)

In this case, foo calls function_as_an_object with the keyword arguments, it has received. But this will result in an error because function_as_an_object only takes in two parameters.

Key word arguments in python return a dictionary with the given names as keys and the given values as values. Allow me to explain the following hypothetical function call of your _dill method:

_dill("filename", function_as_an_object, x="string value", y=20)

In the case shown above, the x and y arguments, are stored in a kwargs object, which is essentially a dictionary. You can access the values by iterating over the kwargs variable for example. See the geeks for geeks tutorial on **kwargs for more information. Now when you pass the kwargs to your function (function_as_an_object(**kwargs)) you are essentially passing the value of x (string) and y (integer) in this order to the function.

All in all, be aware of how many keyword arguments you pass, what you are doing with your kwargs. Otherwise, you can quickly run into errors.

Answered By: ovanov

Picking function to set:get values (and how to pass **kwargs)

Question:

Loads Pickle, otherwise runs function and pickles it

Answers: