Python convert args to kwargs
Question:
I am writing a decorator that needs to call other functions prior to call of the function that it is decorating. The decorated function may have positional arguments, but the functions the decorator will call can only accept keyword arguments. Does anyone have a handy way of converting positional arguments into keyword arguments?
I know that I can get a list of the variable names of the decorated function:
>>> def a(one, two=2):
... pass
>>> a.func_code.co_varnames
('one', 'two')
But I can’t figure out how to tell what was passed in positionally and what was as keyword.
My decorator looks like this:
class mydec(object):
def __init__(self, f, *args, **kwargs):
self.f = f
def __call__(self, *args, **kwargs):
hozer(**kwargs)
self.f(*args, **kwargs)
Is there a way other than just comparing kwargs and co_varnames, adding to kwargs anything not in there, and hoping for the best?
Answers:
Any arg that was passed positionally will be passed to *args. And any arg passed as a keyword will be passed to **kwargs.
If you have positional args values and names then you can do:
kwargs.update(dict(zip(myfunc.func_code.co_varnames, args)))
to convert them all into keyword args.
Note – co_varnames will include local variables as well as keywords. This probably won’t matter, as zip truncates the shorter sequence, but may result in confusing error messages if you pass the wrong number of args.
You can avoid this with func_code.co_varnames[:func_code.co_argcount], but better is to use the inspect module. ie:
import inspect
argnames, varargs, kwargs, defaults = inspect.getargspec(func)
You may also want to handle the case where the function defines **kwargs or *args (even if just to raise an exception when used with the decorator). If these are set, the second and third result from getargspec will return their variable name, otherwise they will be None.
Well, this may be overkill. I wrote it for the dectools package (on PyPi), so you can get updates there. It returns the dictionary taking into account positional, keyword, and default arguments. There is a test suite in the package (test_dict_as_called.py):
def _dict_as_called(function, args, kwargs):
""" return a dict of all the args and kwargs as the keywords they would
be received in a real function call. It does not call function.
"""
names, args_name, kwargs_name, defaults = inspect.getargspec(function)
# assign basic args
params = {}
if args_name:
basic_arg_count = len(names)
params.update(zip(names[:], args)) # zip stops at shorter sequence
params[args_name] = args[basic_arg_count:]
else:
params.update(zip(names, args))
# assign kwargs given
if kwargs_name:
params[kwargs_name] = {}
for kw, value in kwargs.iteritems():
if kw in names:
params[kw] = value
else:
params[kwargs_name][kw] = value
else:
params.update(kwargs)
# assign defaults
if defaults:
for pos, value in enumerate(defaults):
if names[-len(defaults) + pos] not in params:
params[names[-len(defaults) + pos]] = value
# check we did it correctly. Each param and only params are set
assert set(params.iterkeys()) == (set(names)|set([args_name])|set([kwargs_name])
)-set([None])
return params
If you’re using Python >= 2.7 inspect.getcallargs() does this for you out of the box. You’d just pass it the decorated function as the first argument, and then the rest of the arguments exactly as you plan to call it. Example:
>>> def f(p1, p2, k1=None, k2=None, **kwargs):
... pass
>>> from inspect import getcallargs
I’m planning to do f('p1', 'p2', 'p3', k2='k2', extra='kx1') (note that k1 is being passed positionally as p3), so…
>>> call_args = getcallargs(f, 'p1', 'p2', 'p3', k2='k2', extra='kx1')
>>> call_args
{'p2': 'p2', 'k2': 'k2', 'k1': 'p3', 'p1': 'p1', 'kwargs': {'extra': 'kx1'}}
If you know the decorated function won’t use **kwargs, then that key won’t appear in the dict, and you’re done (and I’m assuming there’s no *args, since that would break the requirement that everything have a name). If you do have **kwargs, as I have in this example, and want to include them with the rest of the named arguments, it takes one more line:
>>> call_args.update(call_args.pop('kwargs'))
>>> call_args
{'p2': 'p2', 'k2': 'k2', 'k1': 'p3', 'p1': 'p1', 'extra': 'kx1'}
Update: for Python >= 3.3, see inspect.Signature.bind() and the related inspect.signature function for functionality similar to (but more robust than) inspect.getcallargs().
Here’s a newer method to solve this using inspect.signature (for Python 3.3+). I’ll give an example that can be run / tested yourself first and then show how to modify the original code with it.
Here’s a test function which just sums up any args/kwargs given to it; at least one argument is required (a) and there’s one keyword-only argument with a default value (b), just to test different aspects of function signatures.
def silly_sum(a, *args, b=1, **kwargs):
return a + b + sum(args) + sum(kwargs.values())
Now let’s make a wrapper for silly_sum which can be called in the same way as silly_sum (with an exception which we’ll get to) but that only passes in kwargs to the wrapped silly_sum.
def wrapper(f):
sig = inspect.signature(f)
def wrapped(*args, **kwargs):
bound_args = sig.bind(*args, **kwargs)
bound_args.apply_defaults()
print(bound_args) # just for testing
all_kwargs = bound_args.arguments
assert len(all_kwargs.pop("args", [])) == 0
all_kwargs.update(all_kwargs.pop("kwargs"))
return f(**all_kwargs)
return wrapped
sig.bind returns a BoundArguments object, but this doesn’t take defaults into account unless you call apply_defaults explicitly. Doing so will also generate an empty tuple for args and an empty dict for kwargs if no *args/**kwargs were given.
sum_wrapped = wrapper(silly_sum)
sum_wrapped(1, c=9, d=11)
# prints <BoundArguments (a=1, args=(), b=1, kwargs={'c': 9, 'd': 11})>
# returns 22
Then we just get the dictionary of arguments and add any **kwargs in. The exception to using this wrapper is that *args can’t be passed to the function. This is because there are no names for these, so we can’t convert them into kwargs. If passing them through as a kwarg named args is acceptable, that could be done instead.
Here is how this can be applied to the original code:
import inspect
class mydec(object):
def __init__(self, f, *args, **kwargs):
self.f = f
self._f_sig = inspect.signature(f)
def __call__(self, *args, **kwargs):
bound_args = self._f_sig.bind(*args, **kwargs)
bound_args.apply_defaults()
all_kwargs = bound_args.arguments
assert len(all_kwargs.pop("args"), []) == 0
all_kwargs.update(all_kwargs.pop("kwargs"))
hozer(**all_kwargs)
self.f(*args, **kwargs)
Nadia’s answer is correct, but I feel like a working demo of that answer is useful.
def decorator(func):
def wrapped_func(*args, **kwargs):
kwargs.update(zip(func.__code__.co_varnames, args))
print(kwargs)
return func(**kwargs)
return wrapped_func
@decorator
def thing(a,b):
return a+b
Given this decorated function, the following calls return the appropriate answer:
thing(1, 2) # prints {'a': 1, 'b': 2} returns 3
thing(1, b=2) # prints {'b': 2, 'a': 1} returns 3
thing(a=1, b=2) # prints {'a': 1, 'b': 2} returns 3
Note however that things start getting weird if you start nesting decorators because the decorated function now no longer takes a and b, it takes args and kwargs:
@decorator
@decorator
def thing(a,b):
return a+b
Here thing(1,2) will print {'args': 1, 'kwargs': 2} and error with TypeError: thing() got an unexpected keyword argument 'args'
Fleshing out the (best) solution by @mikenerone here is the solution to original poster’s problem:
import inspect
from functools import wraps
class mydec(object):
def __init__(self, f, *args, **kwargs):
self.f = f
def __call__(self, *args, **kwargs):
call_args = inspect.getcallargs(self.f, *args, **kwargs)
hozer(**call_args)
return self.f(*args, **kwargs)
def hozer(**kwargs):
print('hozer got kwds:', kwargs)
def myadd(i, j=0):
return i + j
o = mydec(myadd)
assert o(1,2) == 3
assert o(1) == 1
assert o(1, j=2) == 3
hozer got kwds: {'i': 1, 'j': 2}
hozer got kwds: {'i': 1, 'j': 0}
hozer got kwds: {'i': 1, 'j': 2}
Here is a generalised decorator that converts and consolidates all parameters to a Python function into kwargs and calls the wrapped function only with those kwargs.
def call_via_kwargs(f):
@wraps(f)
def wrapper(*args, **kwds):
call_args = inspect.getcallargs(f, *args, **kwds)
print('kwargs:', call_args)
return f(**call_args)
return wrapper
@call_via_kwargs
def adder(i, j=0):
return i + j
assert adder(1) == 1
assert adder(i=1) == 1
assert adder(1, j=2) == 3
kwargs: {'i': 1, 'j': 0}
kwargs: {'i': 1, 'j': 0}
kwargs: {'i': 1, 'j': 2}
These solutions correctly handle default parameters.
I am writing a decorator that needs to call other functions prior to call of the function that it is decorating. The decorated function may have positional arguments, but the functions the decorator will call can only accept keyword arguments. Does anyone have a handy way of converting positional arguments into keyword arguments?
I know that I can get a list of the variable names of the decorated function:
>>> def a(one, two=2):
... pass
>>> a.func_code.co_varnames
('one', 'two')
But I can’t figure out how to tell what was passed in positionally and what was as keyword.
My decorator looks like this:
class mydec(object):
def __init__(self, f, *args, **kwargs):
self.f = f
def __call__(self, *args, **kwargs):
hozer(**kwargs)
self.f(*args, **kwargs)
Is there a way other than just comparing kwargs and co_varnames, adding to kwargs anything not in there, and hoping for the best?
Any arg that was passed positionally will be passed to *args. And any arg passed as a keyword will be passed to **kwargs.
If you have positional args values and names then you can do:
kwargs.update(dict(zip(myfunc.func_code.co_varnames, args)))
to convert them all into keyword args.
Note – co_varnames will include local variables as well as keywords. This probably won’t matter, as zip truncates the shorter sequence, but may result in confusing error messages if you pass the wrong number of args.
You can avoid this with func_code.co_varnames[:func_code.co_argcount], but better is to use the inspect module. ie:
import inspect
argnames, varargs, kwargs, defaults = inspect.getargspec(func)
You may also want to handle the case where the function defines **kwargs or *args (even if just to raise an exception when used with the decorator). If these are set, the second and third result from getargspec will return their variable name, otherwise they will be None.
Well, this may be overkill. I wrote it for the dectools package (on PyPi), so you can get updates there. It returns the dictionary taking into account positional, keyword, and default arguments. There is a test suite in the package (test_dict_as_called.py):
def _dict_as_called(function, args, kwargs):
""" return a dict of all the args and kwargs as the keywords they would
be received in a real function call. It does not call function.
"""
names, args_name, kwargs_name, defaults = inspect.getargspec(function)
# assign basic args
params = {}
if args_name:
basic_arg_count = len(names)
params.update(zip(names[:], args)) # zip stops at shorter sequence
params[args_name] = args[basic_arg_count:]
else:
params.update(zip(names, args))
# assign kwargs given
if kwargs_name:
params[kwargs_name] = {}
for kw, value in kwargs.iteritems():
if kw in names:
params[kw] = value
else:
params[kwargs_name][kw] = value
else:
params.update(kwargs)
# assign defaults
if defaults:
for pos, value in enumerate(defaults):
if names[-len(defaults) + pos] not in params:
params[names[-len(defaults) + pos]] = value
# check we did it correctly. Each param and only params are set
assert set(params.iterkeys()) == (set(names)|set([args_name])|set([kwargs_name])
)-set([None])
return params
If you’re using Python >= 2.7 inspect.getcallargs() does this for you out of the box. You’d just pass it the decorated function as the first argument, and then the rest of the arguments exactly as you plan to call it. Example:
>>> def f(p1, p2, k1=None, k2=None, **kwargs):
... pass
>>> from inspect import getcallargs
I’m planning to do f('p1', 'p2', 'p3', k2='k2', extra='kx1') (note that k1 is being passed positionally as p3), so…
>>> call_args = getcallargs(f, 'p1', 'p2', 'p3', k2='k2', extra='kx1')
>>> call_args
{'p2': 'p2', 'k2': 'k2', 'k1': 'p3', 'p1': 'p1', 'kwargs': {'extra': 'kx1'}}
If you know the decorated function won’t use **kwargs, then that key won’t appear in the dict, and you’re done (and I’m assuming there’s no *args, since that would break the requirement that everything have a name). If you do have **kwargs, as I have in this example, and want to include them with the rest of the named arguments, it takes one more line:
>>> call_args.update(call_args.pop('kwargs'))
>>> call_args
{'p2': 'p2', 'k2': 'k2', 'k1': 'p3', 'p1': 'p1', 'extra': 'kx1'}
Update: for Python >= 3.3, see inspect.Signature.bind() and the related inspect.signature function for functionality similar to (but more robust than) inspect.getcallargs().
Here’s a newer method to solve this using inspect.signature (for Python 3.3+). I’ll give an example that can be run / tested yourself first and then show how to modify the original code with it.
Here’s a test function which just sums up any args/kwargs given to it; at least one argument is required (a) and there’s one keyword-only argument with a default value (b), just to test different aspects of function signatures.
def silly_sum(a, *args, b=1, **kwargs):
return a + b + sum(args) + sum(kwargs.values())
Now let’s make a wrapper for silly_sum which can be called in the same way as silly_sum (with an exception which we’ll get to) but that only passes in kwargs to the wrapped silly_sum.
def wrapper(f):
sig = inspect.signature(f)
def wrapped(*args, **kwargs):
bound_args = sig.bind(*args, **kwargs)
bound_args.apply_defaults()
print(bound_args) # just for testing
all_kwargs = bound_args.arguments
assert len(all_kwargs.pop("args", [])) == 0
all_kwargs.update(all_kwargs.pop("kwargs"))
return f(**all_kwargs)
return wrapped
sig.bind returns a BoundArguments object, but this doesn’t take defaults into account unless you call apply_defaults explicitly. Doing so will also generate an empty tuple for args and an empty dict for kwargs if no *args/**kwargs were given.
sum_wrapped = wrapper(silly_sum)
sum_wrapped(1, c=9, d=11)
# prints <BoundArguments (a=1, args=(), b=1, kwargs={'c': 9, 'd': 11})>
# returns 22
Then we just get the dictionary of arguments and add any **kwargs in. The exception to using this wrapper is that *args can’t be passed to the function. This is because there are no names for these, so we can’t convert them into kwargs. If passing them through as a kwarg named args is acceptable, that could be done instead.
Here is how this can be applied to the original code:
import inspect
class mydec(object):
def __init__(self, f, *args, **kwargs):
self.f = f
self._f_sig = inspect.signature(f)
def __call__(self, *args, **kwargs):
bound_args = self._f_sig.bind(*args, **kwargs)
bound_args.apply_defaults()
all_kwargs = bound_args.arguments
assert len(all_kwargs.pop("args"), []) == 0
all_kwargs.update(all_kwargs.pop("kwargs"))
hozer(**all_kwargs)
self.f(*args, **kwargs)
Nadia’s answer is correct, but I feel like a working demo of that answer is useful.
def decorator(func):
def wrapped_func(*args, **kwargs):
kwargs.update(zip(func.__code__.co_varnames, args))
print(kwargs)
return func(**kwargs)
return wrapped_func
@decorator
def thing(a,b):
return a+b
Given this decorated function, the following calls return the appropriate answer:
thing(1, 2) # prints {'a': 1, 'b': 2} returns 3
thing(1, b=2) # prints {'b': 2, 'a': 1} returns 3
thing(a=1, b=2) # prints {'a': 1, 'b': 2} returns 3
Note however that things start getting weird if you start nesting decorators because the decorated function now no longer takes a and b, it takes args and kwargs:
@decorator
@decorator
def thing(a,b):
return a+b
Here thing(1,2) will print {'args': 1, 'kwargs': 2} and error with TypeError: thing() got an unexpected keyword argument 'args'
Fleshing out the (best) solution by @mikenerone here is the solution to original poster’s problem:
import inspect
from functools import wraps
class mydec(object):
def __init__(self, f, *args, **kwargs):
self.f = f
def __call__(self, *args, **kwargs):
call_args = inspect.getcallargs(self.f, *args, **kwargs)
hozer(**call_args)
return self.f(*args, **kwargs)
def hozer(**kwargs):
print('hozer got kwds:', kwargs)
def myadd(i, j=0):
return i + j
o = mydec(myadd)
assert o(1,2) == 3
assert o(1) == 1
assert o(1, j=2) == 3
hozer got kwds: {'i': 1, 'j': 2}
hozer got kwds: {'i': 1, 'j': 0}
hozer got kwds: {'i': 1, 'j': 2}
Here is a generalised decorator that converts and consolidates all parameters to a Python function into kwargs and calls the wrapped function only with those kwargs.
def call_via_kwargs(f):
@wraps(f)
def wrapper(*args, **kwds):
call_args = inspect.getcallargs(f, *args, **kwds)
print('kwargs:', call_args)
return f(**call_args)
return wrapper
@call_via_kwargs
def adder(i, j=0):
return i + j
assert adder(1) == 1
assert adder(i=1) == 1
assert adder(1, j=2) == 3
kwargs: {'i': 1, 'j': 0}
kwargs: {'i': 1, 'j': 0}
kwargs: {'i': 1, 'j': 2}
These solutions correctly handle default parameters.