Calling a Python function with *args,**kwargs and optional / default arguments
Question:
In Python I can define a function as follows:
def func(kw1=None,kw2=None,**kwargs):
...
In this case, I can call func as:
func(kw1=3,kw2=4,who_knows_if_this_will_be_used=7,more_kwargs=Ellipsis)
I can also define a function as:
def func(arg1,arg2,*args):
...
which can be called as
func(3,4,additional,arguments,go,here,Ellipsis)
Finally, I can combine the two forms
def func(arg1,arg2,*args,**kwargs):
...
But, what does not work is calling:
func(arg1,arg2,*args,kw1=None,kw2=None,**kwargs): #SYNTAX ERROR (in Python 2 only, apparently this works in Python 3)
...
My original thought was that this was probably because a function
def func(arg1,arg2,*args,kw1=None):
...
can be called as
func(1,2,3) #kw1 will be assigned 3
So this would introduce some ambiguity as to whether 3 should be packed into args or kwargs. However, with Python 3, there is the ability to specify keyword only arguments:
def func(a,b,*,kw=None): # can be called as func(1,2), func(1,2,kw=3), but NOT func(1,2,3)
...
With this, it seems that there is no syntactic ambiguity with:
def func(a,b,*args,*,kw1=None,**kwargs):
...
However, this still brings up a syntax error (tested with Python3.2). Is there a reason for this that I am missing? And, is there a way to get the behavior I described above (Having *args with default arguments) — I know I can simulate that behavior by manipulating the kwargs dictionary inside the function.
Answers:
You can do that in Python 3.
def func(a,b,*args,kw1=None,**kwargs):
The bare * is only used when you want to specify keyword only arguments without accepting a variable number of positional arguments with *args. You don’t use two *s.
To quote from the grammar, in Python 2, you have
parameter_list ::= (defparameter ",")*
( "*" identifier [, "**" identifier]
| "**" identifier
| defparameter [","] )
while in Python 3, you have
parameter_list ::= (defparameter ",")*
( "*" [parameter] ("," defparameter)*
[, "**" parameter]
| "**" parameter
| defparameter [","] )
which includes a provision for additional parameters after the * parameter.
UPDATE:
Latest Python 3 documentation here.
If you want to do a mixture of both remember that *args and **kwargs must be the last parameters specified.
def func(arg1,arg2,*args,kw1=None,kw2=None,**kwargs): #Invalid
def func(arg1,arg2,kw1=None,kw2=None,*args,**kwargs): #Valid
The comments seem to be based on mixing up how a function definition is constructed compared to how the arguments provided are assigned back to the parameters specified in the definition.
This is the definition of this function which has 6 parameters.
It is called by passing named and unnamed arguments to it in a function call.
For this example…
When an argument is named when calling the function it can be provided out of order.
arg1 and arg2 are mandatory parameters and if not passed to the function as named arguments, then they must be assigned in order from the provided unnamed arguments.
kw1 and kw2 have default values provided in the function definition so they are not mandatory, but if not provided for as named arguments they will take any available values from the remaining provided unnamed arguments.
Any unnamed arguments left over are provided to the function in an array called args
Any named arguments that do not have a corresponding parameter name in the function definition are provided to the function call in a dictionary called kwargs.
If you are looking to do that in Python 2, I have found a workaround explained in this post, using a decorator.
This decorator assigns default kwarg values if they are not strictly defined.
from functools import wraps
def force_kwargs(**defaultKwargs):
def decorator(f):
@wraps(f)
def g(*args, **kwargs):
new_args = {}
new_kwargs = defaultKwargs
varnames = f.__code__.co_varnames
new_kwargs.update(kwargs)
for k, v in defaultKwargs.items():
if k in varnames:
i = varnames.index(k)
new_args[(i, k)] = new_kwargs.pop(k)
# Insert new_args into the correct position of the args.
full_args = list(args)
for i, k in sorted(new_args.keys()):
if i <= len(full_args):
full_args.insert(i, new_args.pop((i, k)))
else:
break
# re-insert the value as a key-value pair
for (i, k), val in new_args.items():
new_kwargs[k] = val
return f(*tuple(full_args), **new_kwargs)
return g
return decorator
Result
@force_kwargs(c=7, z=10)
def f(a, b='B', c='C', d='D', *args, **kw):
return a, b, c, d, args, kw
# a b c d args kwargs
f('r') # 'r', 'B', 7, 'D', (), {'z': 10}
f(1, 2, 3, 4, 5) # 1, 2, 7, 3, (4,5), {'z': 10}
f(1, 2, 3, b=0, c=9, f='F', z=5) # 1, 0, 9, 2, (3,), {'f': 'F', 'z': 5}
Variant
If you want to use the default values as written in the function definition, you could access the argument default values using f.func_defaults, which lists the default values. You would have to zip them with the end of the f.__code__.varnames to match these default values with the variable names.
Clear and concise:
In Python 3.5 or greater:
def foo(a, b=3, *args, **kwargs):
defaultKwargs = { 'c': 10, 'd': 12 }
kwargs = { **defaultKwargs, **kwargs }
print(a, b, args, kwargs)
# Do something else
foo(1) # 1 3 () {'c': 10, 'd': 12}
foo(1, d=5) # 1 3 () {'c': 10, 'd': 5}
foo(1, 2, 4, d=5) # 1 2 (4,) {'c': 10, 'd': 5}
Note: you can use
In Python 2
kwargs = merge_two_dicts(defaultKwargs, kwargs)
In Python 3.5
kwargs = { **defaultKwargs, **kwargs }
In Python 3.9
kwargs = defaultKwargs | kwargs # NOTE: 3.9+ ONLY
In Python I can define a function as follows:
def func(kw1=None,kw2=None,**kwargs):
...
In this case, I can call func as:
func(kw1=3,kw2=4,who_knows_if_this_will_be_used=7,more_kwargs=Ellipsis)
I can also define a function as:
def func(arg1,arg2,*args):
...
which can be called as
func(3,4,additional,arguments,go,here,Ellipsis)
Finally, I can combine the two forms
def func(arg1,arg2,*args,**kwargs):
...
But, what does not work is calling:
func(arg1,arg2,*args,kw1=None,kw2=None,**kwargs): #SYNTAX ERROR (in Python 2 only, apparently this works in Python 3)
...
My original thought was that this was probably because a function
def func(arg1,arg2,*args,kw1=None):
...
can be called as
func(1,2,3) #kw1 will be assigned 3
So this would introduce some ambiguity as to whether 3 should be packed into args or kwargs. However, with Python 3, there is the ability to specify keyword only arguments:
def func(a,b,*,kw=None): # can be called as func(1,2), func(1,2,kw=3), but NOT func(1,2,3)
...
With this, it seems that there is no syntactic ambiguity with:
def func(a,b,*args,*,kw1=None,**kwargs):
...
However, this still brings up a syntax error (tested with Python3.2). Is there a reason for this that I am missing? And, is there a way to get the behavior I described above (Having *args with default arguments) — I know I can simulate that behavior by manipulating the kwargs dictionary inside the function.
You can do that in Python 3.
def func(a,b,*args,kw1=None,**kwargs):
The bare * is only used when you want to specify keyword only arguments without accepting a variable number of positional arguments with *args. You don’t use two *s.
To quote from the grammar, in Python 2, you have
parameter_list ::= (defparameter ",")*
( "*" identifier [, "**" identifier]
| "**" identifier
| defparameter [","] )
while in Python 3, you have
parameter_list ::= (defparameter ",")*
( "*" [parameter] ("," defparameter)*
[, "**" parameter]
| "**" parameter
| defparameter [","] )
which includes a provision for additional parameters after the * parameter.
UPDATE:
Latest Python 3 documentation here.
If you want to do a mixture of both remember that *args and **kwargs must be the last parameters specified.
def func(arg1,arg2,*args,kw1=None,kw2=None,**kwargs): #Invalid
def func(arg1,arg2,kw1=None,kw2=None,*args,**kwargs): #Valid
The comments seem to be based on mixing up how a function definition is constructed compared to how the arguments provided are assigned back to the parameters specified in the definition.
This is the definition of this function which has 6 parameters.
It is called by passing named and unnamed arguments to it in a function call.
For this example…
When an argument is named when calling the function it can be provided out of order.
arg1 and arg2 are mandatory parameters and if not passed to the function as named arguments, then they must be assigned in order from the provided unnamed arguments.
kw1 and kw2 have default values provided in the function definition so they are not mandatory, but if not provided for as named arguments they will take any available values from the remaining provided unnamed arguments.
Any unnamed arguments left over are provided to the function in an array called args
Any named arguments that do not have a corresponding parameter name in the function definition are provided to the function call in a dictionary called kwargs.
If you are looking to do that in Python 2, I have found a workaround explained in this post, using a decorator.
This decorator assigns default kwarg values if they are not strictly defined.
from functools import wraps
def force_kwargs(**defaultKwargs):
def decorator(f):
@wraps(f)
def g(*args, **kwargs):
new_args = {}
new_kwargs = defaultKwargs
varnames = f.__code__.co_varnames
new_kwargs.update(kwargs)
for k, v in defaultKwargs.items():
if k in varnames:
i = varnames.index(k)
new_args[(i, k)] = new_kwargs.pop(k)
# Insert new_args into the correct position of the args.
full_args = list(args)
for i, k in sorted(new_args.keys()):
if i <= len(full_args):
full_args.insert(i, new_args.pop((i, k)))
else:
break
# re-insert the value as a key-value pair
for (i, k), val in new_args.items():
new_kwargs[k] = val
return f(*tuple(full_args), **new_kwargs)
return g
return decorator
Result
@force_kwargs(c=7, z=10)
def f(a, b='B', c='C', d='D', *args, **kw):
return a, b, c, d, args, kw
# a b c d args kwargs
f('r') # 'r', 'B', 7, 'D', (), {'z': 10}
f(1, 2, 3, 4, 5) # 1, 2, 7, 3, (4,5), {'z': 10}
f(1, 2, 3, b=0, c=9, f='F', z=5) # 1, 0, 9, 2, (3,), {'f': 'F', 'z': 5}
Variant
If you want to use the default values as written in the function definition, you could access the argument default values using f.func_defaults, which lists the default values. You would have to zip them with the end of the f.__code__.varnames to match these default values with the variable names.
Clear and concise:
In Python 3.5 or greater:
def foo(a, b=3, *args, **kwargs):
defaultKwargs = { 'c': 10, 'd': 12 }
kwargs = { **defaultKwargs, **kwargs }
print(a, b, args, kwargs)
# Do something else
foo(1) # 1 3 () {'c': 10, 'd': 12}
foo(1, d=5) # 1 3 () {'c': 10, 'd': 5}
foo(1, 2, 4, d=5) # 1 2 (4,) {'c': 10, 'd': 5}
Note: you can use
In Python 2
kwargs = merge_two_dicts(defaultKwargs, kwargs)
In Python 3.5
kwargs = { **defaultKwargs, **kwargs }
In Python 3.9
kwargs = defaultKwargs | kwargs # NOTE: 3.9+ ONLY