Generic vs Specific MyPy types of functions
Question:
I remember reading, or hearing somewhere that for a function, input types should be as generic as possible (Iterable over list), but return types should be as specific as possible.
Is this written down somewhere official that I can reference when this comes up in team discussions? Or am I crazy and this isn’t actually a guideline?
Answers:
A quick google hasn’t found anything "official", but the benefits seem self-evident to me, so I’ll take a crack at explaining them and you can decide whether the explanation sounds official enough.
Making parameter types as general as possible
The main benefit of this is in making the calling code simple. Suppose you have a silly function like this:
def add_ints(nums: list[int]) -> int:
return sum(nums)
This works fine, but what if your caller has a tuple[int, int, int]?
nums = (1, 2, 3)
print(add_ints(nums)) # fails
print(list(add_ints(nums))) # works
This is silly; there’s no good reason for them to have to convert their tuple to a list, other than the fact that you decided to annotate your function to require one. It’s extra code to write (and read) and it’ll also make it a little slower at runtime. You should instead define add_ints to take an Iterable[int]:
from typing import Iterable
def add_ints(nums: Iterable[int]) -> int:
return sum(nums)
A second benefit is that it is easier to infer from the type annotation what the function does. If the function takes a list, there is a possibility that it might mutate it, since the list interface allows mutation; an Iterable isn’t mutable, so we can now tell at a glance that even if we pass add_ints a list, it isn’t going to try to modify it — and mypy will enforce that within the implementation of add_ints as well!
Making return types as specific as possible
This is just the corollary to the above. Suppose you have:
def nums_up_to(top: int) -> Iterable[int]:
return list(range(top))
This is technically valid — but what if our caller needs a list? Again, we’re forcing them to do needless checking/conversion:
nums = nums_up_to(5)
nums.append(add_ints(nums)) # fails, can't append to an iterable
nums = nums_up_to(5)
assert isinstance(nums, list)
nums.append(add_ints(nums)) # works because we narrowed the type with that assert
nums = list(nums_up_to(5))
nums.append(add_ints(nums)) # works because we explicitly constructed a list
Again, this is much more easily fixed by just improving the type annotation:
def nums_up_to(top: int) -> list[int]:
return list(range(top))
nums = nums_up_to(5)
nums.append(add_ints(nums)) # fine!
Remember: YAGNI
It’s worth remembering that applying these guidelines is something that doesn’t necessarily need to be done rigorously up front — widening a parameter type and narrowing a return type are both backwards-compatible changes as far as the caller is concerned.
In practice I usually find myself applying these guidelines when I’m in the process of writing calling code, I find that I’m doing some unnecessary type conversion, and I resolve the issue by loosening/tightening the annotation in the dependency rather than working around it in my own code, trusting that mypy will let me know if my annotation doesn’t match the actual implementation.
I remember reading, or hearing somewhere that for a function, input types should be as generic as possible (Iterable over list), but return types should be as specific as possible.
Is this written down somewhere official that I can reference when this comes up in team discussions? Or am I crazy and this isn’t actually a guideline?
A quick google hasn’t found anything "official", but the benefits seem self-evident to me, so I’ll take a crack at explaining them and you can decide whether the explanation sounds official enough.
Making parameter types as general as possible
The main benefit of this is in making the calling code simple. Suppose you have a silly function like this:
def add_ints(nums: list[int]) -> int:
return sum(nums)
This works fine, but what if your caller has a tuple[int, int, int]?
nums = (1, 2, 3)
print(add_ints(nums)) # fails
print(list(add_ints(nums))) # works
This is silly; there’s no good reason for them to have to convert their tuple to a list, other than the fact that you decided to annotate your function to require one. It’s extra code to write (and read) and it’ll also make it a little slower at runtime. You should instead define add_ints to take an Iterable[int]:
from typing import Iterable
def add_ints(nums: Iterable[int]) -> int:
return sum(nums)
A second benefit is that it is easier to infer from the type annotation what the function does. If the function takes a list, there is a possibility that it might mutate it, since the list interface allows mutation; an Iterable isn’t mutable, so we can now tell at a glance that even if we pass add_ints a list, it isn’t going to try to modify it — and mypy will enforce that within the implementation of add_ints as well!
Making return types as specific as possible
This is just the corollary to the above. Suppose you have:
def nums_up_to(top: int) -> Iterable[int]:
return list(range(top))
This is technically valid — but what if our caller needs a list? Again, we’re forcing them to do needless checking/conversion:
nums = nums_up_to(5)
nums.append(add_ints(nums)) # fails, can't append to an iterable
nums = nums_up_to(5)
assert isinstance(nums, list)
nums.append(add_ints(nums)) # works because we narrowed the type with that assert
nums = list(nums_up_to(5))
nums.append(add_ints(nums)) # works because we explicitly constructed a list
Again, this is much more easily fixed by just improving the type annotation:
def nums_up_to(top: int) -> list[int]:
return list(range(top))
nums = nums_up_to(5)
nums.append(add_ints(nums)) # fine!
Remember: YAGNI
It’s worth remembering that applying these guidelines is something that doesn’t necessarily need to be done rigorously up front — widening a parameter type and narrowing a return type are both backwards-compatible changes as far as the caller is concerned.
In practice I usually find myself applying these guidelines when I’m in the process of writing calling code, I find that I’m doing some unnecessary type conversion, and I resolve the issue by loosening/tightening the annotation in the dependency rather than working around it in my own code, trusting that mypy will let me know if my annotation doesn’t match the actual implementation.