SQLAlchemy returns `list` instead of `Sequence[Row[_TP]]`
Question:
The type-hinting of the method sqlalchemy.engine.result.Result.fetchall() suggests the method is supposed to return a Sequence[Row[_TP]] but when I print print(type(result_1.fetchall())), I get a list.
Is there anything I’m missing?
My code is the following:
if __name__ == "__main__":
engine: sqlalchemy.engine.base.Engine = sqlalchemy.create_engine(
"sqlite:///data_SQL/new_db.db", echo=False
)
with engine.connect() as connection:
result_1: sqlalchemy.engine.cursor.CursorResult = connection.execute(
sqlalchemy.text("SELECT * FROM MY_TABLE WHERE batch = 'sdfg';")
)
data_fetched_1: sqlalchemy.Sequence = result_1.fetchall()
print(type(data_fetched_1))
Answers:
Taking a look at the source code,
def fetchall(self) -> Sequence[Row[_TP]]:
"""A synonym for the :meth:`_engine.Result.all` method."""
return self._allrows()
fetchall just returns self._allrows()
And this is the function signature of self._allrows:
(method) def _allrows() -> List[Row[_TP@Result]]
So that is why you’re getting a type list
list is a subtype of the abstract base class Sequence. You cannot annotate a variable with list and then assign a Sequence to it because that Sequence could be something other than a list, such as a tuple for example, which happens to also be a subtype of Sequence.
from collections.abc import Sequence
print(issubclass(list, Sequence)) # True
print(issubclass(tuple, Sequence)) # True
In general, if you expect your variable to behave like any sequence, i.e. support subscripting (via __getitem__) and all the other methods expected of the sequence protocol, you should annotate it as such, meaning with collections.abc.Sequence. But then you cannot expect it to support e.g. list-specific methods like append.
If you want to make sure you are dealing with a list, every Sequence can be turned into a list by simply calling the list constructor on it (because Sequence is a subtype of Iterable).
More to the point, I am not sure where you are getting your type annotations for SQLAlchemy from, but they are likely outdated. My PyCharm shows the signature of Result.fetchall to return list[Row] and so do the latest official sqlalchemy-stubs. (Same for Result.all by the way.) So a variable of type list should definitely accept the output of that method and your type checker is just using wrong type stubs.
The good news is that when SQLAlchemy 2 finally drops, we will no longer have these issues because it will thankfully feature inline type annotations.
PS
Oops, totally missed that version 2 has been out for a while now. Thanks to @sergenp for pointing it out. The current annotations indeed show Sequence[Row[_TP]].
Since the Result class is specified to inherit from ResultInternal[Row[_TP]], which is parameterized by _R and its _allrows method returns a list[_R], the annotations are still technically correct. The fetchall method returns the output of the _allrows method. list[Row[_TP]] is a subtype of Sequence[Row[_TP]].
It is best practice to declare the return type as narrow as possible, but in this case the developers did not. I can only speculate that they might plan to have fetchall return some other sequence object that is not a list in the future and to ensure stability of the API they annotated it with the broader type. You can always narrow the return type later maintaining backwards-compatibility, but you cannot broaden or change it to a different (not descendant) type later because that breaks backwards compatibility. But this is just guessing on my part.
Either way, as users of their API we must take their type hints at face value and expect whatever they say fetchall returns. We must not assume it returns a list, even though it does now. _allrows is a protected method and thus considered implementation detail.
The advice I gave in my original answer still holds: If you are fine with just any sequence, annotate your variable accordingly. If you need a list, convert the output of fetchall to a list explicitly.
The type-hinting of the method sqlalchemy.engine.result.Result.fetchall() suggests the method is supposed to return a Sequence[Row[_TP]] but when I print print(type(result_1.fetchall())), I get a list.
Is there anything I’m missing?
My code is the following:
if __name__ == "__main__":
engine: sqlalchemy.engine.base.Engine = sqlalchemy.create_engine(
"sqlite:///data_SQL/new_db.db", echo=False
)
with engine.connect() as connection:
result_1: sqlalchemy.engine.cursor.CursorResult = connection.execute(
sqlalchemy.text("SELECT * FROM MY_TABLE WHERE batch = 'sdfg';")
)
data_fetched_1: sqlalchemy.Sequence = result_1.fetchall()
print(type(data_fetched_1))
Taking a look at the source code,
def fetchall(self) -> Sequence[Row[_TP]]:
"""A synonym for the :meth:`_engine.Result.all` method."""
return self._allrows()
fetchall just returns self._allrows()
And this is the function signature of self._allrows:
(method) def _allrows() -> List[Row[_TP@Result]]
So that is why you’re getting a type list
list is a subtype of the abstract base class Sequence. You cannot annotate a variable with list and then assign a Sequence to it because that Sequence could be something other than a list, such as a tuple for example, which happens to also be a subtype of Sequence.
from collections.abc import Sequence
print(issubclass(list, Sequence)) # True
print(issubclass(tuple, Sequence)) # True
In general, if you expect your variable to behave like any sequence, i.e. support subscripting (via __getitem__) and all the other methods expected of the sequence protocol, you should annotate it as such, meaning with collections.abc.Sequence. But then you cannot expect it to support e.g. list-specific methods like append.
If you want to make sure you are dealing with a list, every Sequence can be turned into a list by simply calling the list constructor on it (because Sequence is a subtype of Iterable).
More to the point, I am not sure where you are getting your type annotations for SQLAlchemy from, but they are likely outdated. My PyCharm shows the signature of Result.fetchall to return list[Row] and so do the latest official sqlalchemy-stubs. (Same for Result.all by the way.) So a variable of type list should definitely accept the output of that method and your type checker is just using wrong type stubs.
The good news is that when SQLAlchemy 2 finally drops, we will no longer have these issues because it will thankfully feature inline type annotations.
PS
Oops, totally missed that version 2 has been out for a while now. Thanks to @sergenp for pointing it out. The current annotations indeed show Sequence[Row[_TP]].
Since the Result class is specified to inherit from ResultInternal[Row[_TP]], which is parameterized by _R and its _allrows method returns a list[_R], the annotations are still technically correct. The fetchall method returns the output of the _allrows method. list[Row[_TP]] is a subtype of Sequence[Row[_TP]].
It is best practice to declare the return type as narrow as possible, but in this case the developers did not. I can only speculate that they might plan to have fetchall return some other sequence object that is not a list in the future and to ensure stability of the API they annotated it with the broader type. You can always narrow the return type later maintaining backwards-compatibility, but you cannot broaden or change it to a different (not descendant) type later because that breaks backwards compatibility. But this is just guessing on my part.
Either way, as users of their API we must take their type hints at face value and expect whatever they say fetchall returns. We must not assume it returns a list, even though it does now. _allrows is a protected method and thus considered implementation detail.
The advice I gave in my original answer still holds: If you are fine with just any sequence, annotate your variable accordingly. If you need a list, convert the output of fetchall to a list explicitly.