How to build a self-referencing model in Pydantic with dataclasses?
Question:
I am building an API using FastAPI and pydantic.
As I follow DDD / clean architecture, which separates the definition of the model from the definition of the persistence layer, I use standard lib dataclasses in my model and then map them to SQLAlchemy tables using imperative mapping (ie. classical mapping).
This works perfectly :
@dataclass
class User:
name: str
age: int
@pydantic.dataclasses.dataclass
class PydanticUser(User):
...
However, I encounter a problem when defining a class with self-reference.
✅ class inheriting from Pydantic’s BaseModel can self-reference
Inheriting from pydantic’s BaseModel works, however this is not compatible with SQLAlchemy imperative mapping, which I would like to use to stick to clean architecture / DDD principles.
class BaseModelPerson(BaseModel):
name: str
age: int
parent_person: BaseModelPerson = None
BaseModelPerson.update_forward_refs()
john = BaseModelPerson(name="John", age=49, parent_person=None)
tim = BaseModelPerson(name="Tim", age=14, parent_person=john)
print(john)
# BaseModelPerson(name='John', age=49, parent_person=None)
print(tim)
# BaseModelPerson(name='Tim', age=14, parent_person=BaseModelPerson(name='John', age=49, parent_person=None))
✅ Standard lib dataclasses can also self-reference
from __future__ import annotations
from dataclasses import dataclass
@dataclass
class StdlibPerson:
name: str
age: int
parent: StdlibPerson
john = StdlibPerson(name="John", age=49, parent=None)
tim = StdlibPerson(name="Tim", age=14, parent=john)
print(john)
# StdlibPerson(name='John', age=49, parent=None)
print(tim)
# StdlibPerson(name='Tim', age=14, parent=StdlibPerson(name='John', age=49, parent=None))
❌ Pydantic dataclass conversion causes recursion error
The problem occurs when I try to convert the standard library dataclass into a pydantic dataclass.
Defining a Pydantic dataclass like this:
PydanticPerson = pydantic.dataclasses.dataclass(StdlibPerson)
returns an error:
# output (hundreds of lines - that is recursive indeed)
# The name of an attribute on the class where we store the Field
File "pydantic/main.py", line 990, in pydantic.main.create_model
File "pydantic/main.py", line 299, in pydantic.main.ModelMetaclass.__new__
File "pydantic/fields.py", line 411, in pydantic.fields.ModelField.infer
File "pydantic/fields.py", line 342, in pydantic.fields.ModelField.__init__
File "pydantic/fields.py", line 456, in pydantic.fields.ModelField.prepare
File "pydantic/fields.py", line 673, in pydantic.fields.ModelField.populate_validators
File "pydantic/class_validators.py", line 255, in pydantic.class_validators.prep_validators
File "pydantic/class_validators.py", line 238, in pydantic.class_validators.make_generic_validator
File "/usr/lib/python3.9/inspect.py", line 3111, in signature
return Signature.from_callable(obj, follow_wrapped=follow_wrapped)
File "/usr/lib/python3.9/inspect.py", line 2860, in from_callable
return _signature_from_callable(obj, sigcls=cls,
File "/usr/lib/python3.9/inspect.py", line 2323, in _signature_from_callable
return _signature_from_function(sigcls, obj,
File "/usr/lib/python3.9/inspect.py", line 2155, in _signature_from_function
if _signature_is_functionlike(func):
File "/usr/lib/python3.9/inspect.py", line 1883, in _signature_is_functionlike
if not callable(obj) or isclass(obj):
File "/usr/lib/python3.9/inspect.py", line 79, in isclass
return isinstance(object, type)
RecursionError: maximum recursion depth exceeded while calling a Python object
Defining StdlibPerson like this does not solve the problem:
@dataclass
class StdlibPerson
name: str
age: int
parent: "Person" = None
nor does using the second way provided by pydantic documentation:
@pydantic.dataclasses.dataclass
class PydanticPerson(StdlibPerson)
...
❌ using Pydantic dataclasses directly
from __future__ import annotations
from pydantic.dataclasses import dataclass
from typing import Optional
@pydantic.dataclasses.dataclass
class PydanticDataclassPerson:
name: str
age: int
parent: Optional[PydanticDataclassPerson] = None
john = PydanticDataclassPerson(name="John", age=49, parent=None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 6, in __init__
File "pydantic/dataclasses.py", line 97, in pydantic.dataclasses._generate_pydantic_post_init._pydantic_post_init
# | False | | |
File "pydantic/main.py", line 1040, in pydantic.main.validate_model
File "pydantic/fields.py", line 699, in pydantic.fields.ModelField.validate
pydantic.errors.ConfigError: field "parent" not yet prepared so type is still a ForwardRef, you might need to call PydanticDataclassPerson.update_forward_refs().
>>> PydanticDataclassPerson.update_forward_refs()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: type object 'PydanticDataclassPerson' has no attribute 'update_forward_refs'
Question
how can I define a pydantic model with self-referencing objects so that it is compatible with SQLAlchemy imperative mapping ?
Answers:
It seems there are no easy solution to build a REST API with FastAPI and self-referencing objects.
I have decided to switch to the FastAPI / GraphQL stack, with the Strawberry library which is explicitly recommended in the FastAPI documentation.
No problem so far, Strawberry makes it easy to build a GraphQL server and it handles self-referencing object in a breeze.
#!/usr/bin/env python3.10
# src/my_app/entrypoints/api/schema.py
import typing
import strawberry
@strawberry.type
class Person:
name: str
age: int
parent: Person | None
from typing import ForwardRef
from pydantic import BaseModel
Foo = ForwardRef('Foo')
class Foo(BaseModel):
a: int = 123
b: Foo = None
Foo.update_forward_refs()
print(Foo())
#> a=123 b=None
print(Foo(b={'a': '321'}))
#> a=123 b=Foo(a=321, b=None)
You can use it like this.
This is called as Postponed annotations.
Here is the link to read about it more-
https://pydantic-docs.helpmanual.io/usage/postponed_annotations/
Inheriting also works
class Person(BaseModel):
name: str
age: int
class StdlibPerson(Person):
parent: Person = None
I am building an API using FastAPI and pydantic.
As I follow DDD / clean architecture, which separates the definition of the model from the definition of the persistence layer, I use standard lib dataclasses in my model and then map them to SQLAlchemy tables using imperative mapping (ie. classical mapping).
This works perfectly :
@dataclass
class User:
name: str
age: int
@pydantic.dataclasses.dataclass
class PydanticUser(User):
...
However, I encounter a problem when defining a class with self-reference.
✅ class inheriting from Pydantic’s BaseModel can self-reference
Inheriting from pydantic’s BaseModel works, however this is not compatible with SQLAlchemy imperative mapping, which I would like to use to stick to clean architecture / DDD principles.
class BaseModelPerson(BaseModel):
name: str
age: int
parent_person: BaseModelPerson = None
BaseModelPerson.update_forward_refs()
john = BaseModelPerson(name="John", age=49, parent_person=None)
tim = BaseModelPerson(name="Tim", age=14, parent_person=john)
print(john)
# BaseModelPerson(name='John', age=49, parent_person=None)
print(tim)
# BaseModelPerson(name='Tim', age=14, parent_person=BaseModelPerson(name='John', age=49, parent_person=None))
✅ Standard lib dataclasses can also self-reference
from __future__ import annotations
from dataclasses import dataclass
@dataclass
class StdlibPerson:
name: str
age: int
parent: StdlibPerson
john = StdlibPerson(name="John", age=49, parent=None)
tim = StdlibPerson(name="Tim", age=14, parent=john)
print(john)
# StdlibPerson(name='John', age=49, parent=None)
print(tim)
# StdlibPerson(name='Tim', age=14, parent=StdlibPerson(name='John', age=49, parent=None))
❌ Pydantic dataclass conversion causes recursion error
The problem occurs when I try to convert the standard library dataclass into a pydantic dataclass.
Defining a Pydantic dataclass like this:
PydanticPerson = pydantic.dataclasses.dataclass(StdlibPerson)
returns an error:
# output (hundreds of lines - that is recursive indeed)
# The name of an attribute on the class where we store the Field
File "pydantic/main.py", line 990, in pydantic.main.create_model
File "pydantic/main.py", line 299, in pydantic.main.ModelMetaclass.__new__
File "pydantic/fields.py", line 411, in pydantic.fields.ModelField.infer
File "pydantic/fields.py", line 342, in pydantic.fields.ModelField.__init__
File "pydantic/fields.py", line 456, in pydantic.fields.ModelField.prepare
File "pydantic/fields.py", line 673, in pydantic.fields.ModelField.populate_validators
File "pydantic/class_validators.py", line 255, in pydantic.class_validators.prep_validators
File "pydantic/class_validators.py", line 238, in pydantic.class_validators.make_generic_validator
File "/usr/lib/python3.9/inspect.py", line 3111, in signature
return Signature.from_callable(obj, follow_wrapped=follow_wrapped)
File "/usr/lib/python3.9/inspect.py", line 2860, in from_callable
return _signature_from_callable(obj, sigcls=cls,
File "/usr/lib/python3.9/inspect.py", line 2323, in _signature_from_callable
return _signature_from_function(sigcls, obj,
File "/usr/lib/python3.9/inspect.py", line 2155, in _signature_from_function
if _signature_is_functionlike(func):
File "/usr/lib/python3.9/inspect.py", line 1883, in _signature_is_functionlike
if not callable(obj) or isclass(obj):
File "/usr/lib/python3.9/inspect.py", line 79, in isclass
return isinstance(object, type)
RecursionError: maximum recursion depth exceeded while calling a Python object
Defining StdlibPerson like this does not solve the problem:
@dataclass
class StdlibPerson
name: str
age: int
parent: "Person" = None
nor does using the second way provided by pydantic documentation:
@pydantic.dataclasses.dataclass
class PydanticPerson(StdlibPerson)
...
❌ using Pydantic dataclasses directly
from __future__ import annotations
from pydantic.dataclasses import dataclass
from typing import Optional
@pydantic.dataclasses.dataclass
class PydanticDataclassPerson:
name: str
age: int
parent: Optional[PydanticDataclassPerson] = None
john = PydanticDataclassPerson(name="John", age=49, parent=None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 6, in __init__
File "pydantic/dataclasses.py", line 97, in pydantic.dataclasses._generate_pydantic_post_init._pydantic_post_init
# | False | | |
File "pydantic/main.py", line 1040, in pydantic.main.validate_model
File "pydantic/fields.py", line 699, in pydantic.fields.ModelField.validate
pydantic.errors.ConfigError: field "parent" not yet prepared so type is still a ForwardRef, you might need to call PydanticDataclassPerson.update_forward_refs().
>>> PydanticDataclassPerson.update_forward_refs()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: type object 'PydanticDataclassPerson' has no attribute 'update_forward_refs'
Question
how can I define a pydantic model with self-referencing objects so that it is compatible with SQLAlchemy imperative mapping ?
It seems there are no easy solution to build a REST API with FastAPI and self-referencing objects.
I have decided to switch to the FastAPI / GraphQL stack, with the Strawberry library which is explicitly recommended in the FastAPI documentation.
No problem so far, Strawberry makes it easy to build a GraphQL server and it handles self-referencing object in a breeze.
#!/usr/bin/env python3.10
# src/my_app/entrypoints/api/schema.py
import typing
import strawberry
@strawberry.type
class Person:
name: str
age: int
parent: Person | None
from typing import ForwardRef
from pydantic import BaseModel
Foo = ForwardRef('Foo')
class Foo(BaseModel):
a: int = 123
b: Foo = None
Foo.update_forward_refs()
print(Foo())
#> a=123 b=None
print(Foo(b={'a': '321'}))
#> a=123 b=Foo(a=321, b=None)
You can use it like this.
This is called as Postponed annotations.
Here is the link to read about it more-
https://pydantic-docs.helpmanual.io/usage/postponed_annotations/
Inheriting also works
class Person(BaseModel):
name: str
age: int
class StdlibPerson(Person):
parent: Person = None