Make every fields as optional with Pydantic

Question:

I’m making an API with FastAPI and Pydantic.

I would like to have some PATCH endpoints, where 1 or N fields of a record could be edited at once. Moreover, I would like the client to only pass the necessary fields in the payload.

Example:

class Item(BaseModel):
    name: str
    description: str
    price: float
    tax: float


@app.post("/items", response_model=Item)
async def post_item(item: Item):
    ...

@app.patch("/items/{item_id}", response_model=Item)
async def update_item(item_id: str, item: Item):
    ...

In this example, for the POST request, I want every field to be required. However, in the PATCH endpoint, I don’t mind if the payload only contains, for example, the description field. That’s why I wish to have all fields as optional.

Naive approach:

class UpdateItem(BaseModel):
    name: Optional[str] = None
    description: Optional[str] = None
    price: Optional[float] = None
    tax: Optional[float]

But that would be terrible in terms of code repetition.

Any better option?

Asked By: nolwww

||

Answers:

The problem is once FastAPI sees item: Item in your route definition, it will try to initialize an Item type from the request body, and you can’t declare your model’s fields to be optional sometimes depending on some conditional, such as depending on which route it is used.

I have 3 solutions:

Solution #1: Separate Models

I would say that having separate models for the POST and PATCH payloads seems to be the more logical and readable approach. It might lead to duplicated code, yes, but I think clearly defining which route has an all-required or an all-optional model balances out the maintainability cost.

The FastAPI docs has a section for partially updating models with PUT or PATCH that uses Optional fields, and there’s a note at the end that says something similar:

Notice that the input model is still validated.

So, if you want to receive partial updates that can omit all the attributes, you need to have a model with all the attributes marked as optional (with default values or None).

So…

class NewItem(BaseModel):
    name: str
    description: str
    price: float
    tax: float

class UpdateItem(BaseModel):
    name: Optional[str] = None
    description: Optional[str] = None
    price: Optional[float] = None
    tax: Optional[float] = None

@app.post('/items', response_model=NewItem)
async def post_item(item: NewItem):
    return item

@app.patch('/items/{item_id}',
           response_model=UpdateItem,
           response_model_exclude_none=True)
async def update_item(item_id: str, item: UpdateItem):
    return item

Solution #2: Declare as All-Required, but Manually Validate for PATCH

You can define your model to have all-required fields, then define your payload as a regular Body parameter on the PATCH route, and then initialize the actual Item object "manually" depending on what’s available in the payload.

from fastapi import Body
from typing import Dict

class Item(BaseModel):
    name: str
    description: str
    price: float
    tax: float

@app.post('/items', response_model=Item)
async def post_item(item: Item):
    return item

@app.patch('/items/{item_id}', response_model=Item)
async def update_item(item_id: str, payload: Dict = Body(...)):
    item = Item(
        name=payload.get('name', ''),
        description=payload.get('description', ''),
        price=payload.get('price', 0.0),
        tax=payload.get('tax', 0.0),
    )
    return item

Here, the Item object is initialized with whatever is in the payload, or some default if there isn’t one. You’ll have to manually validate if none of the expected fields are passed, ex.:

from fastapi import HTTPException

@app.patch('/items/{item_id}', response_model=Item)
async def update_item(item_id: str, payload: Dict = Body(...)):
    # Get intersection of keys/fields
    # Must have at least 1 common
    if not (set(payload.keys()) & set(Item.__fields__)):
        raise HTTPException(status_code=400, detail='No common fields')
    ...
$ cat test2.json
{
    "asda": "1923"
}
$ curl -i -H'Content-Type: application/json' --data @test2.json --request PATCH localhost:8000/items/1
HTTP/1.1 400 Bad Request
content-type: application/json

{"detail":"No common fields"}

The behavior for the POST route is as expected: all the fields must be passed.

Solution #3: Declare as All-Optional But Manually Validate for POST

Pydantic’s BaseModel‘s dict method has exclude_defaults and exclude_none options for:

  • exclude_defaults: whether fields which are equal to their default values (whether set or otherwise) should be excluded from the returned dictionary; default False

  • exclude_none: whether fields which are equal to None should be excluded from the returned dictionary; default False

This means, for both POST and PATCH routes, you can use the same Item model, but now with all Optional[T] = None fields. The same item: Item parameter can also be used.

class Item(BaseModel):
    name: Optional[str] = None
    description: Optional[str] = None
    price: Optional[float] = None
    tax: Optional[float] = None

On the POST route, if not all the fields were set, then exclude_defaults and exclude_none will return an incomplete dict, so you can raise an error. Else, you can use the item as your new Item.

@app.post('/items', response_model=Item)
async def post_item(item: Item):
    new_item_values = item.dict(exclude_defaults=True, exclude_none=True)

    # Check if exactly same set of keys/fields
    if set(new_item_values.keys()) != set(Item.__fields__):
        raise HTTPException(status_code=400, detail='Missing some fields..')

    # Use `item` or `new_item_values`
    return item
$ cat test_empty.json
{
}
$ curl -i -H'Content-Type: application/json' --data @test_empty.json --request POST localhost:8000/items
HTTP/1.1 400 Bad Request
content-type: application/json

{"detail":"Missing some fields.."}

$ cat test_incomplete.json 
{
    "name": "test-name",
    "tax": 0.44
}
$ curl -i -H'Content-Type: application/json' --data @test_incomplete.json --request POST localhost:8000/items
HTTP/1.1 400 Bad Request
content-type: application/json

{"detail":"Missing some fields.."}

$ cat test_ok.json
{
    "name": "test-name",
    "description": "test-description",
    "price": 123.456,
    "tax": 0.44
}
$ curl -i -H'Content-Type: application/json' --data @test_ok.json --request POST localhost:8000/items
HTTP/1.1 200 OK
content-type: application/json

{"name":"test-name","description":"test-description","price":123.456,"tax":0.44}

On the PATCH route, if at least 1 value is not default/None, then that will be your update data. Use the same validation from Solution 2 to fail if none of the expected fields were passed in.

@app.patch('/items/{item_id}', response_model=Item)
async def update_item(item_id: str, item: Item):
    update_item_values = item.dict(exclude_defaults=True, exclude_none=True)

    # Get intersection of keys/fields
    # Must have at least 1 common
    if not (set(update_item_values.keys()) & set(Item.__fields__)):
        raise HTTPException(status_code=400, detail='No common fields')

    update_item = Item(**update_item_values)

    return update_item
$ cat test2.json
{
    "asda": "1923"
}
$ curl -i -s -H'Content-Type: application/json' --data @test2.json --request PATCH localhost:8000/items/1
HTTP/1.1 400 Bad Request
content-type: application/json

{"detail":"No common fields"}

$ cat test2.json
{
    "description": "test-description"
}
$ curl -i -s -H'Content-Type: application/json' --data @test2.json --request PATCH localhost:8000/items/1
HTTP/1.1 200 OK
content-type: application/json

{"name":null,"description":"test-description","price":null,"tax":null}
Answered By: Gino Mempin

Solution with metaclasses

I’ve just come up with the following:


class AllOptional(pydantic.main.ModelMetaclass):
    def __new__(cls, name, bases, namespaces, **kwargs):
        annotations = namespaces.get('__annotations__', {})
        for base in bases:
            annotations.update(base.__annotations__)
        for field in annotations:
            if not field.startswith('__'):
                annotations[field] = Optional[annotations[field]]
        namespaces['__annotations__'] = annotations
        return super().__new__(cls, name, bases, namespaces, **kwargs)

Use it as:

class UpdatedItem(Item, metaclass=AllOptional):
    pass

So basically it replace all non optional fields with Optional

Any edits are welcome!

With your example:

from typing import Optional

from fastapi import FastAPI
from pydantic import BaseModel
import pydantic

app = FastAPI()

class Item(BaseModel):
    name: str
    description: str
    price: float
    tax: float


class AllOptional(pydantic.main.ModelMetaclass):
    def __new__(self, name, bases, namespaces, **kwargs):
        annotations = namespaces.get('__annotations__', {})
        for base in bases:
            annotations.update(base.__annotations__)
        for field in annotations:
            if not field.startswith('__'):
                annotations[field] = Optional[annotations[field]]
        namespaces['__annotations__'] = annotations
        return super().__new__(self, name, bases, namespaces, **kwargs)

class UpdatedItem(Item, metaclass=AllOptional):
    pass

# This continues to work correctly
@app.get("/items/{item_id}", response_model=Item)
async def get_item(item_id: int):
    return {
        'name': 'Uzbek Palov',
        'description': 'Palov is my traditional meal',
        'price': 15.0,
        'tax': 0.5,
    }

@app.patch("/items/{item_id}") # does using response_model=UpdatedItem makes mypy sad? idk, i did not check
async def update_item(item_id: str, item: UpdatedItem):
    return item
Answered By: Drdilyor

Modified @Drdilyor solution.
Added checking for nesting of models.

from pydantic.main import ModelMetaclass, BaseModel
from typing import Any, Dict, Optional, Tuple

class _AllOptionalMeta(ModelMetaclass):
    def __new__(self, name: str, bases: Tuple[type], namespaces: Dict[str, Any], **kwargs):
        annotations: dict = namespaces.get('__annotations__', {})

        for base in bases:
            for base_ in base.__mro__:
                if base_ is BaseModel:
                    break

                annotations.update(base_.__annotations__)

        for field in annotations:
            if not field.startswith('__'):
                annotations[field] = Optional[annotations[field]]

        namespaces['__annotations__'] = annotations

        return super().__new__(mcs, name, bases, namespaces, **kwargs)
Answered By: Maxim

Thanks @Drdilyor for a great solution. I’ve made a version that lets you define required arguments in the child class (like the Id of the id of the item you want to update for example) :

class AllOptional(ModelMetaclass):
def __new__(self, name, bases, namespaces, **kwargs):
    annotations = namespaces.get('__annotations__', {})
    for base in bases:
        optionals = {
            key: Optional[value] if not key.startswith('__') else value for key, value in base.__annotations__.items()
        }
        annotations.update(optionals)

    namespaces['__annotations__'] = annotations
    return super().__new__(self, name, bases, namespaces, **kwargs)
Answered By: Toussaint Behaghel

For my case creating a new class was the only solution that worked, but packed into a function it is quite convenient:

from pydantic import BaseModel, create_model
from typing import Optional
from functools import lru_cache

@lru_cache(maxsize=None) # avoids creating many classes with same name
def make_optional(baseclass: Type[BaseModel]) -> Type[BaseModel]:
    # Extracts the fields and validators from the baseclass and make fields optional
    fields = baseclass.__fields__
    validators = {'__validators__': baseclass.__validators__}
    optional_fields = {key: (Optional[item.type_], None)
                       for key, item in fields.items()}
    return create_model(f'{baseclass.__name__}Optional', **optional_fields,
                        __validators__=validators)

class Item(BaseModel):
    name: str
    description: str
    price: float
    tax: float

ItemOptional = make_optional(Item)

Comparing after and before:

> Item.__fields__

{'name': ModelField(name='name', type=str, required=True),
 'description': ModelField(name='description', type=str, required=True),
 'price': ModelField(name='price', type=float, required=True),
 'tax': ModelField(name='tax', type=float, required=True)}

> ItemOptional.__fields__

{'name': ModelField(name='name', type=Optional[str], required=False, default=None),
 'description': ModelField(name='description', type=Optional[str], required=False, default=None),
 'price': ModelField(name='price', type=Optional[float], required=False, default=None),
 'tax': ModelField(name='tax', type=Optional[float], required=False, default=None)}

It does work, and also it allows you to filter out some fields in the dict_comprehension if it is required.

Moreover in fastapi this approach allows you to do something like this:

@app.post("/items", response_model=Item)
async def post_item(item: Item = Depends()):
    ...

@app.patch("/items/{item_id}", response_model=Item)
async def update_item(item_id: str, item: make_optional(Item) = Depends()):
    ...

Which reduces a lot the boilerplate, using the same approach you can also make a function that makes optional the fields and also exclude a field in case your Item has an ID field,the id would will be repeated in your PATCH call. That can be solved like this:

def make_optional_no_id(baseclass):
    ... # same as make optional
    optional_fields = {key: (Optional[item.type_], None) 
                       for key, item in fields.items() if key != 'ID'} # take out here ID
    ... # you can also take out also validators of ID

@app.patch("/items/{item_id}", response_model=Item)
async def update_item(item: make_optional_no_id(Item) = Depends()):
Answered By: Ziur Olpa

Using a decorator

Using separate models seems like a bad idea for large projects. Lots of effectively duplicated code making it much harder to maintain. The goal of this is reusability and flexibility

from typing import Optional, get_type_hints, Type

from pydantic import BaseModel


def make_optional(
    include: Optional[list[str]] = None,
    exclude: Optional[list[str]] = None,
):
    """Return a decorator to make model fields optional"""

    if exclude is None:
        exclude = []

    # Create the decorator
    def decorator(cls: Type[BaseModel]):
        type_hints = get_type_hints(cls)
        fields = cls.__fields__
        if include is None:
            fields = fields.items()
        else:
            # Create iterator for specified fields
            fields = ((name, fields[name]) for name in include if name in fields)
            # Fields in 'include' that are not in the model are simply ignored, as in BaseModel.dict
        for name, field in fields:
            if name in exclude:
                continue
            if not field.required:
                continue
            # Update pydantic ModelField to not required
            field.required = False
            # Update/append annotation
            cls.__annotations__[name] = Optional[type_hints[name]]
        return cls

    return decorator

Usage

In the context of fast-api models

class ModelBase(pydantic.BaseModel):
  a: int
  b: str


class ModelCreate(ModelBase):
  pass

# Make all fields optional
@make_optional()
class ModelUpdate(ModelBase):
  pass
  • By default, all fields are made optional.
  • include specifies which fields to make optional; all other fields remain unchanged.
  • exclude specifies which fields not to affect.
  • exclude takes precedence over include.
# Make only `a` optional
@make_optional(include=["a"])
class ModelUpdate(ModelBase):
  pass

# Make only `b` optional
@make_optional(exclude=["a"])
class ModelUpdate(ModelBase):
  pass

Note: pydantic appears to make copies of the fields when you inherit from a base class, which is why it’s ok to change them in-place

Answered By: mishnea

⛔️ Pay attention, first solution, prevent Fields validation.

Thanks @Drdilyor for the proposed solution but it seems that it cancel all the fields validation
let’s say you have :

from typing import Optional
import pydantic
from pydantic import BaseModel, Field

class AllOptional(pydantic.main.ModelMetaclass):
    def __new__(self, name, bases, namespaces, **kwargs):
        annotations = namespaces.get('__annotations__', {})
        for base in bases:
            annotations.update(base.__annotations__)
        for field in annotations:
            if not field.startswith('__'):
                annotations[field] = Optional[annotations[field]]
        namespaces['__annotations__'] = annotations
        return super().__new__(self, name, bases, namespaces, **kwargs)

class A(BaseModel):
    a:int = Field(gt=1)

class AO(A, metaclass=AllOptional):
    pass

AO(a=-1) # This will pass through the validation even that it's wrong ⛔️

A simple alternative

class AllOptional(pydantic.main.ModelMetaclass):
    def __new__(mcls, name, bases, namespaces, **kwargs):
        cls = super().__new__(mcls, name, bases, namespaces, **kwargs)
        for field in cls.__fields__.values():
            field.required=False
        return cls
Answered By: Amine Bk

This simple trick works for me: build a new model class dynamically and modify fields to be optional as needed.

def make_partial_model(model: Type[BaseModel], optional_fields: Optional[list[str]] = None) -> Type[BaseModel]:
    class NewModel(model):
        ...

    for field in NewModel.__fields__.values():
        if not optional_fields or field in optional_fields:
            field.required = False

    NewModel.__name__ = f'Partial{model.__name__}'
    return NewModel

PartialRequest = cast(Type[RequestModel], make_partial_model(RequestModel))
Answered By: Namoshizun