[RFC] Add optional automatic validation and conversion

[sscherfke]

Validation and conversion of attribute values is currently only done if the user passes the corresponding arguments to attrib().

In contrast, libs like Pydantic automatically validate and convert all values (in __init__() as well as __setattr__()). It also allows the user to easily pass the most common validators like ge=0 or maxlength=50.

This lets you do fancy stuff like this, which is very useful, e.g., for REST API server models and clients, or generally for loading nested JSON data. Settings/config management is another usecase:

>>> from dateatime import datetime                                                    
>>> from pydantic import BaseModel                                                    
>>>                                                                                   
>>> class Child(BaseModel):                                                           
...     x: int                                                                        
...     y: int                                                                        
...     d: datetime                                                                   
...                                                                                   
>>> class Parent(BaseModel):                                                          
...     name: str                                                                     
...     child: Child                                                                  
...                                                                                   
>>> data = {                                                                          
...     'name': 'spam',                                                               
...     'child': {                                                                    
...         'x': 23,                                                                  
...         'y': '42',  # sic!                                                        
...         'd': '2020-05-04T13:37:00',                                               
...     },                                                                            
... }                                                                                 
>>> Parent(**data)                                                                    
Parent(name='spam', child=Child(x=23, y=42, d=datetime.datetime(2020, 5, 4, 13, 37))) 

While pydantic works quite nicely, it is way slower than attrs and also uses inheritance which is not always what users may want.

I tried implementing a similar functionality but failed. IMHO, there are some changes needed directly within attr’s class-buildng facilities, as the following example illustrates:

from datetime import datetime
from functools import partial

import attr


# Two examples for pre-defined validators for later use
def check_ge(inst, attr, value, bound):
    if value < bound:
        raise ValueError(f'"{attr.name}" is not >= {bound}: {value}')

def check_le(inst, attr, value, bound):
    if value > bound:
        raise ValueError(f'"{attr.name}" is not <= {bound}: {value}')


def validating_class(auto_converters=True, **kwargs):
    # Extension wrapper for "attrs()"
    def wrap(cls):
        # if auto_converters:
        #   Here, I'd have to manually iterate/evaluate all attribs.
        #   This duplicates work in "attr.s()".
        cls = attr.s(auto_attribs=True, frozen=True, slots=True, **kwargs)(cls)
        # if auto_converters:
        #   It is very hard (impossible?) to update "cls" here and add (optional) 
        #   converters to all attributes
        return cls
    return wrap


def field(default, ge=None, le=None, **kwargs):
    # Extension wrapper for "field()"
    v = []
    # This does not work, b/c I cannot access type information here
    # if attr_type is int:
    #     if ge is not None:
    #         v.append(partial(check_ge, bound=ge))
    #     if le is not None:
    #         v.append(partial(check_le, bound=le))

    if 'validator' in kwargs:
        v.append(validator)
    if v:
        kwargs['validator'] = v
    return attr.ib(converter=int, validator=v)


@validating_class()
class Child:
    x: int = field(ge=0, le=100)
    y: int = field(ge=0, le=100)
    d: datetime  # auto-add "field(converter=datertime.fromisoformat)"


@validating_class()
class Parent:
    name: str
    child: Child = # auto-add "field(converter=lambda d: Child(**d))"


data = {'name': 'spam', 'child': {'x': 23, 'y': '42', 'd': '2020-05-04T13:37:00'}}
p = Parent(**data)
print(p)
# Parent(name='spam', child=Child(x=-1, y=0, datetime(2020, 5, 4, 13, 37)))

I already talked about this with @hynek and he asked me create this issue.

I also created an issue for orjson asking to add support for attrs.

Summary (what’s needed)

• Optional conversions (and validation) of all attributes. Conversion should only happen when necessary (e.g., do not list(value) if value is already a list). This is the most important part.
• Shortcuts for commonly used validators: attrib(ge=0, le=10).
• An asjson() function that uses stdlib and provides an encoder that can handle attrs classes and datetimes and similar common types.
• Explicit orjson support might not be needed when Add attrs support ijl/orjson#92 gets implemented

[wsanchez]

Thanks for writing this up

[sscherfke]

I added a summary to the issue.

If you have no genereal objections against this issue, I can start working on it and see how far I get.

[Tinche]

Hi,

in my personal opinion, structuring and conversion should not be part of the class but of something else. So I've created a library called cattrs that implements the something else (called a converter).

For example, in your writeup you're using something that looks like ISO 8601 for the datetime instance, but I think you'd be hard pressed to come to a consensus on this (why not unix timestamps?). Also if you want to use the pendulum DateTime instead, where does that logic live?

So pydantic is not equivalent to attrs, it's equivalent to attrs+cattrs. I'm not saying everyone should use cattrs (and I'm struggling to maintain it myself, if you look at the issues page...), I'm saying if we want this functionality in a blessed way for attrs, it should be another package under the attrs umbrella or another module in the attrs repository, but it should definitely be another component and not part of attrs as it exists now.

[hynek]

Super busy right now, but just to put a rough stake in the ground: I'm 100% interested in making it possible to write a package like pydantic using attrs (aka provide hooks and tools). But structuring and conversion remains out of scope for attrs proper.

[sscherfke]

I have tried cattrs and it is even faster than pydantic (except if you use dateutil’s parse()), but attrs+cattrs is imho lacking in user experience. You just have to learn and type too much in order to get basic conversion and evaluation done.

I want something, that is super easy to use for most use cases (and that makes the uncommon use cases possible, of course ;-)) and that is backed by attrs.

I don't necessarily want to create a new package. I’d be perfectly happy if attrs implemented the required hooks and if cattrs added another layer of abstraction for a UX like in the example above. :)

[fbergroth]

I have tried cattrs and it is even faster than pydantic (except if you use dateutil’s parse()),

@sscherfke That got me interested, so I replaced dateutil with ciso8601 in pydantics benchmarks. You can see the results here: pydantic/pydantic#1568.

[Tinche]

@sscherfke So cattrs grew out of a set of converters doing what you suggest, actually. I just found that approach ultimately too limiting.

The simplest solution would be to just include a converter like:

def ensure_cls(cls):
    """If the attribute is an instance of cls, pass, else try constructing."""
    def converter(val):
        if isinstance(val, cls):
            return val
        else:
            return cls(**val)
    return converter

in attrs proper.

Your converter (converter=lambda d: Child(**d)) wouldn't work for passing in an existing instance of Child, though. The real type of the field then becomes a Union[Child, Dict[str, Any]] and this complicates Mypy integration. People will want an ensure_enum later too...

[sscherfke]

@fbergroth

That got me interested, so I replaced dateutil with ciso8601 in pydantics benchmarks. You can see the results here: pydantic/pydantic#1568.

I discovered this while writing an article about attrs, data classes and pydantic which will be published later this week. I just replaced datetuil’s parse with datetime.fromisoformat() :)

@Tinche

So cattrs grew out of a set of converters doing what you suggest, actually. I just found that approach ultimately too limiting.

My idea is to offer a limited set of the most commonly used functionality via a very simple (and possibly limited) API but also provide a more powerful but also more complex API.

The main idea (and problem) is that you do not have to pass a converter to attrib() and that attrs automatically chooses one for you based on the attributes type:

# Just a rough example :)
for attrib in attribs:
    if has(attrib.type):
        c = lambda d: attrib.type(**d)
    elif issubclass(attrib.type, datetime):
        c = datetime.fromisoformat
    else:
        c = attrib.type
    
    # only apply converter if instance is not already of correct type:
    attrib.converter = lambda i: i if isinstance(i, attr.type) else c(i)

The real type of the field then becomes a Union[Child, Dict[str, Any]] and this complicates Mypy integration.

🤔 Yes, this may indeed be a problem. However, I just tested this in pydantic and mypy has no problems with this example:

from pydantic import BaseModel


class Child(BaseModel):
    x: int
    y: int


class Parent(BaseModel):
    name: str
    child: Child


x = Parent(name='spam', child=Child(x=1, y=2))
y = Parent(name='spam', child=Child(x=1, y='2'))  # y is a str!
d = {'name': 'spam', 'child': {'x': 1, 'y': 2}}
z = Parent(**d)  # As when loaded from json
ä = Parent(name=d['name'], child=d['child'])  # Just for fun :)

assert x == y
assert x == z
assert x == ä

[hynek]

I need to point out that datetime.fromisoformat() is absolutely not an adequate parser for ISO strings, as the Stdlib docs clearly say. It’s merely the inverse of datetime.isoformat().

[wsanchez]

I need to point out that datetime.fromisoformat() is absolutely not an adequate parser for ISO strings, as the Stdlib docs clearly say. It’s merely the inverse of datetime.isoformat().

That is (unfortunately) correct. A library like arrow can: arrow.parser.DateTimeParser().parse_iso(text)

[sscherfke]

I’m aware that fromisoformat() is very limited and that there are other/better parsers (like the ones arrow or pendulum provide). But if the use case is that you only transmit iso-dates generated and parsed by the stdlib, fromisoformat() works perfectly fine. Furhtermore, it is not yet time to discuss a proper parser for my suggested changes. ;-)

[sscherfke]

This RFC is the result of an article that I have been writing and that I now finally published: https://stefan.sofa-rockers.org/2020/05/29/attrs-dataclasses-pydantic/

[hynek]

Stefan mentioned to me in private he might get it to work using cattrs so this will wait for now.

I’d like to point out one thing tho: pydantic has a great DX but it comes at the price of correctness and flexibility. That is a very valid tradeoff to make and we can see people loving it for that.

However it’s not a trade-off I am willing to make. Correctness, explicitness, and flexibility are more important to me than typing extra characters. We already have a pydantic, so let’s try to be an actual alternative and not try to imitate them.

(JFTR, I’m not saying anyone wanted that; I’m just writing it out explicitly for clarity so please don’t feel anyone attacked :))

I really hope cattrs can rise to the task because it really seems to have its fans!

[sscherfke]

I needed a quick solution at work that works "good enough" and does not require changes within (c)attrs, so I just created another layer of abstraction above (c)attrs. 🙃

Here is an example:

from datetime import datetime

from mattr import (  # magic attrs – the only thing that mattrs :D
    frozen,  # attrs(auto_attrib, slots, frozen, kw_only)
    field,  # Wrapper around attrib() w/ validation shortcuts
    evolve,  # attrs.evolve()
    fromdict,  # cattrs.structure() with custom converter
    asdict,   # attrs.asdict()
    fromjson,   # cattrs.structure(json.loads()) with custom converter
    asjson,  #  json.dumps(cattrs.unstructure()) with custom converter
)


@frozen()
class Child:
    x: int = field(ge=0)
    y: int = field(ge=0)
    d: datetime


@frozen()
class Parent:
    name: str = field(re='^[a-c]+$', max_length=3)
    child: Child



DATA = {
    'name': 'abc',
    'child': {
        'x': 23,
        'y': '42',  # sic!
        'd': '2020-05-04T13:37:00',
    },
}


p1 = fromdict(DATA, Parent)
print(asdict(p1))

p_json = asjson(p1)
print(p_json)
p2 = fromjson(p_json, Parent)
assert p1 == p2

try:
    p3 = evolve(p2, name='spam')
except ValueError as e:
    print(e)  # ValueError: "name" exceeds the maximum length of 3: 4

The UX is okay, I think. I don’t know how much I (dis)like fromXYZ(data, cls) style. It feels a bit clunky, but looks quite functional and involves less magic than cls(**data). 🤷

Validation is done by attrs, conversion is done by cattrs. I still think that it would maybe be nice if we could teach attrs some basic auto-conversion rules based on the attribute type.

Performance for structuring + validation is a bit slower than just attrs+cattrs but I did not optimizie this yet. Dumping is as fast a attrs+cattrs.