Skip to content

Commit 66f27e3

Browse files
ml-evsJPBergsmaCasperWA
committed
Add models for per-entry & per-property metadata (#2167)
* Addded meta field to entry model * Added validator for meta field. * Update pyyaml version in requirements.txt * Update requirements.txt * Update requirements.txt * Remove test structures to get an idea of what triggers yaml import error on github. * Readding meta field to test_good_structures. * Readding meta field to test_structures.json. * Added handling None for property_metadata to validator + small correctionds. * Added test for validator per entry meta field. * Added test for presence metadata field in test_structures.py. * Remove metadata fields when the fields that they belong to are not returned. * add extra test for bad prefix. * Test if dependancy conflict causes error. * Revert "Test if dependancy conflict causes error." This reverts commit 6251e48. * Revert "Revert "Test if dependancy conflict causes error."" This reverts commit bf9692a. * correct version httpx. * commenting out validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue github. * Slowly reassembling validator to see if this resolves issue Github. * Slowly reassembling validator to see if this resolves issue Github. * Slowly reassembling validator to see if this resolves issue Github.. * Slowly reassembling validator to see if this resolves issue Github.. * Placed yaml import in try except block. * Added more cases to test data and added bugfix for removing associated metadata when a field is excluded. * Removed seemingly unneccesary mypy ignore exception statement. * remove change in version pyyaml in requirements.txt. * correct spelling mistake * moved starts_with_supported_prefix and check_starts_with_supported_prefix functions to BaseresouceMapper. * Expanded docstring check_starts_with_supported_prefix * Expanded docstring check_starts_with_supported_prefix * Added return type for starts_with_supported_prefix and check_starts_with_supported_prefix. * Added return type for starts_with_supported_prefix and check_starts_with_supported_prefix. * Adjusted validators baseed on suggestion Matthew. Co-authored-by: ml-evs [email protected] * small corrections validators. * small correction in removing unrequested metadata. * Added spaces before values 'exmpl_originates_from _project'. * update openapijson * update openapijson * update openapijson * use lstrip instead of manually removing / one at a time. * Update optimade/models/entries.py update description property_metadata field Co-authored-by: Matthew Evans <[email protected]> * Update description per entry meta datafield. * update openapijson. * try to see if moving yaml is still necessary. * Revert "try to see if moving yaml is still necessary." This reverts commit 68f14d3. * Added supported_prefixes field to config.py * Update requirements-client.txt (#1813) removed duplicate requirement * Modernize all Python 3.8 annotations (#1815) * Use Python 3.9 as the 'base' CI version for linting * Update pre-commit hooks * Run `pyupgrade --py39-plus` to upgrade legacy annotations * Add `--exit-non-zero-on-fix` for ruff Co-authored-by: Casper Welzel Andersen <[email protected]> * Use f-string over format Co-authored-by: Casper Welzel Andersen <[email protected]> --------- Co-authored-by: Casper Welzel Andersen <[email protected]> * Update pydantic validators and model configs for 2024 * Remove validation of 'supported prefix' in lieu of just checking leading underscores --------- Co-authored-by: Johan Bergsma <[email protected]> Co-authored-by: Casper Welzel Andersen <[email protected]>
1 parent c69484b commit 66f27e3

File tree

10 files changed

+257
-27
lines changed

10 files changed

+257
-27
lines changed

‎openapi/index_openapi.json‎

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -477,6 +477,19 @@
477477
"title": "BaseRelationshipResource",
478478
"description": "Minimum requirements to represent a relationship resource"
479479
},
480+
"EntryMetadata": {
481+
"properties": {
482+
"property_metadata": {
483+
"type": "object",
484+
"title": "Property Metadata",
485+
"description": "An object containing per-entry and per-property metadata. The keys are the names of the fields in attributes for which metadata is available. The values belonging to these keys are dictionaries containing the relevant metadata fields. See also [Metadata properties](https://github.com/Materials-Consortia/OPTIMADE/blob/develop/optimade.rst#metadata-properties)"
486+
}
487+
},
488+
"additionalProperties": true,
489+
"type": "object",
490+
"title": "EntryMetadata",
491+
"description": "Contains the metadata for the attributes of an entry"
492+
},
480493
"EntryRelationships": {
481494
"properties": {
482495
"references": {
@@ -536,13 +549,13 @@
536549
"meta": {
537550
"anyOf": [
538551
{
539-
"$ref": "#/components/schemas/Meta"
552+
"$ref": "#/components/schemas/EntryMetadata"
540553
},
541554
{
542555
"type": "null"
543556
}
544557
],
545-
"description": "a meta object containing non-standard meta-information about a resource that can not be represented as an attribute or relationship."
558+
"description": "A [JSON API meta object](https://jsonapi.org/format/1.1/#document-meta) that is used to communicate metadata."
546559
},
547560
"attributes": {
548561
"$ref": "#/components/schemas/EntryResourceAttributes",
@@ -1324,13 +1337,13 @@
13241337
"meta": {
13251338
"anyOf": [
13261339
{
1327-
"$ref": "#/components/schemas/Meta"
1340+
"$ref": "#/components/schemas/EntryMetadata"
13281341
},
13291342
{
13301343
"type": "null"
13311344
}
13321345
],
1333-
"description": "a meta object containing non-standard meta-information about a resource that can not be represented as an attribute or relationship."
1346+
"description": "A [JSON API meta object](https://jsonapi.org/format/1.1/#document-meta) that is used to communicate metadata."
13341347
},
13351348
"attributes": {
13361349
"$ref": "#/components/schemas/LinksResourceAttributes",

‎openapi/openapi.json‎

Lines changed: 21 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1838,6 +1838,19 @@
18381838
],
18391839
"title": "EntryInfoResponse"
18401840
},
1841+
"EntryMetadata": {
1842+
"properties": {
1843+
"property_metadata": {
1844+
"type": "object",
1845+
"title": "Property Metadata",
1846+
"description": "An object containing per-entry and per-property metadata. The keys are the names of the fields in attributes for which metadata is available. The values belonging to these keys are dictionaries containing the relevant metadata fields. See also [Metadata properties](https://github.com/Materials-Consortia/OPTIMADE/blob/develop/optimade.rst#metadata-properties)"
1847+
}
1848+
},
1849+
"additionalProperties": true,
1850+
"type": "object",
1851+
"title": "EntryMetadata",
1852+
"description": "Contains the metadata for the attributes of an entry"
1853+
},
18411854
"EntryRelationships": {
18421855
"properties": {
18431856
"references": {
@@ -1897,13 +1910,13 @@
18971910
"meta": {
18981911
"anyOf": [
18991912
{
1900-
"$ref": "#/components/schemas/Meta"
1913+
"$ref": "#/components/schemas/EntryMetadata"
19011914
},
19021915
{
19031916
"type": "null"
19041917
}
19051918
],
1906-
"description": "a meta object containing non-standard meta-information about a resource that can not be represented as an attribute or relationship."
1919+
"description": "A [JSON API meta object](https://jsonapi.org/format/1.1/#document-meta) that is used to communicate metadata."
19071920
},
19081921
"attributes": {
19091922
"$ref": "#/components/schemas/EntryResourceAttributes",
@@ -2481,13 +2494,13 @@
24812494
"meta": {
24822495
"anyOf": [
24832496
{
2484-
"$ref": "#/components/schemas/Meta"
2497+
"$ref": "#/components/schemas/EntryMetadata"
24852498
},
24862499
{
24872500
"type": "null"
24882501
}
24892502
],
2490-
"description": "a meta object containing non-standard meta-information about a resource that can not be represented as an attribute or relationship."
2503+
"description": "A [JSON API meta object](https://jsonapi.org/format/1.1/#document-meta) that is used to communicate metadata."
24912504
},
24922505
"attributes": {
24932506
"$ref": "#/components/schemas/LinksResourceAttributes",
@@ -2978,13 +2991,13 @@
29782991
"meta": {
29792992
"anyOf": [
29802993
{
2981-
"$ref": "#/components/schemas/Meta"
2994+
"$ref": "#/components/schemas/EntryMetadata"
29822995
},
29832996
{
29842997
"type": "null"
29852998
}
29862999
],
2987-
"description": "a meta object containing non-standard meta-information about a resource that can not be represented as an attribute or relationship."
3000+
"description": "A [JSON API meta object](https://jsonapi.org/format/1.1/#document-meta) that is used to communicate metadata."
29883001
},
29893002
"attributes": {
29903003
"$ref": "#/components/schemas/ReferenceResourceAttributes"
@@ -4102,13 +4115,13 @@
41024115
"meta": {
41034116
"anyOf": [
41044117
{
4105-
"$ref": "#/components/schemas/Meta"
4118+
"$ref": "#/components/schemas/EntryMetadata"
41064119
},
41074120
{
41084121
"type": "null"
41094122
}
41104123
],
4111-
"description": "a meta object containing non-standard meta-information about a resource that can not be represented as an attribute or relationship."
4124+
"description": "A [JSON API meta object](https://jsonapi.org/format/1.1/#document-meta) that is used to communicate metadata."
41124125
},
41134126
"attributes": {
41144127
"$ref": "#/components/schemas/StructureResourceAttributes"

‎optimade/models/entries.py‎

Lines changed: 78 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
11
from datetime import datetime
22
from typing import Annotated, Any, ClassVar, Literal
33

4-
from pydantic import BaseModel, field_validator
4+
from pydantic import BaseModel, ValidationInfo, field_validator
55

6-
from optimade.models.jsonapi import Attributes, Relationships, Resource
6+
from optimade.models.jsonapi import Attributes, Meta, Relationships, Resource
77
from optimade.models.optimade_json import (
88
BaseRelationshipResource,
99
DataType,
@@ -117,6 +117,36 @@ def cast_immutable_id_to_str(cls, value: Any) -> str:
117117
return value
118118

119119

120+
class EntryMetadata(Meta):
121+
"""Contains the metadata for the attributes of an entry"""
122+
123+
property_metadata: dict = StrictField(
124+
None,
125+
description="""An object containing per-entry and per-property metadata. The keys are the names of the fields in attributes for which metadata is available. The values belonging to these keys are dictionaries containing the relevant metadata fields. See also [Metadata properties](https://github.com/Materials-Consortia/OPTIMADE/blob/develop/optimade.rst#metadata-properties)""",
126+
)
127+
128+
@field_validator("property_metadata", mode="before")
129+
def check_property_metadata_subfields(cls, value: Any) -> dict:
130+
"""Loop through any per-property metadata field and check that
131+
the subfields are prefixed correctly.
132+
133+
"""
134+
error_fields: list[str] = []
135+
136+
if value is not None and isinstance(value, dict):
137+
for field in value:
138+
if property_metadata := value.get(field):
139+
for subfield in property_metadata:
140+
if not subfield.startswith("_"):
141+
error_fields.append(subfield)
142+
143+
if error_fields:
144+
raise ValueError(
145+
f"The keys under the field `property_metadata` need to be prefixed. The field(s) {error_fields} are not prefixed."
146+
)
147+
return value
148+
149+
120150
class EntryResource(Resource):
121151
"""The base model for an entry resource."""
122152

@@ -171,6 +201,14 @@ class EntryResource(Resource):
171201
),
172202
]
173203

204+
meta: Annotated[
205+
EntryMetadata | None,
206+
StrictField(
207+
None,
208+
description="""A [JSON API meta object](https://jsonapi.org/format/1.1/#document-meta) that is used to communicate metadata.""",
209+
),
210+
] = None
211+
174212
relationships: Annotated[
175213
EntryRelationships | None,
176214
StrictField(
@@ -179,6 +217,44 @@ class EntryResource(Resource):
179217
),
180218
] = None
181219

220+
@field_validator("meta", mode="before")
221+
def check_meta(cls, meta: Any, info: ValidationInfo) -> dict | None:
222+
"""Validator to check whether the per-entry `meta` field is valid,
223+
including stripping out any per-property metadata for properties that
224+
do not otherwise appear in the model.
225+
226+
"""
227+
if not meta:
228+
return meta
229+
230+
if property_metadata := meta.pop("property_metadata", None):
231+
# check that all the fields under property metadata are in attributes
232+
attributes = info.data.get("attributes", {})
233+
property_error_fields: list[str] = []
234+
for subfield in property_metadata:
235+
if subfield not in attributes:
236+
property_error_fields.append(subfield)
237+
238+
if property_error_fields:
239+
raise ValueError(
240+
f"The keys under the field `property_metadata` need to match with the field names in attributes. The field(s) {property_error_fields} are however not present in attributes."
241+
)
242+
243+
meta["property_metadata"] = property_metadata
244+
245+
meta_error_fields: list[str] = []
246+
for field in meta:
247+
if field not in EntryMetadata.model_fields:
248+
if not field.startswith("_"):
249+
meta_error_fields.append(field)
250+
251+
if meta_error_fields:
252+
raise ValueError(
253+
f"The keys under the field `meta` need to be prefixed if not otherwise defined. The field(s) {meta_error_fields} are not defined for per-entry `meta`."
254+
)
255+
256+
return meta
257+
182258

183259
class EntryInfoProperty(BaseModel):
184260
description: Annotated[

‎optimade/server/data/test_structures.json‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,13 @@
33
"_id": {
44
"$oid": "5cfb441f053b174410700d02"
55
},
6+
"meta": {
7+
"property_metadata": {
8+
"elements_ratios": {
9+
"_exmpl_originates_from_project": "Pure Metals"
10+
}
11+
}
12+
},
613
"assemblies": null,
714
"chemsys": "Ac",
815
"cartesian_site_positions": [
@@ -80,6 +87,13 @@
8087
"_id": {
8188
"$oid": "5cfb441f053b174410700d03"
8289
},
90+
"meta": {
91+
"property_metadata": {
92+
"elements_ratios": {
93+
"_exmpl_originates_from_project": "Actinides_Alloys"
94+
}
95+
}
96+
},
8397
"assemblies": null,
8498
"chemsys": "Ac-Ag-Ir",
8599
"cartesian_site_positions": [
@@ -197,6 +211,13 @@
197211
"_id": {
198212
"$oid": "5cfb441f053b174410700d04"
199213
},
214+
"meta": {
215+
"property_metadata": {
216+
"elements_ratios": {
217+
"_exmpl_originates_from_project": "Actinides_Alloys"
218+
}
219+
}
220+
},
200221
"assemblies": null,
201222
"chemsys": "Ac-Ag-Pb",
202223
"cartesian_site_positions": [
@@ -323,6 +344,13 @@
323344
"_id": {
324345
"$oid": "5cfb441f053b174410700d18"
325346
},
347+
"meta": {
348+
"property_metadata": {
349+
"elements_ratios": {
350+
"_exmpl_originates_from_project": "Actinides_Alloys"
351+
}
352+
}
353+
},
326354
"assemblies": null,
327355
"chemsys": "Ac-Mg",
328356
"cartesian_site_positions": [
@@ -413,6 +441,13 @@
413441
"_id": {
414442
"$oid": "5cfb441f053b174410700d1f"
415443
},
444+
"meta": {
445+
"property_metadata": {
446+
"elements_ratios": {
447+
"_exmpl_originates_from_project": null
448+
}
449+
}
450+
},
416451
"assemblies": null,
417452
"chemsys": "Ac-O",
418453
"cartesian_site_positions": [
@@ -515,6 +550,11 @@
515550
"_id": {
516551
"$oid": "5cfb441f053b174410700d6f"
517552
},
553+
"meta": {
554+
"property_metadata": {
555+
"elements_ratios": {}
556+
}
557+
},
518558
"assemblies": null,
519559
"chemsys": "Ac-Cu-F-O",
520560
"cartesian_site_positions": [
@@ -639,6 +679,13 @@
639679
"_id": {
640680
"$oid": "5cfb441f053b174410700dc9"
641681
},
682+
"meta": {
683+
"property_metadata": {
684+
"elements_ratios": {
685+
"_exmpl_originates_from_project": "Pure Metals"
686+
}
687+
}
688+
},
642689
"assemblies": null,
643690
"chemsys": "Ag",
644691
"cartesian_site_positions": [
@@ -706,6 +753,11 @@
706753
"_id": {
707754
"$oid": "5cfb441f053b174410700ddd"
708755
},
756+
"meta": {
757+
"property_metadata": {
758+
"elements_ratios": null
759+
}
760+
},
709761
"assemblies": null,
710762
"chemsys": "Ag-Br-Cl-Te",
711763
"cartesian_site_positions": [
@@ -896,6 +948,9 @@
896948
"_id": {
897949
"$oid": "5cfb441f053b174410700e04"
898950
},
951+
"meta": {
952+
"property_metadata": {}
953+
},
899954
"assemblies": null,
900955
"chemsys": "Ag-C-Cl-N-O-S",
901956
"cartesian_site_positions": [
@@ -1072,6 +1127,9 @@
10721127
"_id": {
10731128
"$oid": "5cfb441f053b174410700e11"
10741129
},
1130+
"meta": {
1131+
"property_metadata": null
1132+
},
10751133
"assemblies": null,
10761134
"chemsys": "Ag-C-Cl-H-N",
10771135
"cartesian_site_positions": [

0 commit comments

Comments
 (0)