search response does not include datetime when null

[duckontheweb]

Description

The STAC Spec requires that Items have a datetime property even when the value of that property is null. However, in the return value of the search function removes the datetime property when it is null, resulting in an invalid STAC Item.

Steps to reproduce

1. Start up the development server
2. In a psql console, add an item with start_datetime and end_datetime values and a null datetime value:

   SELECT *
   FROM create_item('
   {
     "type": "Feature",
     "stac_version": "1.0.0",
     "id": "datetime_range_test",
     "properties": {
        "start_datetime": "2009-10-15T00:00:00Z",
        "end_datetime": "2010-03-01T23:59:59.999999Z",
        "datetime": null
      },
      "geometry": {
        "type": "Polygon",
        "coordinates": [
          [
            [-101, 36],
            [-101, 37],
            [-102, 37],
            [-102, 36],
            [-101, 36]
          ]
        ]
      },
      "links": [],
      "assets": {},
      "collection": "pgstac-test-collection"
   }
   '::jsonb);

3. Use the search function to fetch the Item and examine the properties:

   SELECT "search" -> 'features' -> 0 -> 'properties'
   FROM search('{"ids": ["datetime_range_test"]}');

The resulting properties JSON is missing the datetime property:

{
  "end_datetime": "2010-03-01T23:59:59.999999Z",
  "start_datetime": "2009-10-15T00:00:00Z"
}

[bitner]

Per the spec in the section you linked to "null is allowed, but requires start_datetime and end_datetime from common metadata to be set."

My reading of that and the implementation in pgstac is that an item must EITHER have datetime OR BOTH start_datetime and end_datetime. If there has been a change in the spec or I have been reading that wrong, please reopen this issue, but I am going to close it now.

[duckontheweb]

Per the spec in the section you linked to "null is allowed, but requires start_datetime and end_datetime from common metadata to be set."

My reading of that and the implementation in pgstac is that an item must EITHER have datetime OR BOTH start_datetime and end_datetime.

Maybe @m-mohr can clarify this, but my understanding of the spec is that datetime must be present even if it has a null value. The JSON schema here seems to back this as well: datetime is required in all cases, but may be null if both start_datetime and end_datetime are present.

So, the expected properties for the example above would be:

{
  "end_datetime": "2010-03-01T23:59:59.999999Z",
  "start_datetime": "2009-10-15T00:00:00Z",
  "datetime": null
}

[m-mohr]

Yes, datetime must always be present, either as null (then provide start and end) or as datetime.
null != not present

[bitner]

OK, that is different than how I had read that. Reopening this ticket.

[drnextgis]

In this case, the item is being ingested with the datetime property stripped. Below is the function call stack:

       create_item
            |
 items_staging_triggerfunc
            |
   content_dehydrate
            |
      content_slim
            |
      strip_jsonb
            |
   jsonb_strip_nulls

@bitner what do you believe would be the preferable approach: refraining from stripping the datetime field if it's null during the ingestion stage, or dynamically adding it to the search results when an item lacks datetime? The former might impact the size of the database, while the latter could potentially affect search performance. Are there any alternative approaches that I might have overlooked?

[clausmichele]

This issue is still relevant to us, what's the plan to move forward? datetime is a required field and the ingested items aren't valid if it is being stripped.

[krisaoe]

Is anyone progressing this? In my field there are many valid cases where a single datetime is not suitable for a STAC item.

[krisaoe]

After looking into pgstac a bit more, I think it's acceptable to add the field at content_hydrate time, keeping the fix in-line with pgstac's extract-normalize-hydrate pattern and avoiding data duplication (columns and jsonb).