update to pgstac 0.9.0

[vincentsarago]

PgSTAC 0.9.0 has some breaking change around the context extension and next/prev links

This is not urging, so I think it can after after we make a 3.0 release (once stac-fastapi 3.0 gets released)

[zstatmanweil]

hey @vincentsarago! I am working with pgstac 0.9.X ands stac-fastapi 2.4.5. I missed this issue and now have a lot of data in my database so am trying to proceed as is. Just to clarify, pagination will not work with my current set up? My workaround is setting the limit to 10000 and understanding I can't ever get more than 10,000 results back in a given search. Does that make sense?

[vincentsarago]

Hi @zstatmanweil

Yes pagination works differently with pgstac 0.9.x (links are returned directly by the pgstac response).

you can't get more than 10000 result because there is a hard limit to 10,000 https://github.com/stac-utils/stac-fastapi/blob/main/stac_fastapi/types/stac_fastapi/types/search.py#L18-L23

IMO we should make this configurable or at least raise an error when user set limit>10000

edit:

limit The limit parameter follows the same semantics of the OAFeat Item resource limit parameter. The value is a suggestion to the server as to the maximum number of Item objects the client would prefer in the response. The server may return fewer Item objects, but must not return more. The OpenAPI specification defines the default and maximum values for this parameter. The base specifications define these with a default of 10 and a maximum of 10000, but implementers may choose other values to advertise through their service-desc endpoint. If the limit parameter value is greater than the advertised maximum limit, the server must act as if the request were for the maximum and not return an error

the fact we don't raise an issue is defined by the specification.

You could easily overcome this by providing your own custom Search model

https://github.com/stac-utils/stac-fastapi/blob/main/stac_fastapi/types/stac_fastapi/types/search.py#L55-L70

@attr.s
class BaseSearchGetRequest(APIRequest):
    """Base arguments for GET Request."""

    collections: Optional[str] = attr.ib(default=None, converter=str2list)
    ids: Optional[str] = attr.ib(default=None, converter=str2list)
    bbox: Optional[BBox] = attr.ib(default=None, converter=str2bbox)
    intersects: Optional[str] = attr.ib(default=None)
    datetime: Optional[DateTimeType] = attr.ib(default=None, converter=str_to_interval)
    limit: Optional[int] = attr.ib(default=10)


class BaseSearchPostRequest(Search):
    """Base arguments for POST Request."""

    limit: Optional[int] = 10

🤦 just realizing that the GET request won't apply the 10,000 limit

[zstatmanweil]

Thanks for the response. I am using pystac_client to interact with my API so it is actually throwing an exception: https://github.com/stac-utils/pystac-client/blob/main/pystac_client/item_search.py#L289-L290. So alas, I would also have to work around that.

I think I will wait patiently for pagination, and hey, maybe I will give it a shot! I would enjoy that. I think I understand what is happening and see where in the stac-fastapi code there is a mismatch with what pgstac used to return and 0.9.0 is returning.