-
Notifications
You must be signed in to change notification settings - Fork 41
Age Extension Query with Formatted Age Range #6206
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Have some code cleanup to do before merging. Here is another example of a compiled SQLAlchemy query that gets generated: SELECT
collectionobject.`CollectionObjectID`,
concat_ws(' - ', ifnull(regexp_replace(CAST(agg_subq.max_start_period AS CHAR), '\\.(0+)$', ''), ''), ifnull(regexp_replace(CAST(agg_subq.min_end_period AS CHAR), '\\.(0+)$', ''), '')) AS age,
absoluteage_1.`AbsoluteAge`,
geologictimeperiod_1.`Name`,
geologictimeperiod_2.`Name`,
geologictimeperiod_3.`Name`,
geologictimeperiod_4.`Name`,
geologictimeperiod_5.`Name`,
geologictimeperiod_6.`Name`
FROM
collectionobject
INNER JOIN (
SELECT
unioned.coid AS coid,
min(unioned.endperiod) AS min_end_period,
max(unioned.startperiod) AS max_start_period
FROM
(
SELECT
absoluteage.`CollectionObjectID` AS coid,
CAST(absoluteage.`AbsoluteAge` AS DECIMAL(10, 6)) - coalesce(absoluteage.`AgeUncertainty`, 0) AS startperiod,
CAST(absoluteage.`AbsoluteAge` AS DECIMAL(10, 6)) + coalesce(absoluteage.`AgeUncertainty`, 0) AS endperiod
FROM
absoluteage
WHERE
CAST(absoluteage.`AbsoluteAge` AS DECIMAL(10, 6)) - coalesce(absoluteage.`AgeUncertainty`, 0) <= 2000.0
AND CAST(absoluteage.`AbsoluteAge` AS DECIMAL(10, 6)) + coalesce(absoluteage.`AgeUncertainty`, 0) >= 2.0
UNION ALL
SELECT
r.`CollectionObjectID` AS coid,
CASE
WHEN (r.`AgeNameEndID` IS NOT NULL) THEN greatest(CAST(a.`StartPeriod` AS DECIMAL(10, 6)) + coalesce(a.`StartUncertainty`, 0) + coalesce(r.`AgeUncertainty`, 0), CAST(aend.`StartPeriod` AS DECIMAL(10, 6)) + coalesce(aend.`StartUncertainty`, 0) + coalesce(r.`AgeUncertainty`, 0))
ELSE CAST(a.`StartPeriod` AS DECIMAL(10, 6)) + coalesce(a.`StartUncertainty`, 0) + coalesce(r.`AgeUncertainty`, 0)
END AS startperiod,
CASE
WHEN (r.`AgeNameEndID` IS NOT NULL) THEN least((CAST(a.`EndPeriod` AS DECIMAL(10, 6)) - coalesce(a.`EndUncertainty`, 0)) - coalesce(r.`AgeUncertainty`, 0), (CAST(aend.`EndPeriod` AS DECIMAL(10, 6)) - coalesce(aend.`EndUncertainty`, 0)) - coalesce(r.`AgeUncertainty`, 0))
ELSE (CAST(a.`EndPeriod` AS DECIMAL(10, 6)) - coalesce(a.`EndUncertainty`, 0)) - coalesce(r.`AgeUncertainty`, 0)
END AS endperiod
FROM
relativeage AS r
INNER JOIN geologictimeperiod AS a ON
r.`AgeNameID` = a.`GeologicTimePeriodID`
LEFT OUTER JOIN geologictimeperiod AS aend ON
r.`AgeNameEndID` = aend.`GeologicTimePeriodID`
WHERE
a.`StartPeriod` IS NOT NULL
AND a.`EndPeriod` IS NOT NULL
AND a.`StartPeriod` >= a.`EndPeriod`
AND (r.`AgeNameEndID` IS NULL
OR aend.`StartPeriod` IS NOT NULL
AND aend.`EndPeriod` IS NOT NULL
AND aend.`StartPeriod` >= aend.`EndPeriod`)
AND CASE
WHEN (r.`AgeNameEndID` IS NOT NULL) THEN greatest(CAST(a.`StartPeriod` AS DECIMAL(10, 6)) + coalesce(a.`StartUncertainty`, 0) + coalesce(r.`AgeUncertainty`, 0), CAST(aend.`StartPeriod` AS DECIMAL(10, 6)) + coalesce(aend.`StartUncertainty`, 0) + coalesce(r.`AgeUncertainty`, 0))
ELSE CAST(a.`StartPeriod` AS DECIMAL(10, 6)) + coalesce(a.`StartUncertainty`, 0) + coalesce(r.`AgeUncertainty`, 0)
END <= 2000.0
AND CASE
WHEN (r.`AgeNameEndID` IS NOT NULL) THEN least((CAST(a.`EndPeriod` AS DECIMAL(10, 6)) - coalesce(a.`EndUncertainty`, 0)) - coalesce(r.`AgeUncertainty`, 0), (CAST(aend.`EndPeriod` AS DECIMAL(10, 6)) - coalesce(aend.`EndUncertainty`, 0)) - coalesce(r.`AgeUncertainty`, 0))
ELSE (CAST(a.`EndPeriod` AS DECIMAL(10, 6)) - coalesce(a.`EndUncertainty`, 0)) - coalesce(r.`AgeUncertainty`, 0)
END >= 2.0
UNION ALL
SELECT
DISTINCT c.`CollectionObjectID` AS coid,
CASE
WHEN (p.`ChronosStratEndID` IS NOT NULL) THEN least(CAST(cs.`StartPeriod` AS DECIMAL(10, 6)) + coalesce(cs.`StartUncertainty`, 0), CAST(csend.`StartPeriod` AS DECIMAL(10, 6)) + coalesce(csend.`StartUncertainty`, 0))
ELSE CAST(cs.`StartPeriod` AS DECIMAL(10, 6)) + coalesce(cs.`StartUncertainty`, 0)
END AS startperiod,
CASE
WHEN (p.`ChronosStratEndID` IS NOT NULL) THEN greatest(CAST(cs.`EndPeriod` AS DECIMAL(10, 6)) - coalesce(cs.`EndUncertainty`, 0), CAST(csend.`EndPeriod` AS DECIMAL(10, 6)) - coalesce(csend.`EndUncertainty`, 0))
ELSE CAST(cs.`EndPeriod` AS DECIMAL(10, 6)) - coalesce(cs.`EndUncertainty`, 0)
END AS endperiod
FROM
collectionobject AS c
LEFT OUTER JOIN collectingevent AS ce ON
c.`CollectingEventID` = ce.`CollectingEventID`
LEFT OUTER JOIN locality AS l ON
ce.`LocalityID` = l.`LocalityID`
LEFT OUTER JOIN paleocontext AS p ON
c.`PaleoContextID` = p.`PaleoContextID`
OR ce.`PaleoContextID` = p.`PaleoContextID`
OR l.`PaleoContextID` = p.`PaleoContextID`
LEFT OUTER JOIN geologictimeperiod AS cs ON
p.`ChronosStratID` = cs.`GeologicTimePeriodID`
LEFT OUTER JOIN geologictimeperiod AS csend ON
p.`ChronosStratEndID` = csend.`GeologicTimePeriodID`
WHERE
p.`PaleoContextID` IS NOT NULL
AND cs.`StartPeriod` IS NOT NULL
AND cs.`EndPeriod` IS NOT NULL
AND cs.`StartPeriod` >= cs.`EndPeriod`
AND (p.`ChronosStratEndID` IS NULL
OR csend.`StartPeriod` IS NOT NULL
AND csend.`EndPeriod` IS NOT NULL
AND csend.`StartPeriod` >= csend.`EndPeriod`)
AND CASE
WHEN (p.`ChronosStratEndID` IS NOT NULL) THEN least(CAST(cs.`StartPeriod` AS DECIMAL(10, 6)) + coalesce(cs.`StartUncertainty`, 0), CAST(csend.`StartPeriod` AS DECIMAL(10, 6)) + coalesce(csend.`StartUncertainty`, 0))
ELSE CAST(cs.`StartPeriod` AS DECIMAL(10, 6)) + coalesce(cs.`StartUncertainty`, 0)
END <= 2000.0
AND CASE
WHEN (p.`ChronosStratEndID` IS NOT NULL) THEN greatest(CAST(cs.`EndPeriod` AS DECIMAL(10, 6)) - coalesce(cs.`EndUncertainty`, 0), CAST(csend.`EndPeriod` AS DECIMAL(10, 6)) - coalesce(csend.`EndUncertainty`, 0))
ELSE CAST(cs.`EndPeriod` AS DECIMAL(10, 6)) - coalesce(cs.`EndUncertainty`, 0)
END >= 2.0) AS unioned
GROUP BY
unioned.coid) AS agg_subq ON
collectionobject.`CollectionObjectID` = agg_subq.coid
LEFT OUTER JOIN absoluteage AS absoluteage_1 ON
collectionobject.`CollectionObjectID` = absoluteage_1.`CollectionObjectID`
LEFT OUTER JOIN relativeage AS relativeage_1 ON
collectionobject.`CollectionObjectID` = relativeage_1.`CollectionObjectID`
LEFT OUTER JOIN geologictimeperiod AS geologictimeperiod_1 ON
geologictimeperiod_1.`GeologicTimePeriodID` = relativeage_1.`AgeNameID`
LEFT OUTER JOIN geologictimeperiod AS geologictimeperiod_2 ON
geologictimeperiod_2.`GeologicTimePeriodID` = relativeage_1.`AgeNameEndID`
LEFT OUTER JOIN paleocontext AS paleocontext_1 ON
paleocontext_1.`PaleoContextID` = collectionobject.`PaleoContextID`
LEFT OUTER JOIN geologictimeperiod AS geologictimeperiod_3 ON
geologictimeperiod_3.`GeologicTimePeriodID` = paleocontext_1.`ChronosStratID`
LEFT OUTER JOIN geologictimeperiod AS geologictimeperiod_4 ON
geologictimeperiod_4.`GeologicTimePeriodID` = paleocontext_1.`ChronosStratEndID`
LEFT OUTER JOIN collectingevent AS collectingevent_1 ON
paleocontext_1.`PaleoContextID` = collectingevent_1.`PaleoContextID`
LEFT OUTER JOIN paleocontext AS paleocontext_2 ON
paleocontext_2.`PaleoContextID` = collectingevent_1.`PaleoContextID`
LEFT OUTER JOIN geologictimeperiod AS geologictimeperiod_5 ON
geologictimeperiod_5.`GeologicTimePeriodID` = paleocontext_2.`ChronosStratID`
LEFT OUTER JOIN geologictimeperiod AS geologictimeperiod_6 ON
geologictimeperiod_6.`GeologicTimePeriodID` = paleocontext_2.`ChronosStratEndID`
WHERE
collectionobject.`CollectionID` = 98304; |
|
It seems that the behavior for querying by the time period name isn't working as expected now. Investigating... |
Triggered by 08809c6 on branch refs/heads/issue-6089
|
@specify/ux-testing note: @acwhite211 is going to make a new DB that we can test on for this PR. There are DB issues with the current ones on the test panel. |
|
Hold off on testing temporarily. This PR will be effected by the solution to #6271. I'll also create a new database to make testing this PR better. |
|
Though this PR isn't ready for testing review yet, ran into this issue and was asked to note it in Github. If you have Age in the query set to Any, the results will still display Collection Object ID. Query: AgeTest.json Query results (Same query except changing Age to Range -> 2000 - 2): |
|
Hey @specify/ux-testing, we can go ahead and start retesting this PR for 7.10. I put a database up on the test panel named We might need to push the issue of the age range not showing up when using an "Any" age query. I'm looking into a fix, but it might not be done in time for 7.10, so ignore issue that for now. @grantfitzsimmons checkout if the strict age filter logic is working how you envision, let me know of any CO record examples that are wrong in the query results and I should be able to do a quick fix. Thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tested the functionality on a geo DB, the results returned are not correct.
|
Just to clarify the new intended behavior for the age query. Given:
Strict Filter: (full overlap, a-b is within x-y) Non-Strict Filter: (partial age range overlap, any overlap so x-y can be within a-b or a-b can be within x-y) This is omitting the complication of uncertainty values. The meta age range for each collection object should appear in the age column of the query results. |
|
@acwhite211 , seems correct to me |
|
@specify/ux-testing try retesting this PR. Start of by using the age_geo_test database, and use the guidance of this comment for determining which results should appear. #6206 (comment) The "Any" age query bug is being push to a later issue. For now, do a Age Range query from "13800" to "0" to get all COs with age data (query all of time, the age of the universe :).
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixes #6016
One note here is that you still see the CO ID whenever there is no computed range, which would ideally not show anything. Can this be adjusted?
- See that the "Collection Object - Age" column has a date range.
Fixes #6271
Wonderful.
Tested with this query, and all searching worked as I would expect!
Extended Age Query.json
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- See that the "Collection Object - Age" column has a date range.
Looks good, previous issues seem to be fixed!




Fixes #6089
Fixes #6016
Fixes #6271
Creates a new functions that builds a SQLAlchemy subquery that, in addition to the previous implementation's use case of filter CollectionObjects by all it's paths to chronostrat age, returns a formatted age range for a CollectionObject's maximum start age to its minimum end age. This allows the query QB to display this formatted range in the age column in the query results. This PR also implements the strict query behavior described in #6271.
Here is a compiled SQLAlchemy query that get generated for a simple CollectionObject Age query:
Checklist
self-explanatory (or properly documented)
Testing instructions