pipeline for Quidel covidtest #154
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
jingjtang
commented
Jul 16, 2020
jingjtang
commented
- Pull data from the email attachment
- The pulling step is very time consuming especially when we have much more dates to consider, so now:
- store the intermediate file (aggregated by zip with number of total tests, number of positive tests) as cache
- only update new data when run next time
- ~1.97% invalid tests, Now treat them as negative tests
- Geographical poolling is added since the data volume is now enough. The minimum sample size is 50. If there is no data report for a state/hrr/msa/county, it is filtered out because of this.
- The data start from 5/9 but has only extremely low volume. So, temporarily set the export_start_date to be 5/26.
- As for the time stamp:
- Use test date as time stamp for normal cases
- If storage date is earlier than test date which is abnormal, remove those cases. (0.10%)
- If storage date - test date > 90 days. (0.01%)
- Report data through today (GMT) minus 5 days due to backfill
- Do we want to report county level data? (did not report county level for flu tests due to the data volume problem, there is still the same problem for COVID Test)
krivard
reviewed
Jul 16, 2020
Contributor
krivard
left a comment
krivard
left a comment
There was a problem hiding this comment.
Looks good! This will update from 5/26 to (today minus 5 days) each day it is run, right? I believe that's what we want.
Other todos:
- API documentation
- Missingness documentation
- Some kind of pipeline summary/developer documentation in the README, a la this outline in the facebook package
eujing
reviewed
Jul 16, 2020
eujing
suggested changes
Jul 16, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.