Merge pull request #758 from cmu-delphi/release/delphi-epidata-0.3.1

Release Delphi Epidata 0.3.1

Author: krivard

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.3.0
+current_version = 0.3.1
 commit = False
 tag = False
 

docs/api/covidcast-signals/hhs.md

@@ -24,10 +24,12 @@ datasets is mirrored in Epidata at the following endpoint:
 That dataset contains dozens of columns that break down hospital
 resource usage in different ways.
 
-This indicator makes available several commonly-used combinations of
-those columns, aggregated geographically. In particular, we include
-the sum of all adult and pediatric COVID-19 hospital admissions. This
-sum is used as the "ground truth" for hospitalizations by the [COVID-19 Forecast Hub](https://github.com/reichlab/covid19-forecast-hub/blob/master/data-processed/README.md#hospitalizations).
+This indicator makes available several commonly-used columns and combinations of
+columns, aggregated geographically. In particular, we include the sum of all
+adult and pediatric COVID-19 hospital admissions. This sum is used as the
+"ground truth" for hospitalizations by the 
+[COVID-19 Forecast Hub](https://github.com/reichlab/covid19-forecast-hub/blob/master/data-processed/README.md#hospitalizations).
+We also include influenza hospital admissions.
 
 
 | Signal | 7-day average signal | Geography | Resolution | Description |
@@ -36,6 +38,9 @@ sum is used as the "ground truth" for hospitalizations by the [COVID-19 Forecast
 | `confirmed_admissions_covid_1d_prop` | `confirmed_admissions_covid_1d_prop_7dav`| state | 1 day | Sum of adult and pediatric confirmed COVID-19 hospital admissions occurring each day, per 100,000 population. <br/> **Earliest date available:** 2019-12-31 |
 | `sum_confirmed_suspected_admissions_covid_1d` | `sum_confirmed_suspected_admissions_covid_1d_7dav` | state | 1 day | Sum of adult and pediatric confirmed and suspected COVID-19 hospital admissions occurring each day. <br/> **Earliest date available:** 2019-12-31 |
 | `sum_confirmed_suspected_admissions_covid_1d_prop` | `sum_confirmed_suspected_admissions_covid_1d_prop_7dav` | state | 1 day | Sum of adult and pediatric confirmed and suspected COVID-19 hospital admissions occurring each day, per 100,000 population. <br/> **Earliest date available:** 2019-12-31 |
+| `confirmed_admissions_influenza_1d` | `confirmed_admissions_influenza_1d_7dav` | state | 1 day | All confirmed influenza hospital admissions occurring each day. We made this signal available November 1, 2021. <br/> **Earliest issue available:** 2021-09-20 <br/> **Earliest date available:** 2020-01-02 |
+| `confirmed_admissions_influenza_1d_prop` | `confirmed_admissions_influenza_1d_prop_7dav` | state | 1 day | All confirmed influenza hospital admissions occurring each day, per 100,000 population. We made this signal available November 1, 2021. <br/> **Earliest issue available:** 2021-09-20 <br/> **Earliest date available:** 2020-01-02 |
+
 
 The 7-day average signals are computed by Delphi by calculating
 moving averages of the preceding 7 days, so e.g. the signal for June 7 is the
@@ -49,9 +54,9 @@ average of the underlying data for June 1 through 7, inclusive.
 
 ## Estimation
 
-### Statewise, daily resolution
+### Statewise, daily resolution, COVID-19
 
-Statewise daily resolution signals use the following four columns from
+Statewise daily resolution signals for COVID-19 use the following four columns from
 the HHS state timeseries dataset:
 
 * `previous_day_admission_[adult|pediatric]_covid_[confirmed|suspected]`
@@ -72,6 +77,17 @@ The source data specifies that admissions occurred on the previous
 day. We automatically adjust the date of each result so that
 admissions are incident on that date.
 
+### Statewise, daily resolution, influenza
+
+Statewise daily resolution signals for influenza use the following column from
+the HHS state timeseries dataset:
+
+* `previous_day_admission_influenza_confirmed`
+
+The source data specifies that admissions occurred on the previous day. We
+automatically adjust the date of each result so that admissions are incident on
+that date.
+
 ## Limitations
 
 HHS collects data from state and territorial health departments about many, but

docs/symptom-survey/coding.md

@@ -301,8 +301,8 @@ your protocol.
 Wave 5 was deployed on November 24, 2020. Deployment was phased: a fraction of
 users were invited to take Wave 4, while the majority were invited to Wave 5, so
 data users can determine if changes in responses are due to survey revisions or
-to population changes at the same time. Wave 5 is available in English and all
-the languages introduced in Wave 3.
+to population changes at the same time. This overlap lasted roughly seven days.
+Wave 5 is available in English and all the languages introduced in Wave 3.
 
 Files:
 
@@ -370,8 +370,9 @@ released:
 
 ## Wave 6
 
-Wave 6 was deployed on December 19, 2020. It is available in English and the
-languages introduced in Wave 3.
+Wave 6 was deployed on December 19, 2020. Deployment was phased: for seven days,
+a randomly selected portion of respondents continued to receive Wave 5. Wave 6
+is available in English and the languages introduced in Wave 3.
 
 Files:
 
@@ -405,8 +406,9 @@ when you use responses from multiple waves of this survey.
 
 ## Wave 7
 
-Wave 7 was deployed on January 12, 2021. It is available in English and the
-languages introduced in Wave 3.
+Wave 7 was deployed on January 12, 2021. Because only minor changes were made in
+this wave, there was no overlap between waves 6 and 7. It is available in
+English and the languages introduced in Wave 3.
 
 Files:
 
@@ -486,7 +488,7 @@ administered by the University of Maryland](https://covidmap.umd.edu/).
 
 ## Wave 10
 
-Wave 10 was deployed on March 2, 2021. For the following 7 days, 15% of
+Wave 10 was deployed on March 2, 2021. For the following 7 days, roughly 10% of
 respondents (selected at random) continued to receive Wave 8, allowing for
 comparisons of responses between the two waves. Wave 10 is available in English
 and the languages introduced in Wave 3.
@@ -559,7 +561,7 @@ survey.
 
 ## Wave 11
 
-Wave 11 was deployed on May 20, 2021. For the following 14 days, 15% of
+Wave 11 was deployed on May 20, 2021. For the following 30 days, about 10% of
 respondents (selected at random) continued to receive Wave 10, allowing for
 comparisons of responses between the two waves. It is available in English and
 the languages introduced in Wave 3.
@@ -790,11 +792,11 @@ are not anticipated to change the respondent’s answer.
 Wave 12 was deployed on October 7, 2021. Deployment is in two phases:
 
 * Initially the survey was given only to 15% of survey respondents, the
-  remainder receiving Wave 11. This phase was used to conduct three experiments
-  with (1) demographic module placement, (2) vaccination uptake question, and
-  (3) survey invitation text. Details of these experiments can be found
-  [below](#experiments). In data files, this is marked as wave "12.5" to
-  distinguish from the final version.
+  remainder receiving Wave 11. This phase lasted until October 22. This phase
+  was used to conduct three experiments with (1) demographic module placement,
+  (2) vaccination uptake question, and (3) survey invitation text. Details of
+  these experiments can be found [below](#experiments). In data files, this is
+  marked as wave "12.5" to distinguish from the final version.
 * After results of the experiments are analyzed, a final version will be
   deployed using only one demographic module placement and one vaccine uptake
   question. This will be marked as wave 12 in the data files.

src/acquisition/covid_hosp/common/utils.py

@@ -148,25 +148,34 @@ def update_dataset(database, network, newer_than=None, older_than=None):
       Whether a new dataset was acquired.
     """
     metadata = network.fetch_metadata()
+    datasets = []
     with database.connect() as db:
       max_issue = db.get_max_issue()
-      older_than = datetime.datetime.today().date() if newer_than is None else older_than
-      newer_than = max_issue if newer_than is None else newer_than
-      daily_issues = Utils.issues_to_fetch(metadata, newer_than, older_than)
-      if not daily_issues:
-        print("no new issues, nothing to do")
-        return False
-      for issue, revisions in daily_issues.items():
-        issue_int = int(issue.strftime("%Y%m%d"))
-        # download the dataset and add it to the database
-        dataset = Utils.merge_by_key_cols([network.fetch_dataset(url) for url, _ in revisions],
-                                          db.KEY_COLS)
+
+    older_than = datetime.datetime.today().date() if newer_than is None else older_than
+    newer_than = max_issue if newer_than is None else newer_than
+    daily_issues = Utils.issues_to_fetch(metadata, newer_than, older_than)
+    if not daily_issues:
+      print("no new issues, nothing to do")
+      return False
+    for issue, revisions in daily_issues.items():
+      issue_int = int(issue.strftime("%Y%m%d"))
+      # download the dataset and add it to the database
+      dataset = Utils.merge_by_key_cols([network.fetch_dataset(url) for url, _ in revisions],
+                                        db.KEY_COLS)
+      # add metadata to the database using the last revision seen.
+      last_url, last_index = revisions[-1]
+      metadata_json = metadata.loc[last_index].reset_index().to_json()
+      datasets.append((
+        issue_int,
+        dataset,
+        last_url,
+        metadata_json
+      ))
+    with database.connect() as db:
+      for issue_int, dataset, last_url, metadata_json in datasets:
         db.insert_dataset(issue_int, dataset)
-        # add metadata to the database using the last revision seen.
-        last_url, last_index = revisions[-1]
-        metadata_json = metadata.loc[last_index].reset_index().to_json()
         db.insert_metadata(issue_int, last_url, metadata_json)
-
         print(f'successfully acquired {len(dataset)} rows')
 
       # note that the transaction is committed by exiting the `with` block

src/client/delphi_epidata.R

@@ -15,7 +15,7 @@ Epidata <- (function() {
   # API base url
   BASE_URL <- 'https://delphi.cmu.edu/epidata/api.php'
 
-  client_version <- '0.3.0'
+  client_version <- '0.3.1'
 
   # Helper function to cast values and/or ranges to strings
   .listitem <- function(value) {

src/client/delphi_epidata.js

@@ -22,7 +22,7 @@
   }
 })(this, function (exports, fetchImpl, jQuery) {
   const BASE_URL = "https://delphi.cmu.edu/epidata/";
-  const client_version = "0.3.0";
+  const client_version = "0.3.1";
 
   // Helper function to cast values and/or ranges to strings
   function _listitem(value) {