CHNG historical issues are wrong before 2021-02-21

[rumackaaron]

The CHNG signals are based on queries in a private database. Delphi only began to receive the results of these queries in real time starting 2021-02-21. CHNG provided Delphi with a query to estimate historical issues from the database, but this query was wrong. The query filtered historical issues based on the chc_received_date column but should have filtered by the load_id column. The indicator values based on these erroneous queries are available through the API. Instead, the API should produce the load_id-based indicator values.

To solve this issue, we will need to:

1. Run load_id queries in the CHNG private environment (done).
2. Transfer the resulting aggregated counts to the SFTP server (done).
3. Run the CHNG indicator script using the new aggregated counts.
4. Load the indicator csv files to the server.
5. Upload the indicator values into the database and overwrite the old ones.

[krivard]

Recommend the following modification:
4. Convert the covidcast-format CSVs to database-format TSVs
5. Drop bad data and insert good data
since we'll be able to load the repaired data way faster that way (hours vs weeks)

a database-format TSV file has the following columns:

• source
• signal
• time_type
• geo_type
• time_value
• geo_value
• value
• stderr (\N)
• sample_size (\N)
• issue (compute using eg strptime(sftp_filename[:8], "%Y%m%d") + timedelta(days=1))
• lag (compute using eg (issue - time_value) / timedelta(days=1))
• is_latest_issue (0 or 1. Tricky to compute but crucial to get right)
• is_wip (0)
• missing_value (0)
• missing_stderr (3 for "censored")
• missing_sample_size (3 for "censored")