Merge pull request #217 from cmu-delphi/rf_geo_refactor

Further refactoring for the geo coding utility

Author: krivard

.gitignore

@@ -116,6 +116,9 @@ venv.bak/
 # mkdocs documentation
 /site
 
+# VSCode settings
+*.vscode
+
 # mypy
 .mypy_cache/
 

_delphi_utils_python/data_proc/geomap/02_20_uszips.csv

@@ -1,115 +1,65 @@
 # Geocoding data processing pipeline
 
-Authors: Jingjing Tang, James Sharpnack
-
-The data_proc/geomap directory contains original source data, processing scripts, and notes for processing from original source to crosswalk tables in the data directory for the delphi_utils package.
+Authors: Jingjing Tang, James Sharpnack, Dmitry Shemetov
 
 ## Usage
 
 Requires the following source files below.
 
-Run the following to write the cross files in the package data dir...
+Run the following to build the crosswalk tables in `covidcast-indicators/_delph_utils_python/delph_utils/data`
 ```
 $ python geo_data_proc.py
 ```
-this will build the following files...
-- fips_msa_cross.csv
-- zip_fips_cross.csv
-- state_codes.csv
 
 You can see consistency checks and diffs with old sources in ./consistency_checks.ipynb
 
-## Source files
-
-- 03_20_MSAs.xls : [US Census Bureau](https://www.census.gov/geographies/reference-files/time-series/demo/metro-micro/delineation-files.html)
-- 02_20_uszips.csv : Hand edited file from Jingjing, we only use the fips,zip encoding and also extract the states from these
-- Crosswalk files from https://www.huduser.gov/portal/datasets/usps_crosswalk.html
-- JHU crosswalk table: https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data#uid-lookup-table-logic
-- ZIP/County population: https://www.census.gov/geographies/reference-files/time-series/geo/relationship-files.html#par_textimage_674173622, https://www2.census.gov/geo/docs/maps-data/data/rel/zcta_county_rel_10.txt?#
-
-## Todo
-
-- make direct cross tables for fips -> hrr and zip -> msa / state
-- use hud for zip -> fips?
-
-## Notes
-
-Some of the source files were constructed by hand, most notably 02_20_uszips.csv.
-
-The 02_20_uszips.csv file is based on the newest consensus data including 5-digit zipcode, fips code, county name, state, population, HRR, HSA (I downloaded the original file from here https://simplemaps.com/data/us-zips. This file matches best to the most recent (2020) situation in terms of the population. But there still exist some matching problems. I manually checked and corrected those lines (~20) with zip-codes.com (https://www.zip-codes.com/zip-code/58439/zip-code-58439.asp). The mapping from 5-digit zipcode to HRR is based on the file in 2017 version downloaded from https://atlasdata.dartmouth.edu/static/supp_research_data
-
-transStateToHRR.csv and transfipsToHRR.csv are used to transform data from state level or county level to HRR respectively. For example, x is the horizontal vector of covid cases for different states in 04/10/20, then we have x @ H = y, where H is the table provided in these two csv files and y is a horizontal vector of covid cases for different HRRs.
+## Geo Codes
 
-HRRs are represented by hrrnum. There are 306 hrrs in total. They are not named as consecutive numbers.
+We support the following geocodes.
 
--Jingjing
+- The ZIP code and the FIPS code are the most granular geocodes we support. 
+  - The [ZIP code](https://en.wikipedia.org/wiki/ZIP_Code) is a US postal code used by the USPS and the [FIPS code](https://en.wikipedia.org/wiki/FIPS_county_code) is an identifier for US counties and other associated territories. The ZIP code is five digit code (with leading zeros). 
+  - The FIPS code is a five digit code (with leading zeros), where the first two digits are a two-digit state code and the last three are a three-digit county code (see this [US Census Bureau page](https://www.census.gov/library/reference/code-lists/ansi.html) for detailed information).
+- The Metropolitan Statistical Area (MSA) code refers to regions around cities (these are sometimes referred to as CBSA codes). More information on these can be found at the [US Census Bureau](https://www.census.gov/programs-surveys/metro-micro/about.html).
+  - We are reserving 10001-10099 for states codes of the form 100XX where XX is the FIPS code for the state (the current smallest CBSA is 10100). In the case that the CBSA codes change then it should be verified that these are not used. 
+- State codes are a series of equivalent identifiers for US state. They include the state name, the state number (state_id), and the state two-letter abbreviation (state_code). The state number is the state FIPS code. See [here](https://en.wikipedia.org/wiki/List_of_U.S._state_and_territory_abbreviations) for more.
+- The Hospital Referral Region (HRR) and the Hospital Service Area (HSA). More information [here](https://www.dartmouthatlas.org/covid-19/hrr-mapping/).
+- The JHU signal contains its own geographic identifier, labeled the UID. Documentation is provided at [their repo](https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data#uid-lookup-table-logic). Its FIPS codes depart in some special cases, so we produce manual changes listed below.
 
+## Source files
 
-04/14/20: 'msa_id' and 'msa_name' are added according to the msa_list.csv that Aaron found from https://apps.bea.gov/regional/docs/msalist.cfm (2019)   
-
-04/15/20:
-The newly updated(added columns) are based on cbsatocountycrosswalk.csv from https://data.nber.org/data/cbsa-fips-county-crosswalk.html
-- 'msa' : MSA ID
-- 'msaname': Name of the MSA
-- 'cbsa': CBSA ID
-- 'cbsaname': Name of the CBSA
-
-
-04/19/20:
-Changed to msa_list.csv again. 
-
-05/20/20: Updated msa_list.csv to include MSAs in Puerto Rico, using the delineations file from March 2020: https://www.census.gov/geographies/reference-files/time-series/demo/metro-micro/delineation-files.html
-
-06/15/20:
-Added file co-est2019-annres.csv, which gives 2019 population estimates for each county by name 
+The source files are requested from a government URL when `geo_data_proc.py` is run (see the top of said script for the URLs). Below we describe the locations to find updated versions of the source files, if they are ever needed.
 
-Source: Annual Estimates of the Resident Population for Counties in the United States: April 1, 2010 to July 1, 2019 (CO-EST2019-ANNRES). U.S. Census Bureau, Population Division. Release Date: March 2020
-Note: The estimates are based on the 2010 Census and reflect changes to the April 1, 2010 population due to the Count Question Resolution program and geographic program revisions. All geographic boundaries for the 2019 population estimates are as of January 1, 2019. For population estimates methodology statements, see http://www.census.gov/programs-surveys/popest/technical-documentation/methodology.html.
+- ZIP -> FIPS (county) population tables available from [US Census](https://www.census.gov/geographies/reference-files/time-series/geo/relationship-files.html#par_textimage_674173622). This file contains the population of the intersections between ZIP and FIPS regions, allowing the creation of a population-weighted transform between the two.
+- ZIP -> HRR -> HSA crosswalk file comes from the 2018 version at the [Dartmouth Atlas Project](https://atlasdata.dartmouth.edu/static/supp_research_data).
+- FIPS -> MSA crosswalk file comes from the September 2018 version of the delineation files at the [US Census Bureau](https://www.census.gov/geographies/reference-files/time-series/demo/metro-micro/delineation-files.html).
+- State Code -> State ID -> State Name comes from the ANSI standard at the [US Census](https://www.census.gov/library/reference/code-lists/ansi.html#par_textimage_3). The first two digits of a FIPS codes should match the state code here.
+- JHU UID -> FIPS comes from [the JHU documentation](https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data#uid-lookup-table-logic). We have to do some hand modifications to the JHU UID because the mapping to FIPS isn't always consistent.
 
-Note: The 6,222 people in Bedford city, Virginia, which was an independent city as of the 2010 Census, are not included in the April 1, 2010 Census enumerated population presented in the county estimates. In July 2013, the legal status of Bedford changed from a city to a town and it became dependent within (or part of) Bedford County, Virginia. This population of Bedford town is now included in the April 1, 2010 estimates base and all July 1 estimates for Bedford County. Because it is no longer an independent city, Bedford town is not listed in this table. As a result, the sum of the April 1, 2010 census values for Virginia counties and independent cities does not equal the 2010 Census count for Virginia, and the sum of April 1, 2010 census values for all counties and independent cities in the United States does not equal the 2010 Census count for the United States. Substantial geographic changes to counties can be found on the Census Bureau website at https://www.census.gov/programs-surveys/geography/technical-documentation/county-changes.html.
+## Derived files
 
+The rest of the crosswalk tables are derived from the mappings above. We provide crosswalk functions from granular to coarser codes, but not the other way around. This is because there is no information gained when crosswalking from coarse to granular.
 
-07/07/2020:
-Introduced the March 2020 MSA file, source is [US Census Bureau](https://www.census.gov/geographies/reference-files/time-series/demo/metro-micro/delineation-files.html).  This file seems to differ in a few fips codes from the source for the 02_20_uszip file which Jingjing constructed.  There are at least 10 additional fips in 03_20_msa that are not in the uszip file, and one of the msa codes seems to be incorrect: 49020 (a google search confirms that it is incorrect in uszip and correct in the census data). 
+## JHU UID mapping changes
 
-07/08/2020:
-We are reserving 00001-00099 for states codes of the form 100XX where XX is the fips code for the state.  In the case that the CBSA codes change then it should be verified that these are not used.  The current smallest CBSA is 10100.
+- Dukes and Nantucket counties in Massachusets are aggregated, so we split them with population-proportional weights (approximately 2/3 Dukes and 1/3 Nantucket).
+- The same procedure is followed by Kansas City and four of its counties.
+- Kusilvak, Alaska is mapped to the FIPS code 02270.
+- Ogalala Lakota, South Dakota is mapped to the FIPS code 46113.
+- Utah reports at a territory level, so we only report it at in a state level megaFIPS 49000.
+- JHU places cases and deaths that cannot be localized to a single county into "Out of State" and "Unassigned" categories. We map these to the "megaFIPS" code XX000, where XX is the state FIPS code. This way, the data is recovered when aggregating up to the state level, but does not interfere with other counties.
 
--James
+## Deprecated source files
 
-07/22/2020:
-- Introducing the COUNTY_ZIP and ZIP_COUNTY crosswalk files from https://www.huduser.gov/portal/datasets/usps_crosswalk.html
-- Also the ZIP to HRR Crosswalk file (from 2018) from https://atlasdata.dartmouth.edu/static/supp_research_data
-- Added the JHU crosswalk table and created a jhu_uid to fips crosswalk table: https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data#uid-lookup-table-logic
+- ZIP to FIPS to HRR to states: `02_20_uszips.csv` comes from a version of the table [here](https://simplemaps.com/data/us-zips) modified by Jingjing to include population weights.
+  - The `02_20_uszips.csv` file is based on the newest consensus data including 5-digit zipcode, fips code, county name, state, population, HRR, HSA (I downloaded the original file from [here](https://simplemaps.com/data/us-zips). This file matches best to the most recent (2020) situation in terms of the population. But there still exist some matching problems. I manually checked and corrected those lines (~20) with [zip-codes](https://www.zip-codes.com/zip-code/58439/zip-code-58439.asp). The mapping from 5-digit zipcode to HRR is based on the file in 2017 version downloaded from [here](https://atlasdata.dartmouth.edu/static/supp_research_data).
+- ZIP -> FIPS is provided by [huduser.gov](https://www.huduser.gov/portal/datasets/usps_crosswalk.html) for zip -> fips?
+- FIPS county population data from [US Census Bureau](http://www.census.gov/programs-surveys/popest/technical-documentation/methodology.html). Details of Bedford, Virginia counting [here](https://www.census.gov/programs-surveys/geography/technical-documentation/county-changes.html).
+- JHU UID crosswalk table [here](https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data#uid-lookup-table-logic)
+- CBSA -> FIPS crosswalk from [here](https://data.nber.org/data/cbsa-fips-county-crosswalk.html) (the file is `cbsatocountycrosswalk.csv`).
+- MSA tables from March 2020 [here](https://www.census.gov/geographies/reference-files/time-series/demo/metro-micro/delineation-files.html). This file seems to differ in a few fips codes from the source for the 02_20_uszip file which Jingjing constructed. There are at least 10 additional fips in 03_20_msa that are not in the uszip file, and one of the msa codes seems to be incorrect: 49020 (a google search confirms that it is incorrect in uszip and correct in the census data).
+- MSA tables from 2019 [here](https://apps.bea.gov/regional/docs/msalist.cfm)
 
-There are NaN fips in the JHU tables, so to resolve this we are moving over to using the JHU unique id.
-We have to deal with the NaN fips by hand, which are 
-```
-748                                                   US
-887                                        Recovered, US
-888               Dukes and Nantucket, Massachusetts, US
-889                            Kansas City, Missouri, US
-890    Michigan Department of Corrections (MDOC), Mic...
-891    Federal Correctional Institution (FCI), Michig...
-892                           Air Force, US Military, US
-893                                Army, US Military, US
-894                        Marine Corps, US Military, US
-895                                Navy, US Military, US
-896                          Unassigned, US Military, US
-897                                      US Military, US
-898               Inmates, Federal Bureau of Prisons, US
-899                 Staff, Federal Bureau of Prisons, US
-900                        Federal Bureau of Prisons, US
-901                                 Bear River, Utah, US
-902                               Central Utah, Utah, US
-903                             Southeast Utah, Utah, US
-904                             Southwest Utah, Utah, US
-905                                  TriCounty, Utah, US
-906                               Weber-Morgan, Utah, US
-907                                Veteran Hospitals, US
-```
-Is you look at geo_data.py::
+## Notes 
 
-08/04/2020:
-Large changes in MSA from 2018 version from bea.gov (msa_list.csv), and the new 2020 version from census bureau (03_20_MSAs.xls).
-Trying to use 2018 version instead from https://www.census.gov/geographies/reference-files/time-series/demo/metro-micro/delineation-files.html
+- The NAs in the coding currently zero-fills.