From 8f61d35feffc388772066ed89901f80b9aaf1e10 Mon Sep 17 00:00:00 2001 From: Alex Reinhart Date: Sat, 23 Mar 2024 13:04:09 -0400 Subject: [PATCH 01/17] Add the 49th and 50th CTIS publications (not counting our PNAS paper) --- docs/symptom-survey/publications.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/symptom-survey/publications.md b/docs/symptom-survey/publications.md index 8b0eb768e..dbe553b10 100644 --- a/docs/symptom-survey/publications.md +++ b/docs/symptom-survey/publications.md @@ -26,6 +26,15 @@ Pandemic"](https://www.pnas.org/topic/548) in *PNAS*: Research publications using the survey data include: +- P. Porebski, S. Venkatramanan, A. Adiga, B. Klahn, B. Hurt, M. L. Wilson, + J. Chen, A. Vullikanti, M. Marathe & B. Lewis (2024). [Data-driven + mechanistic framework with stratified immunity and effective transmissibility + for COVID-19 scenario projections](https://doi.org/10.1016/j.epidem.2024.100761). + *Epidemics* 100761. +- V. Nelson, B. Bashyal, P.-N. Tan & Y. A. Argyris (2024). [Vaccine rhetoric + on social media and COVID-19 vaccine uptake rates: A triangulation using + self-reported vaccine acceptance](https://doi.org/10.1016/j.socscimed.2024.116775). + *Social Science & Medicine* 116775. - R.R. Andridge (2024). [Using proxy pattern-mixture models to explain bias in estimates of COVID-19 vaccine uptake from two large surveys](https://doi.org/10.1093/jrsssa/qnae005). *Journal of the Royal Statistical Society Series A: Statistics in Society*. @@ -34,7 +43,7 @@ Research publications using the survey data include: *IISE Transactions*. - de Vries, M., Kim, J.Y. & Han, H. (2023). [The unequal landscape of civic opportunity in America](https://doi.org/10.1038/s41562-023-01743-1). *Nature - Human Behavior*. + Human Behavior* 8, 256-263. - E. Tuzhilina, T. J. Hastie, D. J. McDonald, J. K. Tay & R. Tibshirani (2023). [Smooth multi-period forecasting with application to prediction of COVID-19 cases](https://doi.org/10.1080/10618600.2023.2285337). *Journal of Computational From c5b7b7bd54e962ecd921544d8187dadae787437b Mon Sep 17 00:00:00 2001 From: Alex Reinhart Date: Thu, 28 Mar 2024 09:38:23 -0400 Subject: [PATCH 02/17] Add one more paper --- docs/symptom-survey/publications.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/symptom-survey/publications.md b/docs/symptom-survey/publications.md index dbe553b10..7f01a8b76 100644 --- a/docs/symptom-survey/publications.md +++ b/docs/symptom-survey/publications.md @@ -26,6 +26,10 @@ Pandemic"](https://www.pnas.org/topic/548) in *PNAS*: Research publications using the survey data include: +- A. Srivastava, J. M. Ramirez, S. Díaz-Aranda, J. Aguilar, A. F. Anta, A. Ortega, + and R. E. Lillo (2024). [Nowcasting temporal trends using indirect surveys](https://doi.org/10.1609/aaai.v38i20.30242). + In *Proceedings of the 38th AAAI Conference on Artificial Intelligence* 38, + 22359–22367. - P. Porebski, S. Venkatramanan, A. Adiga, B. Klahn, B. Hurt, M. L. Wilson, J. Chen, A. Vullikanti, M. Marathe & B. Lewis (2024). [Data-driven mechanistic framework with stratified immunity and effective transmissibility From 4c699707b55fc84cb8fb2d3fd53010ed5d8bcdc4 Mon Sep 17 00:00:00 2001 From: Tina Townes Date: Mon, 1 Apr 2024 22:09:01 -0400 Subject: [PATCH 03/17] Added four signals to jhu-csse documentation Added below, which were not listed in documentation: confirmed_7dav_cumulative_num confirmed_7dav_cumulative_prop deaths_7dav_cumulative_num deaths_7dav_cumulative_prop --- docs/api/covidcast-signals/jhu-csse.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/api/covidcast-signals/jhu-csse.md b/docs/api/covidcast-signals/jhu-csse.md index 99cd6d707..42b6abb00 100644 --- a/docs/api/covidcast-signals/jhu-csse.md +++ b/docs/api/covidcast-signals/jhu-csse.md @@ -23,14 +23,18 @@ University. | Signal and 7-day average signal | Description | |---|---| -| `confirmed_cumulative_num` | Cumulative number of confirmed COVID-19 cases
**Earliest date available:** 2020-01-22 & 2020-02-20 | -| `confirmed_cumulative_prop` | Cumulative number of confirmed COVID-19 cases per 100,000 population
**Earliest date available:** 2020-01-22 & 2020-02-20 | +| `confirmed_cumulative_num` and `confirmed_7dav_cumulative_num` | Cumulative number of confirmed COVID-19 cases
**Earliest date available:** 2020-01-22 & 2020-02-20 | +| `confirmed_cumulative_prop` and `confirmed_7dav_cumulative_prop` | Cumulative number of confirmed COVID-19 cases per 100,000 population
**Earliest date available:** 2020-01-22 & 2020-02-20 | | `confirmed_incidence_num` and `confirmed_7dav_incidence_num` | Number of new confirmed COVID-19 cases, daily
**Earliest date available:** 2020-01-22 & 2020-02-20 | | `confirmed_incidence_prop` and `confirmed_7dav_incidence_prop` | Number of new confirmed COVID-19 cases per 100,000 population, daily
**Earliest date available:** 2020-01-22 & 2020-02-20 | -| `deaths_cumulative_num` | Cumulative number of confirmed deaths due to COVID-19
**Earliest date available:** 2020-01-22 & 2020-02-20 | -| `deaths_cumulative_prop` | Cumulative number of confirmed due to COVID-19, per 100,000 population
**Earliest date available:** 2020-01-22 & 2020-02-20 | +| `deaths_cumulative_num` and `deaths_7dav_cumulative_num` | Cumulative number of confirmed deaths due to COVID-19
**Earliest date available:** 2020-01-22 & 2020-02-20 | +| `deaths_cumulative_prop` and `deaths_7dav_cumulative_prop` | Cumulative number of confirmed due to COVID-19, per 100,000 population
**Earliest date available:** 2020-01-22 & 2020-02-20 | | `deaths_incidence_num` and `deaths_7dav_incidence_num` | Number of new confirmed deaths due to COVID-19, daily
**Earliest date available:** 2020-01-22 & 2020-02-20 | | `deaths_incidence_prop` and `deaths_7dav_incidence_prop` | Number of new confirmed deaths due to COVID-19 per 100,000 population, daily
**Earliest date available:** 2020-01-22 & 2020-02-20 | +| `confirmed_7dav_cumulative_num` | Cumulative number of confirmed COVID-19 cases
**Earliest date available:** 2020-01-22 & 2020-02-20 | +| `confirmed_7dav_cumulative_prop` | Cumulative number of confirmed COVID-19 cases
**Earliest date available:** 2020-01-22 & 2020-02-20 | +| `deaths_7dav_cumulative_num` | Cumulative number of confirmed COVID-19 cases
**Earliest date available:** 2020-01-22 & 2020-02-20 | +| `deaths_7dav_cumulative_prop` | Cumulative number of confirmed COVID-19 cases
**Earliest date available:** 2020-01-22 & 2020-02-20 | These signals are taken directly from the JHU CSSE [COVID-19 GitHub repository](https://github.com/CSSEGISandData/COVID-19) without changes. The From 2f1770910059443a892e1c62d8a22551a8ca3466 Mon Sep 17 00:00:00 2001 From: Tina Townes Date: Tue, 2 Apr 2024 14:08:31 -0400 Subject: [PATCH 04/17] Update docs/api/covidcast-signals/jhu-csse.md Co-authored-by: nmdefries <42820733+nmdefries@users.noreply.github.com> --- docs/api/covidcast-signals/jhu-csse.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/docs/api/covidcast-signals/jhu-csse.md b/docs/api/covidcast-signals/jhu-csse.md index 42b6abb00..19e66bfff 100644 --- a/docs/api/covidcast-signals/jhu-csse.md +++ b/docs/api/covidcast-signals/jhu-csse.md @@ -31,10 +31,6 @@ University. | `deaths_cumulative_prop` and `deaths_7dav_cumulative_prop` | Cumulative number of confirmed due to COVID-19, per 100,000 population
**Earliest date available:** 2020-01-22 & 2020-02-20 | | `deaths_incidence_num` and `deaths_7dav_incidence_num` | Number of new confirmed deaths due to COVID-19, daily
**Earliest date available:** 2020-01-22 & 2020-02-20 | | `deaths_incidence_prop` and `deaths_7dav_incidence_prop` | Number of new confirmed deaths due to COVID-19 per 100,000 population, daily
**Earliest date available:** 2020-01-22 & 2020-02-20 | -| `confirmed_7dav_cumulative_num` | Cumulative number of confirmed COVID-19 cases
**Earliest date available:** 2020-01-22 & 2020-02-20 | -| `confirmed_7dav_cumulative_prop` | Cumulative number of confirmed COVID-19 cases
**Earliest date available:** 2020-01-22 & 2020-02-20 | -| `deaths_7dav_cumulative_num` | Cumulative number of confirmed COVID-19 cases
**Earliest date available:** 2020-01-22 & 2020-02-20 | -| `deaths_7dav_cumulative_prop` | Cumulative number of confirmed COVID-19 cases
**Earliest date available:** 2020-01-22 & 2020-02-20 | These signals are taken directly from the JHU CSSE [COVID-19 GitHub repository](https://github.com/CSSEGISandData/COVID-19) without changes. The From cdda8b8f80fd91a5f61bf827a6ee61acdce7194b Mon Sep 17 00:00:00 2001 From: Tina Townes Date: Wed, 10 Apr 2024 21:01:47 -0400 Subject: [PATCH 05/17] Updated broken link to CDC epiweek Updated broken link to CDC epiweek and checked changes on locally built site. --- docs/api/covidcast-signals/nchs-mortality.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/api/covidcast-signals/nchs-mortality.md b/docs/api/covidcast-signals/nchs-mortality.md index 1a51a7bcf..9e46aa4d4 100644 --- a/docs/api/covidcast-signals/nchs-mortality.md +++ b/docs/api/covidcast-signals/nchs-mortality.md @@ -68,7 +68,7 @@ York State in our reports. We report the NCHS Mortality data in a weekly format (`time_type=week` & `time_value={YYYYWW}`, where `YYYYWW` refers to an epiweek). The CDC defines -the [epiweek](https://wwwn.cdc.gov/nndss/document/MMWR_Week_overview.pdf) as +the [epiweek](https://web.archive.org/web/20210623224758/https://wwwn.cdc.gov/nndss/document/MMWR_Week_overview.pdf) as seven days, from Sunday to Saturday. We check the week-ending dates provided in the NCHS morality data and use Python package [epiweeks](https://pypi.org/project/epiweeks/) to convert them into epiweek From 04269af8a1bd6782fc340df47d7ca10a935ff5be Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 18 Apr 2024 15:48:33 +0000 Subject: [PATCH 06/17] chore(deps-dev): bump aiohttp from 3.9.2 to 3.9.4 Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.9.2 to 3.9.4. - [Release notes](https://github.com/aio-libs/aiohttp/releases) - [Changelog](https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst) - [Commits](https://github.com/aio-libs/aiohttp/compare/v3.9.2...v3.9.4) --- updated-dependencies: - dependency-name: aiohttp dependency-type: direct:development ... Signed-off-by: dependabot[bot] --- requirements.dev.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.dev.txt b/requirements.dev.txt index 818a21553..051a61fdf 100644 --- a/requirements.dev.txt +++ b/requirements.dev.txt @@ -1,4 +1,4 @@ -aiohttp==3.9.2 +aiohttp==3.9.4 black>=20.8b1 bump2version==1.0.1 covidcast==0.1.5 From 988f10a086aa3b5fbca4f9bc2fa64ad003a86108 Mon Sep 17 00:00:00 2001 From: Rostyslav Zatserkovnyi Date: Thu, 25 Apr 2024 16:49:11 +0300 Subject: [PATCH 07/17] Add inspection modes to Python client library (#1418) * Add inspection modes to Python client library * Reset sandbox & debug globals * Update based on reviews * Add auth logging Co-authored-by: george --------- Co-authored-by: george --- dev/local/setup.cfg | 1 + integrations/client/test_delphi_epidata.py | 79 ++++++++++++++++++++++ src/client/delphi_epidata.py | 23 +++++++ 3 files changed, 103 insertions(+) diff --git a/dev/local/setup.cfg b/dev/local/setup.cfg index 7295590c8..b0e16d0cd 100644 --- a/dev/local/setup.cfg +++ b/dev/local/setup.cfg @@ -24,6 +24,7 @@ packages = delphi.epidata.acquisition.twtr delphi.epidata.acquisition.wiki delphi.epidata.client + delphi.epidata.common delphi.epidata.server delphi.epidata.server.admin delphi.epidata.server.admin.templates diff --git a/integrations/client/test_delphi_epidata.py b/integrations/client/test_delphi_epidata.py index 2af923b2b..02a1a9275 100644 --- a/integrations/client/test_delphi_epidata.py +++ b/integrations/client/test_delphi_epidata.py @@ -3,6 +3,7 @@ # standard library import time from json import JSONDecodeError +from requests.models import Response from unittest.mock import MagicMock, patch # first party @@ -41,6 +42,8 @@ def localSetUp(self): # use the local instance of the Epidata API Epidata.BASE_URL = 'http://delphi_web_epidata/epidata' Epidata.auth = ('epidata', 'key') + Epidata.debug = False + Epidata.sandbox = False # use the local instance of the epidata database secrets.db.host = 'delphi_database_epidata' @@ -221,6 +224,82 @@ def test_retry_request(self, get): {'result': 0, 'message': 'error: Expecting value: line 1 column 1 (char 0)'} ) + @patch('requests.post') + @patch('requests.get') + def test_debug(self, get, post): + """Test that in debug mode request params are correctly logged.""" + class MockResponse: + def __init__(self, content, status_code): + self.content = content + self.status_code = status_code + def raise_for_status(self): pass + + Epidata.debug = True + + try: + with self.subTest(name='test multiple GET'): + with self.assertLogs('delphi_epidata_client', level='INFO') as logs: + get.reset_mock() + get.return_value = MockResponse(b'{"key": "value"}', 200) + Epidata._request_with_retry("test_endpoint1", params={"key1": "value1"}) + Epidata._request_with_retry("test_endpoint2", params={"key2": "value2"}) + + output = logs.output + self.assertEqual(len(output), 4) # [request, response, request, response] + self.assertIn("Sending GET request", output[0]) + self.assertIn("\"url\": \"http://delphi_web_epidata/epidata/test_endpoint1/\"", output[0]) + self.assertIn("\"params\": {\"key1\": \"value1\"}", output[0]) + self.assertIn("Received response", output[1]) + self.assertIn("\"status_code\": 200", output[1]) + self.assertIn("\"len\": 16", output[1]) + self.assertIn("Sending GET request", output[2]) + self.assertIn("\"url\": \"http://delphi_web_epidata/epidata/test_endpoint2/\"", output[2]) + self.assertIn("\"params\": {\"key2\": \"value2\"}", output[2]) + self.assertIn("Received response", output[3]) + self.assertIn("\"status_code\": 200", output[3]) + self.assertIn("\"len\": 16", output[3]) + + with self.subTest(name='test GET and POST'): + with self.assertLogs('delphi_epidata_client', level='INFO') as logs: + get.reset_mock() + get.return_value = MockResponse(b'{"key": "value"}', 414) + post.reset_mock() + post.return_value = MockResponse(b'{"key": "value"}', 200) + Epidata._request_with_retry("test_endpoint3", params={"key3": "value3"}) + + output = logs.output + self.assertEqual(len(output), 3) # [request, response, request, response] + self.assertIn("Sending GET request", output[0]) + self.assertIn("\"url\": \"http://delphi_web_epidata/epidata/test_endpoint3/\"", output[0]) + self.assertIn("\"params\": {\"key3\": \"value3\"}", output[0]) + self.assertIn("Received 414 response, retrying as POST request", output[1]) + self.assertIn("\"url\": \"http://delphi_web_epidata/epidata/test_endpoint3/\"", output[1]) + self.assertIn("\"params\": {\"key3\": \"value3\"}", output[1]) + self.assertIn("Received response", output[2]) + self.assertIn("\"status_code\": 200", output[2]) + self.assertIn("\"len\": 16", output[2]) + finally: # make sure this global is always reset + Epidata.debug = False + + @patch('requests.post') + @patch('requests.get') + def test_sandbox(self, get, post): + """Test that in debug + sandbox mode request params are correctly logged, but no queries are sent.""" + Epidata.debug = True + Epidata.sandbox = True + try: + with self.assertLogs('delphi_epidata_client', level='INFO') as logs: + Epidata.covidcast('src', 'sig', 'day', 'county', 20200414, '01234') + output = logs.output + self.assertEqual(len(output), 1) + self.assertIn("Sending GET request", output[0]) + self.assertIn("\"url\": \"http://delphi_web_epidata/epidata/covidcast/\"", output[0]) + get.assert_not_called() + post.assert_not_called() + finally: # make sure these globals are always reset + Epidata.debug = False + Epidata.sandbox = False + def test_geo_value(self): """test different variants of geo types: single, *, multi.""" diff --git a/src/client/delphi_epidata.py b/src/client/delphi_epidata.py index 22fd4d4e2..bc8a2441c 100644 --- a/src/client/delphi_epidata.py +++ b/src/client/delphi_epidata.py @@ -10,12 +10,15 @@ # External modules import requests +import time import asyncio from tenacity import retry, stop_after_attempt from aiohttp import ClientSession, TCPConnector, BasicAuth from importlib.metadata import version, PackageNotFoundError +from delphi.epidata.common.logger import get_structured_logger + # Obtain package version for the user-agent. Uses the installed version by # preference, even if you've installed it and then use this script independently # by accident. @@ -49,6 +52,10 @@ class Epidata: client_version = _version + logger = get_structured_logger('delphi_epidata_client') + debug = False # if True, prints extra logging statements + sandbox = False # if True, will not execute any queries + # Helper function to cast values and/or ranges to strings @staticmethod def _listitem(value): @@ -71,9 +78,25 @@ def _list(values): def _request_with_retry(endpoint, params={}): """Make request with a retry if an exception is thrown.""" request_url = f"{Epidata.BASE_URL}/{endpoint}/" + if Epidata.debug: + Epidata.logger.info("Sending GET request", url=request_url, params=params, headers=_HEADERS, auth=Epidata.auth) + if Epidata.sandbox: + resp = requests.Response() + resp._content = b'true' + return resp + start_time = time.time() req = requests.get(request_url, params, auth=Epidata.auth, headers=_HEADERS) if req.status_code == 414: + if Epidata.debug: + Epidata.logger.info("Received 414 response, retrying as POST request", url=request_url, params=params, headers=_HEADERS) req = requests.post(request_url, params, auth=Epidata.auth, headers=_HEADERS) + if Epidata.debug: + Epidata.logger.info( + "Received response", + status_code=req.status_code, + len=len(req.content), + time=round(time.time() - start_time, 4) + ) # handle 401 and 429 req.raise_for_status() return req From c37aec13a4c0fd0d34b1cc3139b042a22d527b52 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 6 May 2024 20:40:22 +0000 Subject: [PATCH 08/17] chore(deps): bump jinja2 from 3.1.3 to 3.1.4 Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.3 to 3.1.4. - [Release notes](https://github.com/pallets/jinja/releases) - [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst) - [Commits](https://github.com/pallets/jinja/compare/3.1.3...3.1.4) --- updated-dependencies: - dependency-name: jinja2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] --- requirements.api.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.api.txt b/requirements.api.txt index dfaedc0ff..5a04355f9 100644 --- a/requirements.api.txt +++ b/requirements.api.txt @@ -2,7 +2,7 @@ delphi_utils==0.3.15 epiweeks==2.1.2 Flask==2.2.5 Flask-Limiter==3.3.0 -jinja2==3.1.3 +jinja2==3.1.4 more_itertools==8.4.0 mysqlclient==2.1.1 orjson==3.9.15 From e123a94df21630980844b185957c12bb0051e82c Mon Sep 17 00:00:00 2001 From: Tina Townes Date: Mon, 6 May 2024 20:40:13 -0400 Subject: [PATCH 09/17] Moving JHU to inactive Moving JHU to inactive section --- docs/api/covidcast-signals/jhu-csse.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/api/covidcast-signals/jhu-csse.md b/docs/api/covidcast-signals/jhu-csse.md index 19e66bfff..44955430c 100644 --- a/docs/api/covidcast-signals/jhu-csse.md +++ b/docs/api/covidcast-signals/jhu-csse.md @@ -1,6 +1,6 @@ --- title: JHU Cases and Deaths -parent: Data Sources and Signals +parent: Inactive Signals grand_parent: COVIDcast Main Endpoint --- From 58175ce94a2d93cc08e77493fe5fdc7c521ce1bd Mon Sep 17 00:00:00 2001 From: Rostyslav Zatserkovnyi Date: Mon, 13 May 2024 17:10:23 +0300 Subject: [PATCH 10/17] Log HTTP referer and origin (#1429) --- src/server/_common.py | 37 ++++------------------------------- tests/server/test_validate.py | 37 +++++++++++++++++++++++++++++++++++ 2 files changed, 41 insertions(+), 33 deletions(-) diff --git a/src/server/_common.py b/src/server/_common.py index 8633d07fd..33a3f9c48 100644 --- a/src/server/_common.py +++ b/src/server/_common.py @@ -68,6 +68,7 @@ def log_info_with_request(message, **kwargs): remote_addr=request.remote_addr, real_remote_addr=get_real_ip_addr(request), user_agent=request.user_agent.string, + referrer=request.referrer or request.origin, api_key=resolve_auth_token(), user_id=(current_user and current_user.id), **kwargs @@ -114,19 +115,7 @@ def before_request_execute(): user = current_user api_key = resolve_auth_token() - # TODO: replace this next call with: log_info_with_request("Received API request") - get_structured_logger("server_api").info( - "Received API request", - method=request.method, - url=request.url, - form_args=request.form, - req_length=request.content_length, - remote_addr=request.remote_addr, - real_remote_addr=get_real_ip_addr(request), - user_agent=request.user_agent.string, - api_key=api_key, - user_id=(user and user.id) - ) + log_info_with_request("Received API request") if not _is_public_route() and api_key and not user: # if this is a privleged endpoint, and an api key was given but it does not look up to a user, raise exception: @@ -150,28 +139,10 @@ def after_request_execute(response): # Convert to milliseconds total_time *= 1000 - api_key = resolve_auth_token() - update_key_last_time_used(current_user) - # TODO: replace this next call with: log_info_with_request_and_response("Served API request", response, elapsed_time_ms=total_time) - get_structured_logger("server_api").info( - "Served API request", - method=request.method, - url=request.url, - form_args=request.form, - req_length=request.content_length, - remote_addr=request.remote_addr, - real_remote_addr=get_real_ip_addr(request), - user_agent=request.user_agent.string, - api_key=api_key, - values=request.values.to_dict(flat=False), - blueprint=request.blueprint, - endpoint=request.endpoint, - response_status=response.status, - content_length=response.calculate_content_length(), - elapsed_time_ms=total_time, - ) + log_info_with_request_and_response("Served API request", response, elapsed_time_ms=total_time) + return response diff --git a/tests/server/test_validate.py b/tests/server/test_validate.py index f06e9e997..eff7e9c9e 100644 --- a/tests/server/test_validate.py +++ b/tests/server/test_validate.py @@ -26,6 +26,7 @@ def setUp(self): app.config["TESTING"] = True app.config["WTF_CSRF_ENABLED"] = False app.config["DEBUG"] = False + self.client = app.test_client() def test_require_all(self): with self.subTest("all given"): @@ -60,3 +61,39 @@ def test_require_any(self): with self.subTest("one options given with is empty but ok"): with app.test_request_context("/?abc="): self.assertTrue(require_any(request, "abc", empty=True)) + + def test_origin_headers(self): + with self.subTest("referer only"): + with self.assertLogs("server_api", level='INFO') as logs: + self.client.get("/signal_dashboard_status", headers={ + "Referer": "https://test.com/test" + }) + output = logs.output + self.assertEqual(len(output), 2) # [before_request, after_request] + self.assertIn("Received API request", output[0]) + self.assertIn("\"referrer\": \"https://test.com/test\"", output[0]) + self.assertIn("Served API request", output[1]) + self.assertIn("\"referrer\": \"https://test.com/test\"", output[1]) + with self.subTest("origin only"): + with self.assertLogs("server_api", level='INFO') as logs: + self.client.get("/signal_dashboard_status", headers={ + "Origin": "https://test.com" + }) + output = logs.output + self.assertEqual(len(output), 2) # [before_request, after_request] + self.assertIn("Received API request", output[0]) + self.assertIn("\"referrer\": \"https://test.com\"", output[0]) + self.assertIn("Served API request", output[1]) + self.assertIn("\"referrer\": \"https://test.com\"", output[1]) + with self.subTest("referer overrides origin"): + with self.assertLogs("server_api", level='INFO') as logs: + self.client.get("/signal_dashboard_status", headers={ + "Referer": "https://test.com/test", + "Origin": "https://test.com" + }) + output = logs.output + self.assertEqual(len(output), 2) # [before_request, after_request] + self.assertIn("Received API request", output[0]) + self.assertIn("\"referrer\": \"https://test.com/test\"", output[0]) + self.assertIn("Served API request", output[1]) + self.assertIn("\"referrer\": \"https://test.com/test\"", output[1]) From e10de2e66947585bb4eb1e60bb873564ff708339 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 13 May 2024 23:39:22 +0000 Subject: [PATCH 11/17] chore(deps-dev): bump nokogiri from 1.16.2 to 1.16.5 in /docs Bumps [nokogiri](https://github.com/sparklemotion/nokogiri) from 1.16.2 to 1.16.5. - [Release notes](https://github.com/sparklemotion/nokogiri/releases) - [Changelog](https://github.com/sparklemotion/nokogiri/blob/main/CHANGELOG.md) - [Commits](https://github.com/sparklemotion/nokogiri/compare/v1.16.2...v1.16.5) --- updated-dependencies: - dependency-name: nokogiri dependency-type: indirect ... Signed-off-by: dependabot[bot] --- docs/Gemfile.lock | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Gemfile.lock b/docs/Gemfile.lock index 8ecc2115b..6ba650506 100644 --- a/docs/Gemfile.lock +++ b/docs/Gemfile.lock @@ -205,14 +205,14 @@ GEM rb-fsevent (~> 0.10, >= 0.10.3) rb-inotify (~> 0.9, >= 0.9.10) mercenary (0.3.6) - mini_portile2 (2.8.5) + mini_portile2 (2.8.6) minima (2.5.1) jekyll (>= 3.5, < 5.0) jekyll-feed (~> 0.9) jekyll-seo-tag (~> 2.1) minitest (5.17.0) multipart-post (2.1.1) - nokogiri (1.16.2) + nokogiri (1.16.5) mini_portile2 (~> 2.8.2) racc (~> 1.4) octokit (4.20.0) From 23bb71dbc3da58d60d76de1b9f444f629eb26f18 Mon Sep 17 00:00:00 2001 From: Rostyslav Zatserkovnyi Date: Wed, 15 May 2024 18:15:20 +0300 Subject: [PATCH 12/17] Update Python client CHANGELOG (#1437) --- src/client/packaging/pypi/CHANGELOG.md | 30 ++++++++++++++++++++++++++ src/client/packaging/pypi/setup.py | 2 +- 2 files changed, 31 insertions(+), 1 deletion(-) diff --git a/src/client/packaging/pypi/CHANGELOG.md b/src/client/packaging/pypi/CHANGELOG.md index ee95128f7..56af46125 100644 --- a/src/client/packaging/pypi/CHANGELOG.md +++ b/src/client/packaging/pypi/CHANGELOG.md @@ -3,6 +3,36 @@ All notable future changes to the `delphi_epidata` python client will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/). +## [4.1.21] - TBD + +### Includes +- https://github.com/cmu-delphi/delphi-epidata/pull/1418 + +### Added +- Adds two debug flags: + - `debug` logs info about HTTP requests and responses + - `sandbox` prevents any HTTP requests from actually executing, allowing for tests that do not incur server load. + +## [4.1.17] - 2024-01-30 + +### Includes +- https://github.com/cmu-delphi/delphi-epidata/pull/1363 + +### Changed +- Replaced use of deprecated setuptools' `pkg_resources` library with the native `importlib.metadata` library. + +## [4.1.13] - 2023-11-04 + +### Includes +- https://github.com/cmu-delphi/delphi-epidata/pull/1323 +- https://github.com/cmu-delphi/delphi-epidata/pull/1330 + +### Changed +- Appends a trailing slash to URLs requested by the Python client, which should prevent an automatic redirect and an extra request to the server. + +### Removed +- Removed the `covidcast_nowcast()` method, as the associated API endpoint is no longer available. + ## [4.1.11] - 2023-10-12 ### Includes diff --git a/src/client/packaging/pypi/setup.py b/src/client/packaging/pypi/setup.py index 7c0ee051e..e67a03289 100644 --- a/src/client/packaging/pypi/setup.py +++ b/src/client/packaging/pypi/setup.py @@ -13,7 +13,7 @@ long_description_content_type="text/markdown", url="https://github.com/cmu-delphi/delphi-epidata", project_urls={ - "Changelog": "https://github.com/cmu-delphi/delphi-epidata/blob/dev/src/client/packaging/pypi/CHANGELOG.md", + "Changelog": "https://github.com/cmu-delphi/delphi-epidata/blob/main/src/client/packaging/pypi/CHANGELOG.md", }, packages=setuptools.find_packages(), install_requires=["aiohttp", "requests>=2.7.0", "tenacity"], From a84a71bd433c3451d663a4c32258b5e79cf16167 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 16 May 2024 18:53:49 +0000 Subject: [PATCH 13/17] chore(deps): bump rexml from 3.2.5 to 3.2.8 in /docs Bumps [rexml](https://github.com/ruby/rexml) from 3.2.5 to 3.2.8. - [Release notes](https://github.com/ruby/rexml/releases) - [Changelog](https://github.com/ruby/rexml/blob/master/NEWS.md) - [Commits](https://github.com/ruby/rexml/compare/v3.2.5...v3.2.8) --- updated-dependencies: - dependency-name: rexml dependency-type: indirect ... Signed-off-by: dependabot[bot] --- docs/Gemfile.lock | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/Gemfile.lock b/docs/Gemfile.lock index 6ba650506..af94c77e4 100644 --- a/docs/Gemfile.lock +++ b/docs/Gemfile.lock @@ -225,7 +225,8 @@ GEM rb-fsevent (0.10.4) rb-inotify (0.10.1) ffi (~> 1.0) - rexml (3.2.5) + rexml (3.2.8) + strscan (>= 3.0.9) rouge (3.26.0) ruby-enum (0.9.0) i18n @@ -242,6 +243,7 @@ GEM faraday (> 0.8, < 2.0) simpleidn (0.2.1) unf (~> 0.1.4) + strscan (3.1.0) terminal-table (1.8.0) unicode-display_width (~> 1.1, >= 1.1.1) thread_safe (0.3.6) From d0240822dd83f8588805156a0c3ce4d712ffd025 Mon Sep 17 00:00:00 2001 From: Alex Reinhart Date: Fri, 17 May 2024 09:53:55 -0400 Subject: [PATCH 14/17] Another CTIS publication --- docs/symptom-survey/publications.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/symptom-survey/publications.md b/docs/symptom-survey/publications.md index 7f01a8b76..8571cbe3d 100644 --- a/docs/symptom-survey/publications.md +++ b/docs/symptom-survey/publications.md @@ -26,6 +26,10 @@ Pandemic"](https://www.pnas.org/topic/548) in *PNAS*: Research publications using the survey data include: +- Z. Yang, R. Krishnan, and B. Li (2024). [The interplay between individual + mobility, health risk, and economic choice: A holistic model for COVID-19 + policy intervention](https://doi.org/10.1287/ijds.2023.0013). *INFORMS + Journal on Data Science*. - A. Srivastava, J. M. Ramirez, S. Díaz-Aranda, J. Aguilar, A. F. Anta, A. Ortega, and R. E. Lillo (2024). [Nowcasting temporal trends using indirect surveys](https://doi.org/10.1609/aaai.v38i20.30242). In *Proceedings of the 38th AAAI Conference on Artificial Intelligence* 38, From 68ded7ccdf413ac36e251274a77de36faa730edd Mon Sep 17 00:00:00 2001 From: Rostyslav Zatserkovnyi Date: Sat, 18 May 2024 01:19:40 +0300 Subject: [PATCH 15/17] Use bumpversion for epidata client (#1436) --- .bumpversion.cfg | 4 ++-- src/client/delphi_epidata.py | 13 +++---------- src/client/packaging/pypi/CHANGELOG.md | 2 ++ .../packaging/pypi/delphi_epidata/__init__.py | 3 +-- 4 files changed, 8 insertions(+), 14 deletions(-) diff --git a/.bumpversion.cfg b/.bumpversion.cfg index 234774926..21bf3dbb2 100644 --- a/.bumpversion.cfg +++ b/.bumpversion.cfg @@ -9,10 +9,10 @@ tag = False [bumpversion:file:src/client/delphi_epidata.R] +[bumpversion:file:src/client/delphi_epidata.py] + [bumpversion:file:src/client/packaging/npm/package.json] [bumpversion:file:src/client/packaging/pypi/setup.py] -[bumpversion:file:src/client/packaging/pypi/delphi_epidata/__init__.py] - [bumpversion:file:dev/local/setup.cfg] diff --git a/src/client/delphi_epidata.py b/src/client/delphi_epidata.py index bc8a2441c..8c4d54308 100644 --- a/src/client/delphi_epidata.py +++ b/src/client/delphi_epidata.py @@ -15,19 +15,12 @@ from tenacity import retry, stop_after_attempt from aiohttp import ClientSession, TCPConnector, BasicAuth -from importlib.metadata import version, PackageNotFoundError from delphi.epidata.common.logger import get_structured_logger -# Obtain package version for the user-agent. Uses the installed version by -# preference, even if you've installed it and then use this script independently -# by accident. -try: - _version = version("delphi-epidata") -except PackageNotFoundError: - _version = "0.script" +__version__ = "4.1.20" -_HEADERS = {"user-agent": "delphi_epidata/" + _version + " (Python)"} +_HEADERS = {"user-agent": "delphi_epidata/" + __version__ + " (Python)"} class EpidataException(Exception): @@ -50,7 +43,7 @@ class Epidata: BASE_URL = "https://api.delphi.cmu.edu/epidata" auth = None - client_version = _version + client_version = __version__ logger = get_structured_logger('delphi_epidata_client') debug = False # if True, prints extra logging statements diff --git a/src/client/packaging/pypi/CHANGELOG.md b/src/client/packaging/pypi/CHANGELOG.md index 56af46125..aca1e0155 100644 --- a/src/client/packaging/pypi/CHANGELOG.md +++ b/src/client/packaging/pypi/CHANGELOG.md @@ -7,11 +7,13 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/). ### Includes - https://github.com/cmu-delphi/delphi-epidata/pull/1418 +- https://github.com/cmu-delphi/delphi-epidata/pull/1436 ### Added - Adds two debug flags: - `debug` logs info about HTTP requests and responses - `sandbox` prevents any HTTP requests from actually executing, allowing for tests that do not incur server load. +- Fixes the `user-agent` version so that it is correctly set to match the current client release. ## [4.1.17] - 2024-01-30 diff --git a/src/client/packaging/pypi/delphi_epidata/__init__.py b/src/client/packaging/pypi/delphi_epidata/__init__.py index c50efd740..2c92252a6 100644 --- a/src/client/packaging/pypi/delphi_epidata/__init__.py +++ b/src/client/packaging/pypi/delphi_epidata/__init__.py @@ -1,4 +1,3 @@ -from .delphi_epidata import Epidata +from .delphi_epidata import Epidata, __version__ name = "delphi_epidata" -__version__ = "4.1.20" From 05100bb6a29629d553f96d76ddb35511243cc7d1 Mon Sep 17 00:00:00 2001 From: george Date: Fri, 17 May 2024 21:22:44 -0400 Subject: [PATCH 16/17] Update CHANGELOG.md w/ expected release date (#1441) --- src/client/packaging/pypi/CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/client/packaging/pypi/CHANGELOG.md b/src/client/packaging/pypi/CHANGELOG.md index aca1e0155..7cefbf96c 100644 --- a/src/client/packaging/pypi/CHANGELOG.md +++ b/src/client/packaging/pypi/CHANGELOG.md @@ -3,7 +3,7 @@ All notable future changes to the `delphi_epidata` python client will be documented in this file. The format is based on [Keep a Changelog](http://keepachangelog.com/). -## [4.1.21] - TBD +## [4.1.21] - 2024-05-20 ### Includes - https://github.com/cmu-delphi/delphi-epidata/pull/1418 From d05f7fb674910a2f9bace7bc43b0832bd4151c77 Mon Sep 17 00:00:00 2001 From: melange396 Date: Sat, 18 May 2024 05:21:07 +0000 Subject: [PATCH 17/17] chore: release delphi-epidata 4.1.21 --- .bumpversion.cfg | 2 +- dev/local/setup.cfg | 2 +- src/client/delphi_epidata.R | 2 +- src/client/delphi_epidata.js | 2 +- src/client/delphi_epidata.py | 2 +- src/client/packaging/npm/package.json | 2 +- src/client/packaging/pypi/setup.py | 2 +- src/server/_config.py | 2 +- 8 files changed, 8 insertions(+), 8 deletions(-) diff --git a/.bumpversion.cfg b/.bumpversion.cfg index 21bf3dbb2..ec621723a 100644 --- a/.bumpversion.cfg +++ b/.bumpversion.cfg @@ -1,5 +1,5 @@ [bumpversion] -current_version = 4.1.20 +current_version = 4.1.21 commit = False tag = False diff --git a/dev/local/setup.cfg b/dev/local/setup.cfg index b0e16d0cd..6cd99cf69 100644 --- a/dev/local/setup.cfg +++ b/dev/local/setup.cfg @@ -1,6 +1,6 @@ [metadata] name = Delphi Development -version = 4.1.20 +version = 4.1.21 [options] packages = diff --git a/src/client/delphi_epidata.R b/src/client/delphi_epidata.R index e4024b204..268f03a2f 100644 --- a/src/client/delphi_epidata.R +++ b/src/client/delphi_epidata.R @@ -15,7 +15,7 @@ Epidata <- (function() { # API base url BASE_URL <- getOption('epidata.url', default = 'https://api.delphi.cmu.edu/epidata/') - client_version <- '4.1.20' + client_version <- '4.1.21' auth <- getOption("epidata.auth", default = NA) diff --git a/src/client/delphi_epidata.js b/src/client/delphi_epidata.js index 7c684fb78..e53828b15 100644 --- a/src/client/delphi_epidata.js +++ b/src/client/delphi_epidata.js @@ -22,7 +22,7 @@ } })(this, function (exports, fetchImpl, jQuery) { const BASE_URL = "https://api.delphi.cmu.edu/epidata/"; - const client_version = "4.1.20"; + const client_version = "4.1.21"; // Helper function to cast values and/or ranges to strings function _listitem(value) { diff --git a/src/client/delphi_epidata.py b/src/client/delphi_epidata.py index 8c4d54308..6863e4261 100644 --- a/src/client/delphi_epidata.py +++ b/src/client/delphi_epidata.py @@ -18,7 +18,7 @@ from delphi.epidata.common.logger import get_structured_logger -__version__ = "4.1.20" +__version__ = "4.1.21" _HEADERS = {"user-agent": "delphi_epidata/" + __version__ + " (Python)"} diff --git a/src/client/packaging/npm/package.json b/src/client/packaging/npm/package.json index 460042303..95bb6fdf2 100644 --- a/src/client/packaging/npm/package.json +++ b/src/client/packaging/npm/package.json @@ -2,7 +2,7 @@ "name": "delphi_epidata", "description": "Delphi Epidata API Client", "authors": "Delphi Group", - "version": "4.1.20", + "version": "4.1.21", "license": "MIT", "homepage": "https://github.com/cmu-delphi/delphi-epidata", "bugs": { diff --git a/src/client/packaging/pypi/setup.py b/src/client/packaging/pypi/setup.py index e67a03289..c274b466f 100644 --- a/src/client/packaging/pypi/setup.py +++ b/src/client/packaging/pypi/setup.py @@ -5,7 +5,7 @@ setuptools.setup( name="delphi_epidata", - version="4.1.20", + version="4.1.21", author="David Farrow", author_email="dfarrow0@gmail.com", description="A programmatic interface to Delphi's Epidata API.", diff --git a/src/server/_config.py b/src/server/_config.py index c48de0908..8d50ba199 100644 --- a/src/server/_config.py +++ b/src/server/_config.py @@ -7,7 +7,7 @@ load_dotenv() -VERSION = "4.1.20" +VERSION = "4.1.21" MAX_RESULTS = int(10e6) MAX_COMPATIBILITY_RESULTS = int(3650)