Allow read_surfrad to parse http files (#1459)

* Add cams.get_cams_radiation function

* Revert "Add cams.get_cams_radiation function"

This reverts commit d7deb80.

* Allow parsing of http files

* Add test for https file

* Squashed commit of the following:

commit 5047b26
Author: Prajwal Borkar <[email protected]>
Date:   Tue May 17 19:14:53 2022 +0530

    Updated get_cams protocol to https #1457 (#1458)

    * Updated get_cams protocol to https #1457

    * Updated instances of http to https. #1457

    * Updated documentation links to https

    * Added Contributor

commit a0812b1
Author: roger-lcc <[email protected]>
Date:   Wed May 4 20:01:42 2022 +0800

    CI asv check (#1454)

    * CI asv check

    * added CI asv check

    * CI asv check

    * CI asv check

    * updated CI asv check

    * Update docs/sphinx/source/whatsnew/v0.9.2.rst

    updated v0.9.2.rst

    Co-authored-by: Kevin Anderson <[email protected]>

    Co-authored-by: Kevin Anderson <[email protected]>

commit 83e379a
Author: Kevin Anderson <[email protected]>
Date:   Thu Apr 28 19:26:09 2022 -0400

    Bump pandas to 0.25.0; test updates (#1448)

    * bump pandas min from 0.22.0 to 0.25.0

    * fix buggy test__check_pandas_assert_kwargs

    don't use monkeypatch and mocker in the same test function.  pytest-dev/pytest-mock#289

    * fix psm3 test (apparent_zenith -> solar_zenith)

    * whatsnew

    * better UTC conversion in sun_rise_set_transit_ephem

    * helpful comments

    * more whatsnew

    * '3.0' -> '3' in read_crn test?

    * apply dtypes during parsing in read_crn

    * move dropna() post-processing into read_fwf call

    * fix read_crn for pandas<1.2.0

    * Update pvlib/solarposition.py

    Co-authored-by: Will Holmgren <[email protected]>

    * nix pytz

    * UTC -> utc

    * address simd arccos issue in tracking.singleaxis

    Co-authored-by: Will Holmgren <[email protected]>

commit 8d0f863
Author: Naman Priyadarshi <[email protected]>
Date:   Tue Apr 12 22:55:58 2022 +0530

    Advance numba from 0.36.1 to 0.40.0 in asv py3.6 environment (#1440)

    * Advance numba from 0.36.1 to 0.40.0

    * Advance numba from 0.36.1 to 0.40.0

    * Updated whatsnew.rst

commit 5cb695d
Author: Naman Priyadarshi <[email protected]>
Date:   Wed Apr 6 23:58:03 2022 +0530

    Remove unnecessary **kwargs from spa_python and get_total_irradiance (#1437)

    * Update Solarposition.py

    Removed **kwargs from pvlib.solarposition.spa_python

    * Added v0.9.2.rst, changes in pvlib/irradiance.py and pvlib/location.py

    Made new v0.9.2.rst and removed **kwargs from pvlib/irradiance.py (Line 309) and pvlib/location.py (Line 234-235)

    * Update docs/sphinx/source/whatsnew/v0.9.2.rst

    * Update docs/sphinx/source/whatsnew/v0.9.2.rst

    Co-authored-by: Kevin Anderson <[email protected]>

commit 8460b36
Author: Kevin Anderson <[email protected]>
Date:   Tue Mar 29 15:31:25 2022 -0600

    Finalize 0.9.1 (#1431)

    * fix heading levels in user_guide/bifacial.rst

    * whatsnew cleanup

    * fix readme html

    missing tag, maybe unicode zero-width spaces?

    * readme: link to universal zenodo doi

    * readme: update installation link for #1173

    * whatsnew date

    * additional contributors

    * delete errant space

commit edbf2a6
Author: RoyCoding8 <[email protected]>
Date:   Wed Mar 30 01:58:18 2022 +0530

    Updated plot_singlediode.py (#1434)

    * Update plot_singlediode.py

    Changed the unit from C to degree C (°C)

    * Update plot_singlediode.py

    Changed to LaTeX \degree symbol in matplotlib which avoids any encoding issues with using Unicode characters.

    * Update v0.9.1.rst

    Added name to the contributors' list

    * Update v0.9.1.rst

commit cf4a8ad
Author: Kevin Anderson <[email protected]>
Date:   Tue Mar 29 14:04:40 2022 -0600

    Update sphinx to 4.5.0 (#1435)

    * bump sphinx and pydata-sphinx-theme versions

    * clean up sphinx conf.py

    * fix distutils strangeness, maybe

    * use freshly-released sphinx==4.5.0

commit 884a153
Author: Kevin Anderson <[email protected]>
Date:   Wed Mar 23 13:41:35 2022 -0600

    Clarify delta_t docstring descriptions (#1429)

    * clarify delta_t docstrings

    * whatsnew

commit c243183
Author: Kevin Anderson <[email protected]>
Date:   Thu Mar 17 12:01:57 2022 -0600

    Deprecate pvlib.forecast (#1426)

    * deprecate pvlib.forecast classes

    * catch warnings in tests

    * add warning admonition to forecasts.rst

    * whatsnew

    * stickler

    * pin pytest < 7.1.0

    * pin pytest in the right place this time

    * more warning suppression in tests

    * unpin pytest

    * Update docs/sphinx/source/whatsnew/v0.9.1.rst

    * copy warning to reference/forecasting.rst

commit e3baa12
Author: Kevin Anderson <[email protected]>
Date:   Thu Mar 17 11:28:56 2022 -0600

    Fix conditional dependency on dataclasses (#1422)

    * better conditional dependency on dataclasses

    * whatsnew

commit 27cba7a
Author: Naman Priyadarshi <[email protected]>
Date:   Thu Mar 17 22:48:08 2022 +0530

    Added asv benchmarking badge to the table of badges in the main README. (#1427)

    * Update Readme.md

    Added benchmarks asv badge to the badge section

    * Updated v.0.9.1.rst

    Added my name to the list of Contributers.

commit 1893b20
Author: Adam R. Jensen <[email protected]>
Date:   Mon Mar 14 18:37:58 2022 +0100

    Add variable mapping of psm3 (#1374)

    * Add variable mapping of psm3

    * Add enhancement entry in whatsnew

    * Fix stickler

    * Map keys in metadata dict

    * Remove double spaces in docs

    * Fix stickler

    * Doc update

    Co-authored-by: Kevin Anderson <[email protected]>

    * Reformatting - changes by kanderso-nrel

    * Update docstring table with 2020

    * Add deprecation warning test coverage

    * Rename to VARIABLE_MAP

    * Change apparent_zenith to solar_zenith

    Based on the decision in #1403

    * Update attributes docstring

    * Change elevation to altitude when mapping variables

    * Update psm3 variable mapping test

    Co-authored-by: Kevin Anderson <[email protected]>

* Revert "Squashed commit of the following:"

This reverts commit b313c64.

* Update whatsnew

* Update read_surfrad documentation

Co-authored-by: AdamRJensen <[email protected]>

Author: AdamRJensen

docs/sphinx/source/whatsnew/v0.9.2.rst

@@ -8,6 +8,9 @@ Deprecations
 
 Enhancements
 ~~~~~~~~~~~~
+* :py:func:`pvlib.iotools.read_surfrad` now also accepts remote files
+  with https links in addition to files on the SURFRAD FTP server
+  (:pull:`1459`)
 * Add :py:func:`pvlib.tracking.calc_surface_orientation` for calculating
   single-axis tracker ``surface_tilt`` and ``surface_azimuth`` from
   rotation angles. (:issue:`1471`, :pull:`1480`)
@@ -47,8 +50,9 @@ Requirements
 
 Contributors
 ~~~~~~~~~~~~
+* Adam R. Jensen (:ghuser:`AdamRJensen`)
 * Naman Priyadarshi (:ghuser:`Naman-Priyadarshi`)
 * Chencheng Luo (:ghuser:`roger-lcc`)
 * Prajwal Borkar (:ghuser:`PrajwalBorkar`) 
 * Kevin Anderson (:ghuser:`kanderso-nrel`)
-* Cliff Hansen (:ghuser:`cwhanse`)
+* Cliff Hansen (:ghuser:`cwhanse`)

pvlib/iotools/surfrad.py

@@ -44,7 +44,7 @@ def read_surfrad(filename, map_variables=True):
     Parameters
     ----------
     filename: str
-        Filepath or url.
+        Filepath or URL. URL can be either FTP or HTTP.
     map_variables: bool
         When true, renames columns of the Dataframe to pvlib variable names
         where applicable. See variable :const:`VARIABLE_MAP`.
@@ -113,7 +113,8 @@ def read_surfrad(filename, map_variables=True):
     =======================  ======  ==========================================
 
     See README files located in the station directories in the SURFRAD
-    data archives [2]_ for details on SURFRAD daily data files.
+    data archives [2]_ for details on SURFRAD daily data files. In addition to
+    the FTP server, the SURFRAD files are also available via HTTP access [3]_.
 
     References
     ----------
@@ -122,8 +123,10 @@ def read_surfrad(filename, map_variables=True):
        `SURFRAD Homepage <https://www.esrl.noaa.gov/gmd/grad/surfrad/>`_
     .. [2] NOAA SURFRAD Data Archive
        `SURFRAD Archive <ftp://aftp.cmdl.noaa.gov/data/radiation/surfrad/>`_
+    .. [3] `NOAA SURFRAD HTTP Index
+       <https://gml.noaa.gov/aftp/data/radiation/surfrad/>`_
     """
-    if str(filename).startswith('ftp'):
+    if str(filename).startswith('ftp') or str(filename).startswith('http'):
         req = Request(filename)
         response = urlopen(req)
         file_buffer = io.StringIO(response.read().decode(errors='ignore'))

pvlib/tests/iotools/test_surfrad.py

@@ -7,6 +7,8 @@
 testfile = DATA_DIR / 'surfrad-slv16001.dat'
 network_testfile = ('ftp://aftp.cmdl.noaa.gov/data/radiation/surfrad/'
                     'Alamosa_CO/2016/slv16001.dat')
+https_testfile = ('https://gml.noaa.gov/aftp/data/radiation/surfrad/'
+                  'Alamosa_CO/2016/slv16001.dat')
 
 
 @pytest.mark.remote_data
@@ -19,6 +21,17 @@ def test_read_surfrad_network():
     assert local_data.equals(network_data)
 
 
+@pytest.mark.remote_data
+@pytest.mark.flaky(reruns=RERUNS, reruns_delay=RERUNS_DELAY)
+def test_read_surfrad_https():
+    # Test reading of https files.
+    # If this test begins failing, SURFRAD's data structure or data
+    # archive may have changed.
+    local_data, _ = surfrad.read_surfrad(testfile)
+    network_data, _ = surfrad.read_surfrad(https_testfile)
+    assert local_data.equals(network_data)
+
+
 def test_read_surfrad_columns_no_map():
     data, _ = surfrad.read_surfrad(testfile, map_variables=False)
     assert 'zen' in data.columns