Host external data files in a separate repository

[andersy005]

The most recent PRs introduced notebooks that require external data files. I was wondering if we could host external data files in a separate repository so as to avoid blotted git repositories. Hosting these files in an external files would make it easy for repositories to share data files, and we could use pooch to cache these files on first access.

Relevant PRs:

• Drop in Unidata pandas workshop notebook #34

• Drop in Unidata xarray workshop notebook #33

[dopplershift]

I think some kind of centralized hosting of the data file makes sense. While we may want to have some caching for build/CI purposes, we should avoid exposing the details of that to the users of the training content. Not only does it raise the cognitive load by introducing additional (unrelated) concepts, but it can also just be a general source of confusion. MetPy's examples have a get_test_data helper that pulls data from our static copies on GitHub; we have repeatedly received support requests where users plug their own filenames into the call to get_test_data and wonder why it fails.

[brian-rose]

I'll be delighted if there's an elegant solution to this problem that avoids confusing the audience. I've struggled the data caching issue with climlab. I'll have to take a look at pooch, that's new to me.

[clyne]

It might be worth looking at what the GeoCAT team does for their examples gallery: https://geocat-examples.readthedocs.io/en/latest/

CC'ing @erogluorhan for his input

[erogluorhan]

We have been using a solution (i.e. our geocat-datafiles repo) to such a problem for a while, which is similar to what @andersy005 suggested in the beginning (i.e. we are using pooch). You can get insights from README file, but let me detail our workflow a bit more:

• Add data files into relevant category folders (e.g. "netcdf_files", "hdf_files", etc.) in this repo.
• Keep file records in registry file
• This repo has a trivial user API as it provides geocat.datafiles.get(fname) function. This functionality is distributed as an anaconda package. So, wherever data files are needed, you install geocat-datafiles and then access its get() function to fetch remote files as follows:

   import geocat.datafiles as gdf
   datafile = gdf.get("netcdf_files/meccatemp.cdf")
   ds = xr.open_dataset(datafile, decode_times=False)

Note: As @clyne suggested, we have been using this thing in several of our repos, mostly in GeoCAT-examples. Check for example this script.

What can be the shortcoming of this approach?

• Individual files in a Github repository are strictly limited to a 100 MB maximum size limit. So, when a bigger file is in hand, it is required to subset that file below 100 MB. It could only be doable if the variables of interest in that file are a subset of the entire dataset, and this subsetting adds a little extra work. Furthermore, a single Github repo is recommended to be kept under 5 GB, but I am not sure if it can be negotiable with Github.

Let me know if a similar thing would be useful here.

[clyne]

Re GitHub's 100 MB limit: we have an allocation on NCAR's public DASH repository to host data sets used by Pythia. It's a good choice for larger data sets and was part of our Data Management Plan for the proposal.

[clyne]

Consensus: move forward with the pooch approach.

[andersy005]

As an update: I put together a repo for hosting sample data files. This repo includes a small package for retrieving/caching these data files: https://github.com/ProjectPythia/pythia-datasets. The package can be installed via

python -m pip install git+https://github.com/ProjectPythia/pythia-datasets

for the time being....

In [1]: from pythia_datasets import DATASETS

In [2]: DATASETS.registry_files
Out[2]: ['jan-17-co-asos.txt.xz', 'NARR_19930313_0000.nc']

In [3]: import pandas as pd

In [4]: filepath = DATASETS.fetch('jan-17-co-asos.txt.xz')

In [5]: filepath
Out[5]: '/Users/abanihi/Library/Caches/pythia-datasets/jan-17-co-asos.txt.xz'

In [6]: df = pd.read_csv(filepath)

To update this registry of data files, please open a PR and place the sample data files in the data folder.

[clyne]

@andersy005 will revisit when possible. This is becoming a priority for Max

[andersy005]

I added some documentation here on (1) how to add new data files to the repository, (2) how to use these data files. If folks have some additional feedback, please let me know.

[brian-rose]

I added some documentation here on (1) how to add new data files to the repository, (2) how to use these data files. If folks have some additional feedback, please let me know.

I see both pythia-datasets and pythia_datasets mentioned on the README page. Are some of those typos?

[brian-rose]

Disregard my previous comment!