Implement pin_download, pin_upload

[machow]

Background

For parity with R pins, a user should be able to do the following:

• upload (R code)
  • accept a List[str] | str of paths to files to upload
  • bundle up as a pin version of type "files"
  • take largely the same arguments as pin_write: name, title, description, metdata
  • create one flat list of all files and directories (included those within subdirectories, etc..)
    • e.g. ['a', 'a/b', 'a/b/file.txt']
  • delegates version uploading to pin_store
• download
  • basically equivalent to pin_read, with type="files" set in data.txt.
  • the R version returns a local cache path, so the python version should return the remote path, since caching happens through wrapping the remote filesystem (???).

Notes on RStudio Connect

Since RStudio Connect already uploads a bunch of files by default (e.g. data.txt, index.html, js assets), the mechanism exists to pin_store arbitrary files / directories. One issue in the R pins implementation is in the following flow:

• Internally, files are just dumped at the top-level first, then its default index.html is added.
• A user attempting to pin_upload "index.html" would have it overwritten by the one pins generates.

Research needed

• Is there a need for pin_download in python pins? Should it download locally by default (e.g. into a folder, similar to what R pins is calling the cache)?
• How does pin_read(...) when type is "files", differ from pin_download?

[machow]

pin_download lists all data files in the pin

Note that for local boards it doesn't need to cache, but for e.g. s3, it lists the files from the cache

[jimhester]

One use-case where this API would be useful is using a different method for reading parquet files than pandas. e.g. using pyarrow, polars or duckdb to read the parquet lazily and avoid reading the full file up front with pandas. On the R side you can use pin_download() and then pass the file to whatever reading function you want to use. On the python side there is currently no equivalent AFAIK.

Alternatively this particular use case could be handled by allowing users to pass a custom function to pin_read().

[machow]

Ah, interesting -- since you can read files directly using fsspec, you should be able to read the files lazily / query them with duckdb.

For example, see the example in this PR:

• fix: RsConnectFs now strips protocol #193

I wonder if there's a good workflow here that allows people to do things similar to pin_upload() and pin_download(), but capitalizes on the power of fsspec?

[jimhester]

Oh, cool, I like the example in the PR👌

[cpcloud]

Just to chime in here, I would definitely prefer pin_download return a list of cached paths, or a thing that would allow me to easily access those cached paths.

[isabelizimm]

merged in #201