RFC: A new user experience for connectapi

[nealrichardson]

Follow pattern of posit-sdk: base Resources and Resource classes, containers and object inheriting from them.

Resources (collections) have an as.data.frame() method to return a data frame (like the get_content() etc. functions do now). Otherwise they are just interfaces.

Assuming R6, you could chain things together with $, like client$users$get_by_id(1)$update(first_name = "Jane"), but that's not idiomatic R. So we can add some helper functions like this:

users <- function(object) {
  object$users
}

# actually, update is a S3 generic in stats, so you'd do this with methods
update <- function(object, ...) {
  object$update(...)
}

Then you can pipe.

client |>
  users() |>
  create(first_name = "John", last_name = "Doe")

client |>
  users() |>
  get_by_id(1) |>
  update(first_name = "Jane")

client |>
  content() |>
  as.data.frame()

client |>
  content() |>
  get_all() # returns a list

Defer all requests until actually needed. I.e. client |> users() doesn't request all users because you don't need them if you're then going to do create() or get_by_id().

The reason to use R6 would be for memoizing the requests. For example, if you do client |> users() |> get_by_id(1), we still haven't made any HTTP requests. Only if you do client |> users() |> get_by_id(1) |> first_name() would we actually make the request, then cache the response data in the object, so that if we then did last_name(), we wouldn't have to request again. And if we instead did client |> users() |> get_by_id(1) |> update(first_name == "Jane"), we would only make the PATCH request, no GET.

We could also handle this caching at the HTTP layer, and then we wouldn't necessarily benefit from R6. https://enpiar.com/r/httpcache/ is a package I wrote for this a while back that we could use.

[toph-allen]

@nealrichardson This issue proposes an update to the way we handle the Connect client: #359

[toph-allen]

I was thinking about this while thinking about naming for functions in #356. I'm going to use that workflow as a way to think about some of the features of that implementation.

One of the things I get hung up on is: Treating API objects as data and accessing them as a data frame (e.g. maybe with jobs() |> as.data.frame()) is really powerful for many workflows, but accessing them as a list of (or as individual) Job objects (e.g. maybe with jobs() |> as.list()) feels necessary for, well, method dispatch / object oriented design. (In this example, jobs() is a function that returns a resource class, a collection of jobs.)

Concretely: #356 is about getting the process log for a content job. At its most basic, the workflow would be something like this:

client() |>
  content() |>
  get_by_id(GUID) |>
  jobs() |>
  get_by_id(x) |> # Returns a Job object
  get_log()

This appears succinct, but it doesn't feel like a real-world workflow — it assumes we already know our job key. That over-simplification elides some of the wrinkles with this approach. A more realistic workflow might involve getting the log for the latest process of a given piece of content. In that approach, you want to:

1. Get all jobs
2. Figure out which one has the latest end date
3. Get the log for that job

You could get the jobs as a data frame, and find your job of interest that way.

item <- client() |>
  content() |>
  get_by_id(GUID)

jobs_df <- item |>
  jobs() |>
  as.data.frame() |>
  dplyr::slice_max(start_time) |>
  dplyr::pull(key)

log <- item |>
  jobs() |>
  get_by_id(key) |>
  get_log()

This feels ergonomically weird, for a few reasons. You have already found the job you want when you call dplyr::pull(key), but you then have to take that key and feed it back into the resource object to get the job object you need to then call a function on it.

You could get the jobs as a list, and find your job of interest that way. This approach is more concise, but worse to read and write.

item <- client() |>
  content() |>
  get_by_id(GUID)

item |>
  jobs() |>
  as.list() |>
  purrr::keep(~ .x$start_date == max(map_dbl(., ~ .x$start_date)))[[1]] |>
  get_log()

Both of those are probably fine. But it'd be nicer to be able to go directly back and forth between one or more rows of a data frame representing jobs and list of job objects. A few galaxy brain ideas:

• The data frame-like object you get from as.data.frame.Jobs() includes a class that lets you call as.list() on it to convert it to a list of job objects, and vice versa.
• The data frame returned by as.data.frame() includes object list-column that contains the objects.

In the first case, you could do something like this:

item <- client() |>
  content() |>
  get_by_id(GUID)

jobs_df <- item |>
  jobs() |>
  as.data.frame() |>
  dplyr::slice_max(start_time) |>
  as.list()[[1]] |>
  get_log()

It may also be that this is too clever by half, and that we're fine with the workflows outlined in the second and third code chunks of this comment. :)

[toph-allen]

Continuing with the jobs example, another way to go from the output of the data frame would be to have a job(key) function that constructed a job object using the API, given a job key. (This would mean unnecessary http calls, but ignoring that for now — I'm assuming this is why @nealrichardson talked about using httpcache)

Mentioned this workflow in a comment here — reformatted to match this proposal.

client |>
  content()
  get_by_id(GUID) |>
  jobs() |>
  as.data.frame()
  slice_max(end_date) |> # get the most recent
  pull(key) |>
  job() |> # construct a job object — in this case just a list
  get_job_log()

But job() would need the client, and in connectapi that's usually the first argument, so this workflow would need to break that paradigm to make the workflow pipe compatible. This is why I've been thinking about #359 (default client as a trailing arg).