investigate `parse_connectapi_typed` performance

[toph-allen]

Performance of this can sometimes be slow for large blobs of data. Investigate performance and see if it's even necessary to enforce typing like this.

From posit-dev/connect-extensions#38 (comment).

[toph-allen]

> length(usage_raw)
[1] 485794
> system.time(connectapi:::parse_connectapi_typed(usage_raw, usage_dtype))
   user  system elapsed 
 28.245   1.488  29.798 
> system.time(connectapi:::parse_connectapi(usage_raw))
   user  system elapsed 
 17.602   0.330  18.128 

When parsing data from Connect's new metrics firehose endpoint, approx. half a million rows takes 30 seconds. 2/3 of this time is the baseline conversion from a list to a data frame (i.e. parse_connectapi). Investigate ways to speed this up:

parse_connectapi <- function(data) {
  tibble::as_tibble(purrr::map_df(
    data,
    function(x) {
      purrr::map(
        .x = x,
        .f = function(y) {
          if (is.list(y)) {
            # empty list object gets null
            prep <- purrr::pluck(y, .default = NULL)
          } else {
            # otherwise NA
            prep <- purrr::pluck(y, .default = NA)
          }
          if (length(prep) > 1) {
            prep <- list(prep)
          }
          return(prep)
        }
      )
    }
  ))
}

[nealrichardson]

Some thoughts:

• Is parse_connectapi_typed even necessary? Connect's API is pretty strict in what it returns, isn't it? It feels overly defensive.
• Maybe jsonlite's simplifyDataFrame could be an option, and would surely be faster?
• If we wanted to enforce a column schema after doing simplifyDataFrame, we could still call parse_connectapi_typed on that result, and that would be 1 call instead of N.

[jonkeane]

Is parse_connectapi_typed even necessary? Connect's API is pretty strict in what it returns, isn't it? It feels overly defensive.

I'm not going to go dig all the past examples up, but I've frequently thought the same thing — it feels overly expensive and is a non-trivial amount of work to keep up to date / friction when adding new things. IMO 🚮