Search and Pagination for API keys

[ywangd]

Updated 2021-07-22

List of Tasks

To smooth the development and review processes, here is a proposed list of tasks, each rougly corrsponds to a PR:

• Core search feature (A new search API for API keys - core search function #75335)
• Pagination with from, size, sort, search_after (Add sort and pagination support for QueryApiKey API #76144)
• Rest API spec and yaml tests (Query API key - Rest spec and yaml tests #76238)
• Documentation (Add docs for QueryApiKey API #76521)
• HLRC (HLRC support for query API key API #76520)

- [ ] Consistent data view, i.e. PIT support (optional?)

The target release is 7.15 (except support for consistent data view) (edit: we decided to not support consistent data view)

Description

We need a new search API for API keys. It will be a separate one from the existing GetApiKey API because:

• Different verbs (GET, POST) has already been assigned for the endpoint _security/api_key. It is not possible to support GET with a request body (required for complex searches) while POST is used for a different purpose (creating API keys). There are tests actively forbid it.
• The Get API key API has different handling for user security context compare to what normally would be expected from searches. So it is better to have a new API that behaves more like search.

This new API should also support pagination so that it is possible to (1) retrieve a certain page of the search hits; (2) completely enumerate through all search hits especially when the total hit size is large; (3) (optionally) offer a consistent view of search results when necessary.

Examples of proposed request and response are as follows:

Request

GET /_security/_query/api_key
{
  "query": {
    "bool": {
      "must": [
        {
          "wildcard": {
            "username": "org-user*"
          }
        }
      ],
      "filter": [
        {
          "term": {
            "metadata.status": "production"
          }
        },
        {
          "terms": {
            "name": [ "west-key", "east-key" ]
            ]
          }
        }
      ],
      "must_not": [
        {
          "term": {
            "metadata.application": "ad-hoc"
          }
        }
      ]
    }
  },
  "from": 0,
  "size": 1000,
  "sort": [ 
    "name", 
    { 
      "creation_time": {
        "order": "asc",
        "Format":"strict_date_time"
      } 
    }
  ],
  "search_after": [
    "key-2048",
    "2021-07-01T00:00:59.000Z"
  ]
}

Response

{
  "total": 12345,
  "count": 1000,
  "api_keys": [
    {
      "id" : "QsFr8nMBF_G4lMlA7e1R",
      "name" : "key-1",
      "creation" : 1597500026177,
      "invalidated" : false,
      "username" : "foo",
      "realm" : "native",
      "Metadata": { ... }
    },
    ...
  ]
}

Note the usage of from, size, sort and search_after. They work the same way as those for the regular search API.
Note that the total field shows the exact total number of the API keys and count field shows the number of API keys returned in this response.

Consistent Data View (Optional?)

PIT is needed to achieve a consistent view of the search results. But directly exposing PIT for this API is both inefficient and insecure. We propose to have an indirection of the actual pit_id and let the API handles it automatically. Sample APIs are as follows:

GET /_security/_query/api_key
{ "sort": [ "creation_time" ] }

// PIT is automatically created when the search has an explicit sort field. It also gets automatically closed if not used within 1 min.
{
  "total": 12345,
  "count": 10,
  "api_keys": [
    {
      ...
    },
    ...
  ],
  "sort": [
    "2021-07-01T00:00:00.000Z"
  ],
  "pit_id": "b026324"
}

// The pit_id can be passed to the subsequent searches to achieve consistent data view
GET /_security/_query/api_key?pit_id=b026324

---

The following content is original to the issue but now obsolete. It's kept here just for historical reference:

Today API keys can be search/retrieved by specifying a number of search criteria, including id, name, realm_name, username and owner. Since #70292 API keys can now be created with user provided metadata, which takes arbitrary and potentially nested values. It is desirable that API keys can be searched by their metadata.

A few things need to be considered for the development:

• The current GetApi key request only takes query parameters, which is not friendly for specifying complex values. We need come up with a better way to support the metadata search syntax. Ideally this syntax should be expandable to other security entities since many of them have metadata as well.
• Metadata is not secret. Two users can un/intentionally specify the same metadata for different keys (owned by different users). Therefore API keys matched for a common metadata criteria has no guarantee on their ownships.

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[sorenlouv]

As I understand the current `GetApiKey' api returns a single api key (or none if nothing matches). What we'll need for APM's use case is a "find" api that returns all API keys containing specific metadata.

Thanks!

[bytebilly]

I think the approach is already addressing your requirement.
The _security/api_key API returns all API keys that match the search criteria:

GET /_security/api_key?name=key*

{
  "api_keys" : [
    {
      "id" : "qujUgngBiGgjUVdX6w9f",
      "name" : "key1",
      "creation" : 1617102695247,
      "invalidated" : false,
      "username" : "3039038030",
      "realm" : "cloud-saml-kibana-bf0ba2d707054f2eb23c913171381cc9"
    },
    {
      "id" : "tujUgngBiGgjUVdX8Q-i",
      "name" : "key2",
      "creation" : 1617102696780,
      "invalidated" : false,
      "username" : "3039038030",
      "realm" : "cloud-saml-kibana-bf0ba2d707054f2eb23c913171381cc9"
    }
  ]
}

[sorenlouv]

Perfect, just wanted to be sure.