Enhance SEA HTTP Client

[varun-edachali-dbx]

What type of PR is this?

• Feature

Description

Enhance the current, simplified SEA Http Client by introducing retries. Most of the auth related functionality is pulled from the Thrift backend. All of the retry-related tests are expected to pass since they represent user-facing functionality (in terms of exceptions to be thrown in particular cases).

We continue using a low-level connection pool instead of using the higher level requests module because the latter does not have first-class support for passing a password to a client certificate, which we want to support. There are workarounds involving subclassing the HttpAdapter and invoking init_poolmanager, but the docstring of the method states that it should not be invoked from client code, so we do not consider it a long term solution.

The exceptions thrown by the SEA and Thrift backends are not completely the same (yet), because this PR passes None to the context and session id of the RequestError object. This misparity will be logged and addressed later. Here, the context is expected to have the following keys as per the existing class definition:

class RequestError(OperationalError):
    """Thrown if there was a error during request to the server.
    Its context will have the following keys:
    "method": The RPC method name that failed
    "session-id": The Thrift session guid
    "query-id": The Thrift query guid (if available)
    "http-code": HTTP response code to RPC request (if available)
    "error-message": Error message from the HTTP headers (if available)
    "original-exception": The Python level original exception
    "no-retry-reason": Why the request wasn't retried (if available)
    "bounded-retry-delay": The maximum amount of time an error will be retried before giving up
    "attempt": current retry number / maximum number of retries
    "elapsed-seconds": time that has elapsed since first attempting the RPC request
    """

    pass

How?

In _make_request(), the client sets up the retry context before making the HTTP request:

• Determines the operation type (EXECUTE_STATEMENT, GET_OPERATION_STATUS, etc.) based on the API path and HTTP method
• Sets the command type on the DatabricksRetryPolicy - this influences retry decisions (e.g., EXECUTE_STATEMENT is only retried for 429/503 status codes)
• Starts the retry timer to track total request duration across all retry attempts
• Makes the HTTP request through urllib3 with the retry policy attached

The retry policy then handles all retry logic automatically - determining when to retry based on status codes and command types, calculating backoff delays, and enforcing attempt/duration limits. If retries are exhausted, urllib3 raises a MaxRetryError which gets caught and wrapped in a RequestError.

How is this tested?

• Unit tests
• E2E Tests
• Manually
• N/A

Related Tickets & Documents

Design Doc

[jayantsing-db]

Few comments

[jayantsing-db]

The exceptions thrown by the SEA and Thrift backends are not completely the same (yet), because this PR passes None to the context and session id of the RequestError object. This misparity will be logged and addressed later.

Could you add details on this? Is there a follow-up PR? what is the context and session id?

[jayantsing-db]

LGTM