Fix 406 Client Error: Not Acceptable errors in test suite because new kind of Fastly filtering/blocking

[amotl]

Problem

There is a test case which validates that the HTTP client croaks appropriately when connecting to non-CrateDB servers.

crate-python/src/crate/client/doctests/http.txt

Lines 184 to 191 in f773d76

	When connecting to non Crate servers the HttpClient will raise a ConnectionError like this::
	>>> http_client = HttpClient(["https://crate.io"])
	>>> http_client.server_infos(http_client._get_server())
	Traceback (most recent call last):
	...
	crate.client.exceptions.ProgrammingError: Invalid server response of content-type 'text/html; charset=UTF-8':
	...

Within that test, http://crate.io was used as an arbitrary domain to connect to. However, it looks like Fastly now blocks excessive/suspicious requests to that domain, for example when running the test matrix on CI, which invokes quite a number of requests, where their origin might also qualify as "unusual" for some fraud detection algorithms.

By default, the Signal Sciences agent returns a “406” response code when a request is blocked (similar to an HTTP 406 NOT ACCEPTABLE response).

-- https://docs.fastly.com/signalsciences/faq/response-codes/#what-is-a-406-agent-response-code

Traceback from CI

We started to observe those hiccups on behalf of CI jobs for #391 on May 13, 2022 at ~18 o'clock MEST.

Failure in test /home/runner/work/crate-python/crate-python/src/crate/client/doctests/http.txt
Failed doctest test for http.txt
  File "/home/runner/work/crate-python/crate-python/src/crate/client/doctests/http.txt", line 0

----------------------------------------------------------------------
File "/home/runner/work/crate-python/crate-python/src/crate/client/doctests/http.txt", line 187, in http.txt
Failed example:
    http_client.server_infos(http_client._get_server())
Expected:
    Traceback (most recent call last):
    ...
    crate.client.exceptions.ProgrammingError: Invalid server response of content-type 'text/html; charset=UTF-8':
    ...
Got:
    Traceback (most recent call last):
      File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/doctest.py", line 1337, in __run
        compileflags, 1), test.globs)
      File "<doctest http.txt[48]>", line 1, in <module>
        http_client.server_infos(http_client._get_server())
      File "/home/runner/work/crate-python/crate-python/src/crate/client/http.py", line 419, in server_infos
        _raise_for_status(response)
      File "/home/runner/work/crate-python/crate-python/src/crate/client/http.py", line 209, in _raise_for_status
        raise ProgrammingError(message)
    crate.client.exceptions.ProgrammingError: 406 Client Error: Not Acceptable

-- https://github.com/crate/crate-python/runs/6409411353#step:6:196

Solution

I thought about using the webserver internal to the testsuite, but just changing the URL to https://httpbin.org/html was an even easier fix. We can come back to the former solution if this ones proves to be brittle as well.

/cc @WalBeh

[seut]

👍