test std lib TLS implementation against many real world servers #14172
Description
Extracted from #13980.
Here is a small test program:
const std = @import("std");
const http = std.http;
pub fn main() !void {
var arena_instance = std.heap.ArenaAllocator.init(std.heap.page_allocator);
const arena = arena_instance.allocator();
const gpa = arena;
const args = try std.process.argsAlloc(arena);
var client: http.Client = .{
.allocator = gpa,
};
try client.ca_bundle.rescan(gpa);
const url = try std.Url.parse(args[1]);
var req = try client.request(url, .{});
try req.addHeader("Connection", "close");
try req.end();
var bw = std.io.bufferedWriter(std.io.getStdOut().writer());
const w = bw.writer();
var total: usize = 0;
var buf: [5000]u8 = undefined;
while (true) {
const amt = try req.readAll(&buf);
total += amt;
if (amt == 0) break;
std.debug.print("got {d} bytes (total {d})\n", .{ amt, total });
try w.writeAll(buf[0..amt]);
}
try bw.flush();
}This issue tracks real world web servers that cannot be accessed. Here's one for starters:
$ ./test https://www.wikipedia.org/ >out.html
error: TlsDecryptError
/home/andy/Downloads/zig/lib/std/crypto/tls/Client.zig:573:41: 0x34ffb6 in init__anon_6275 (test)
return error.TlsDecryptError;
^
/home/andy/Downloads/zig/lib/std/http/Client.zig:142:30: 0x364fbc in request (test)
req.tls_client = try std.crypto.tls.Client.init(req.stream, client.ca_bundle, url.host);
^
/home/andy/Downloads/zig/build-release/test.zig:17:15: 0x37640f in main (test)
var req = try client.request(url, .{});
^
This issue can be closed when, idk, the "top 100 most popular websites" or some list like that can all be retrieved successfully.
May require working on the http client as well to support chunked encoding and things like this.