Latest version not working behind proxy!

[aadrian]

Hi,

The latest version of TLDR (3.3.7) does not work behind proxy :(.

Below the errors that are displayed (proxy host and user masked).

Thank you.

C:\work>tldr scp
× Page not found. Updating cache...
Error: Request failed with status code 400
    at createError (C:\Users\XXXXX\AppData\Roaming\npm\node_modules\tldr\node_modules\axios\lib\core\createError.js:16:15)
    at settle (C:\Users\XXXXX\AppData\Roaming\npm\node_modules\tldr\node_modules\axios\lib\core\settle.js:17:12)
    at RedirectableRequest.handleResponse (C:\Users\XXXXX\AppData\Roaming\npm\node_modules\tldr\node_modules\axios\lib\adapters\http.js:231:9)
    at RedirectableRequest.emit (node:events:369:20)
    at RedirectableRequest._processResponse (C:\Users\XXXXX\AppData\Roaming\npm\node_modules\tldr\node_modules\follow-redirects\index.js:401:10)
    at ClientRequest.RedirectableRequest._onNativeResponse (C:\Users\XXXXX\AppData\Roaming\npm\node_modules\tldr\node_modules\follow-redirects\index.js:56:10)
    at Object.onceWrapper (node:events:476:26)
    at ClientRequest.emit (node:events:369:20)
    at HTTPParser.parserOnIncomingClient (node:_http_client:636:27)
    at HTTPParser.parserOnHeadersComplete (node:_http_common:129:17) {
  config: {
    url: 'https://tldr-pages.github.io/assets/tldr.zip',
    method: 'get',
    headers: {
      Accept: 'application/json, text/plain, */*',
      'User-Agent': 'tldr-node-client',
      host: 'tldr-pages.github.io'
    },
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    adapter: [Function: httpAdapter],
    responseType: 'stream',
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    validateStatus: [Function: validateStatus],
    data: undefined
  },
  request: <ref *1> ClientRequest {
    _events: [Object: null prototype] {
      abort: [Function (anonymous)],
      aborted: [Function (anonymous)],
      connect: [Function (anonymous)],
      error: [Function (anonymous)],
      socket: [Function (anonymous)],
      timeout: [Function (anonymous)],
      prefinish: [Function: requestOnPrefinish]
    },
    _eventsCount: 7,
    _maxListeners: undefined,
    outputData: [],
    outputSize: 0,
    writable: true,
    destroyed: false,
    _last: true,
    chunkedEncoding: false,
    shouldKeepAlive: false,
    _defaultKeepAlive: true,
    useChunkedEncodingByDefault: false,
    sendDate: false,
    _removedConnection: false,
    _removedContLen: false,
    _removedTE: false,
    _contentLength: 0,
    _hasBody: true,
    _trailer: '',
    finished: true,
    _headerSent: true,
    _closed: false,
    socket: Socket {
      connecting: false,
      _hadError: false,
      _parent: null,
      _host: 'XXXXX-PROXY',
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 6,
      _maxListeners: undefined,
      _writableState: [WritableState],
      allowHalfOpen: false,
      _sockname: null,
      _pendingData: null,
      _pendingEncoding: '',
      server: null,
      _server: null,
      parser: null,
      _httpMessage: [Circular *1],
      [Symbol(async_id_symbol)]: 24,
      [Symbol(kHandle)]: [TCP],
      [Symbol(kSetNoDelay)]: false,
      [Symbol(lastWriteQueueSize)]: 0,
      [Symbol(timeout)]: null,
      [Symbol(kBuffer)]: null,
      [Symbol(kBufferCb)]: null,
      [Symbol(kBufferGen)]: null,
      [Symbol(kCapture)]: false,
      [Symbol(kBytesRead)]: 0,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(RequestTimeout)]: undefined
    },
    _header: 'GET https://tldr-pages.github.io/assets/tldr.zip HTTP/1.1\r\n' +
      'Accept: application/json, text/plain, */*\r\n' +
      'User-Agent: tldr-node-client\r\n' +
      'host: tldr-pages.github.io\r\n' +
      'Connection: close\r\n' +
      '\r\n',
    _keepAliveTimeout: 0,
    _onPendingData: {},
    agent: Agent {
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      defaultPort: 80,
      protocol: 'http:',
      options: [Object],
      requests: {},
      sockets: [Object],
      freeSockets: {},
      keepAliveMsecs: 1000,
      keepAlive: false,
      maxSockets: Infinity,
      maxFreeSockets: 256,
      scheduling: 'lifo',
      maxTotalSockets: Infinity,
      totalSocketCount: 1,
      [Symbol(kCapture)]: false
    },
    socketPath: undefined,
    method: 'GET',
    maxHeaderSize: undefined,
    insecureHTTPParser: undefined,
    path: 'https://tldr-pages.github.io/assets/tldr.zip',
    _ended: false,
    res: IncomingMessage {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 1,
      _maxListeners: undefined,
      socket: [Socket],
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: true,
      rawHeaders: [Array],
      rawTrailers: [],
      aborted: false,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 400,
      statusMessage: 'Bad Request',
      client: [Socket],
      _consuming: false,
      _dumped: false,
      req: [Circular *1],
      responseUrl: 'http://XXXXX-PROXY:8080/https://tldr-pages.github.io/assets/tldr.zip',
      redirects: [],
      [Symbol(kCapture)]: false,
      [Symbol(kHeaders)]: [Object],
      [Symbol(kHeadersCount)]: 14,
      [Symbol(kTrailers)]: null,
      [Symbol(kTrailersCount)]: 0,
      [Symbol(RequestTimeout)]: undefined
    },
    aborted: false,
    timeoutCb: null,
    upgradeOrConnect: false,
    parser: null,
    maxHeadersCount: null,
    reusedSocket: false,
    host: 'XXXXX-PROXY',
    protocol: 'http:',
    _redirectable: Writable {
      _writableState: [WritableState],
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      _options: [Object],
      _ended: true,
      _ending: true,
      _redirectCount: 0,
      _redirects: [],
      _requestBodyLength: 0,
      _requestBodyBuffers: [],
      _onNativeResponse: [Function (anonymous)],
      _currentRequest: [Circular *1],
      _currentUrl: 'http://XXXXX-PROXY:8080/https://tldr-pages.github.io/assets/tldr.zip',
      [Symbol(kCapture)]: false
    },
    [Symbol(kCapture)]: false,
    [Symbol(kNeedDrain)]: false,
    [Symbol(corked)]: 0,
    [Symbol(kOutHeaders)]: [Object: null prototype] {
      accept: [Array],
      'user-agent': [Array],
      host: [Array]
    }
  },
  response: {
    status: 400,
    statusText: 'Bad Request',
    headers: {
      'cache-control': 'no-cache',
      pragma: 'no-cache',
      'x-xss-protection': '1',
      'content-type': 'text/html; charset=utf-8',
      'proxy-connection': 'close',
      connection: 'close',
      'content-length': '727'
    },
    config: {
      url: 'https://tldr-pages.github.io/assets/tldr.zip',
      method: 'get',
      headers: [Object],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      adapter: [Function: httpAdapter],
      responseType: 'stream',
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: -1,
      maxBodyLength: -1,
      validateStatus: [Function: validateStatus],
      data: undefined
    },
    request: <ref *1> ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 7,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: false,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: false,
      socket: [Socket],
      _header: 'GET https://tldr-pages.github.io/assets/tldr.zip HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'User-Agent: tldr-node-client\r\n' +
        'host: tldr-pages.github.io\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: {},
      agent: [Agent],
      socketPath: undefined,
      method: 'GET',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: 'https://tldr-pages.github.io/assets/tldr.zip',
      _ended: false,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'XXXXX-PROXY',
      protocol: 'http:',
      _redirectable: [Writable],
      [Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype]
    },
    data: IncomingMessage {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 1,
      _maxListeners: undefined,
      socket: [Socket],
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: true,
      rawHeaders: [Array],
      rawTrailers: [],
      aborted: false,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 400,
      statusMessage: 'Bad Request',
      client: [Socket],
      _consuming: false,
      _dumped: false,
      req: [ClientRequest],
      responseUrl: 'http://XXXXX-PROXY:8080/https://tldr-pages.github.io/assets/tldr.zip',
      redirects: [],
      [Symbol(kCapture)]: false,
      [Symbol(kHeaders)]: [Object],
      [Symbol(kHeadersCount)]: 14,
      [Symbol(kTrailers)]: null,
      [Symbol(kTrailersCount)]: 0,
      [Symbol(RequestTimeout)]: undefined
    }
  },
  isAxiosError: true,
  toJSON: [Function: toJSON]
}

[Waples]

Which version (python/r/etc..) of the client are you using?

[MasterOdin]

Looks like Node: 'User-Agent': 'tldr-node-client',

[bl-ue]

That would be the node client, version 3.3.7. I'll transfer this issue there.

[bl-ue]

@aadrian see #326. Do setting the HTTP_PROXY and HTTPS_PROXY environment variables work?

[aadrian]

Do setting the HTTP_PROXY and HTTPS_PROXY environment variables work?

@bl-ue of course they do.

Without them, NPM would not have been able to install TLDR as global in first place with npm install -g tldr and show me the version with tldr --version . I also have several other NPM global tools that also work behind the proxy.

It's also visible that those variables are being used from the error message where you seee something like XXXXX-PROXY - since I replaced the original domain name of the proxy host.

[bl-ue]

Are you able to download https://tldr-pages.github.io/assets/tldr.zip from a browser?

[aadrian]

@bl-ue of course I can get it with FF or Chrome - 1.8 MB.

But what does that have to do with the browser? Browsers don't use HTTP_PROXY variables in corporate environments but dynamic proxy.pac files.

But I can also download it with wget or curl or httpie (and they also use all use those env variables).

[MasterOdin]

It might be that because https://tldr-pages.github.io/assets/tldr.zip is returning a 301, it messes up the client from getting it?

[bl-ue]

I'm sorry for the inconvenience but @aadrian will you please try another client like the Python client? If it works in that one than we can be sure it's a bug in the node-client.

[aadrian]

will you please try another client like the Python client?

@bl-ue I uninstalled the node based client and installed the pip based version:

• it's a totally different version:

C:\>tldr --version
tldr 1.2.0 (Client Specification 1.4)

• it works behind proxy on the first try
• the output is however totally borked on Windows:

C:\>tldr scp

  �[1mscp�[0m

  Secure copy.�[0m
  Copy files between hosts using Secure Copy Protocol over SSH.�[0m
  More information: https://man.openbsd.org/scp.�[0m

  �[32m- Copy a local file to a remote host:�[0m
    �[31mscp �[0mpath/to/local_file�[0m�[31m �[0mremote_host�[0m�[31m:�[0mpath/to/remote_file�[0m�[31m�[0m

  �[32m- Use a specific port when connecting to the remote host:�[0m
    �[31mscp -P �[0mport�[0m�[31m �[0mpath/to/local_file�[0m�[31m �[0mremote_host�[0m�[31m:�[0mpath/to/remote_file�[0m�[31m�[0m

  �[32m- Copy a file from a remote host to a local directory:�[0m
    �[31mscp �[0mremote_host�[0m�[31m:�[0mpath/to/remote_file�[0m�[31m �[0mpath/to/local_directory�[0m�[31m�[0m

  �[32m- Recursively copy the contents of a directory from a remote host to a local directory:�[0m
    �[31mscp -r �[0mremote_host�[0m�[31m:�[0mpath/to/remote_directory�[0m�[31m �[0mpath/to/local_directory�[0m�[31m�[0m

  �[32m- Copy a file between two remote hosts transferring through the local host:�[0m
    �[31mscp -3 �[0mhost1�[0m�[31m:�[0mpath/to/remote_file�[0m�[31m �[0mhost2�[0m�[31m:�[0mpath/to/remote_directory�[0m�[31m�[0m

  �[32m- Use a specific username when connecting to the remote host:�[0m
    �[31mscp �[0mpath/to/local_file�[0m�[31m �[0mremote_username�[0m�[31m@�[0mremote_host�[0m�[31m:�[0mpath/to/remote_directory�[0m�[31m�[0m

  �[32m- Use a specific ssh private key for authentication with the remote host:�[0m
    �[31mscp -i �[0m~/.ssh/private_key�[0m�[31m �[0mlocal_file�[0m�[31m �[0mremote_host�[0m�[31m:�[0m/path/remote_file�[0m�[31m�[0m

[bl-ue]

it's a totally different version

Yeah, that's because it's actually a completely different project. No issue with that, however.

it works behind proxy on the first try

This is unfortunate to hear and quite odd, I guess there might be a bug in the Node.js client.

the output is however totally borked on Windows:

Unfortunate but expected. The garbage in that output is ANSI escape codes used to color the text. They only work on macOS, Linux, and Windows 10 1909+. Node.js uses libuv which intercepts ANSI escape codes from stdout and uses native Windows APIs to change the color of the console.

I'll investigate this within the next few days, for right now I suggest using the Python client until we can some up with a new release or instructions to remedy the sitution (in case it is in fact a user error.)

[bl-ue]

@aadrian will you please send the contents of

• C:\Users\XXXXX\AppData\Roaming\npm\node_modules\tldr\config.json
• C:\Users\XXXXX\.tldrrc (if it exists, it may not)

[aadrian]

... will you please send the contents of

@bl-ue

I uninstalled the PIP version, and installed the Node version again (otherwise the file you requested would be missing) - it's still not working, with the same error as I posted above.

Here is the file zipped, since GitHub does not allow to attache files with JSON extension:
config.zip

It looks exactly like the file on other Windows computer (without proxy where it works).

There's no .tldrrc there yet.