DEV Community

Cover image for wget proxy: https_proxy, auth, .wgetrc and exit codes
proxymint
proxymint

Posted on Originally published at proxymint.com

wget proxy: https_proxy, auth, .wgetrc and exit codes

Originally published on proxymint.

Run wget through a proxy with https_proxy or -e, pass credentials safely, set .wgetrc and no_proxy, read exit codes 4 and 8, and see why SOCKS5 fails.

TL;DR

  • Set https_proxy, in lowercase, for https:// downloads. wget picks the variable by the scheme of the URL you fetch, so http_proxy alone leaves every https:// download going direct, and it never reads HTTPS_PROXY in capitals.
  • For one command, use -e. wget -e use_proxy=yes -e https_proxy=http://HOST:PORT URL changes nothing else on the machine, which is what you want while you are still testing.
  • Put the password in --proxy-password if it has an @ or a colon. In the URL form a raw @ breaks the parse (p@ss:word stops with 'Bad port number'), and the flags win over URL credentials anyway.
  • wget can't do SOCKS5. A socks5:// value fails with 'Unsupported scheme', so you use the provider's HTTP endpoint, or curl, for that job.
  • A 407 on an https:// URL exits 4, not 6. wget prints 'Proxy tunneling failed: Proxy Authentication Required' and reports it as a network failure, which throws off scripts that check for code 6.

Key takeaways

  • Set http_proxy and https_proxy together. A gateway that redirects http:// to https:// sends the second leg through https_proxy, so a half-set shell quietly goes direct after the redirect.
  • Check the first output line before trusting a run. 'Connecting to HOST:PORT' is the proxy; 'Resolving site...' followed by the site's own address means wget went around it.
  • Keep .wgetrc at chmod 600 if it holds a password. And remember it overrides your environment variables, which is the usual reason an exported proxy seems to do nothing.
  • Don't share -d output as is. It prints the Proxy-Authorization header, and that base64 string decodes straight back to your username and password.
  • Test rotation with separate wget runs. Read the exit address from an IP echo endpoint each time and compare the country too, not just the address.

Read the full guide on proxymint, with the commands, tables and sources.

Top comments (0)