Chrome headless proxy authentication

Ever since chrome headless was launched recently folks have been trying to make it work against authenticated proxies. Chrome headless now receives events for proxy authentication via the experimental AuthChallengeResponse handle so it's possible (in theory) to make it work somehow. Puppeteer seems to have introduced a method authenticate that does just that but, by the looks of it, is not yet working for everybody so I'll stay on the safe side of things and provide an alternative.

Puppeteer code (with complaints) should be checked first - maybe it's fixed:

My solution (well not mine but it's what I use) involves setting up an intermediary server that acts as a proxy chain and does the authentication against the proxy server. Our code will connect to this proxy chain which connects to the real server and authenticates. The proxy chain server will not prompt for password against chrome and we will not have an authentication dialog preventing us from doing our job.

It may seem a little too much but it's not that hard since it simply involves another code being executed. You can do this easily with docker and docker-compose and I highly recommend using this technologies for any botting or scraping activities.

For this job we will use a nodejs package called proxy-chain which acts as a middleman. This is the proxy server that we're pointing our headless chrome to. Keep in mind that we're not setting the actual/real proxies on chrome but in our proxy chain server code:

Now that we have our proxy server in place we can go ahead and start our headless chrome instance and point the --proxy-server flag towards our proxy chain server:

My old hack involved setting an extension inside chrome that does the authentication. It's convenient because you don't need an extra server and you have more control over the proxy being used, on what instance and so on. Use what you need, we have solutions

For proxies we recommend checking out our partners at anonymous proxies. They offer shared, dedicated and residential proxies and SOCKSv5.