How to set a proxy in selenium and Chrome browser

Reading from my previous posts one would quickly realize that my tool of choice when doing any sort of scraping or browser automation is Selenium. I like to play with real browsers that have real "display" on my monitor so I can keep a close eye and also appear as "natural" as possible.

With Selenium I usually pick Firefox as my browser for one reason alone: proxy management is part of the browser's network settings. Chrome on the other hand, although it also has proxy settings, likes to apply those proxies on the system network settings instead. This means that my system is altered and, if that wasn't enough, I can't have more than one browser instance running at any given time and that's a show stopper.

In some instances and for various reasons I'm forced to use Chrome. I could use Firefox and fake the browser header but I don't like to do that because it can be easily traced. Do not despair...this post brings solutions.

Chrome can be made to use a different proxy on each instance by using a plugin/extension. This will require us to spawn the browser with extensions enabled by default. We will also pack and inject an extension for each instance. We have to do it this way because proxies require authentication and the server will prompt for authentication. The plugin needs to fill that in automatically which means that we will have to hardcode the credentials.

I kept the code clean and simple to follow. The chrome driver version used in this example is 74.0.3729.6 and it is tested and working. Use this as the building block for your next scraper and don't forget to read my other articles covering some other aspects when scraping, especially with proxies.

I also created a git repo that also contains the chromedriver that I used to test things out. Just clone the repository, edit scraper.py in your IDE and set your proxy details. When you're done...python scraper.py and watch it go.