Web Scraping With Selenium And A Raspberry Pi

Introduction

The web and it's websites are complicated. So basic web scraping just won't cut it when it comes to things like logins, forms, and pagination. Well, let's learn how to get what we want using Python, Selenium and a Raspberry Pi.

What is Selenium?

Selenium is software that makes web scraping and automation easier. In a previous tutorial, we talked about the basics of web scraping using the BeautifulSoup Python library, but it has issues if the web data is behind a login, or if the webpage is paginated. Selenium has the capability to bypass all those hurdles, making it a great assistant or replacement for BeautifulSoup.

Coding

A great website for testing our scraping code is quotes.toscrape.com. It's a great resource for trying out and testing scraping techniques. Before we can use Selenium, it first has to be installed. So open up the Raspberry Pi terminal and type

pip3 install selenium

And if you're using a Raspberry Pi for this project, you should also install this Chromium driver

sudo apt-get install chromium-chromedriver

Now you can open up a Python editor and add this code.

Conclusion