Web Scraping With Selenium And A Raspberry Pi
Introduction
The web and it's websites are complicated. So basic web scraping just won't cut it when it comes to things like logins, forms, and pagination. Well, let's learn how to get what we want using Python, Selenium and a Raspberry Pi.
What is Selenium?
Selenium is software that makes web scraping and automation easier. In a previous tutorial, we talked about the basics of web scraping using the BeautifulSoup Python library, but it has issues if the web data is behind a login, or if the webpage is paginated. Selenium has the capability to bypass all those hurdles, making it a great assistant or replacement for BeautifulSoup.
Coding
A great website for testing our scraping code is quotes.toscrape.com. It's a great resource for trying out and testing scraping techniques. Before we can use Selenium, it first has to be installed. So open up the Raspberry Pi terminal and type
pip3 install selenium
And if you're using a Raspberry Pi for this project, you should also install this Chromium driver
sudo apt-get install chromium-chromedriver
Now you can open up a Python editor and add this code.
Comments
This post currently has no responses.