In this post, which builds on our tutorial about scraping the web without being blocked, we will cover almost all of the available scraping tools for Python. We will discuss both pros and cons of each, starting with the basic one. It’s impossible for us to cover every aspect of every tool, but we hope and expect that this post will give you a good understanding of what each tool does, and when to use it.
It should be assumed that when I talk about Python I am referring to Python 3.
Web Fundamentals
It is not easy to view a simple web page in your browser because of the many underlying technologies and concepts involved. As I said, I don’t pretend to explain all, but I will explain the most important aspects of the Python scrape website.
Protocol for transmitting hypertext
Client-server technology is used by HTTP (Hypertext Transfer Protocol). Clients (browsers, Python programs, cURL, Requests…) open connections and send messages (“I want to see the /product page”) to HTTP servers (Nginx, Apache…). Once the server responds (the HTML code for example), the connection is closed.
Below are some of the most relevant header fields:
- Server name: This is a domain name associated with the server. The default port number is 80 if none is specified.
- UA: Includes information about the client that originated the request, such as the operating system. Chrome is the web browser on Mac OS I am using in this case. This header serves two purposes: it can be used for statistics (how many users visit my website on mobile versus desktop) or to prevent bots from violating my website. In this context, “Header Spoofing” is defined as the modification of headers sent by clients. Our scrapers will look just like regular web browsers – just like a regular web browser.
- Here are the types of content that are accepted as responses. Many types and subtypes of content are available: plain text, plain html, image/jpg, application/json…
- Cookie : The header contains a pair of names and values (name1=value1, name2=value2). Data is stored in these session cookies. The use of cookies by websites aids in the authentication of the user as well as storing data in the browser. A server checks whether the credentials you enter when you fill out a login form, for example. In that case, you’ll be redirected and a cookie will be stored in your browser. After that, every time you visit that server, your browser will send that cookie.
- Referrer: This header specifies the URL that was requested to determine the actual URL. It is important to understand this header because websites use it to change their behavior depending on where a user is coming from. You can view the full content of many news websites even if you have a subscription, but if you come from an aggregator such as Reddit, you can view the whole post. They check the referrer to see this. To extract the content we want, we will sometimes need to spoof this header.
