Using robotparser package in Python

Robot Parser can help you to understand the robots.txt file before scraping a web site.

Photo by Glenn Carstens-Peters on Unsplash

Introduction

During my daily reads in Medium, I bumped into this quick article where the author gives a couple of tips for those who want to go web scraping.

One of the tips that caught my attention was about the package robotparser from urllib in Python…