Most small websites don’t need this, but if you’re a massive site, this is a huge optimization. For example, we had this at AirBNB, when we were getting tens of millions of Googlebot requests a day, but half of the pages were unchanged. By returning a304 Not Modifiedheader, you ...
Expected behavior:When sending seemingly identical requests to the same URL from the same IP address between a Scrapy request vs request module request, I expected both to return the same result with the same HTTP status code. Actual behavior:The Scrapy request returns 403 forbidden while the req...