Prevent Web Scrapers

Posted on : 7th Oct 2018

There is only one full-proof method against scrapers, that is captcha. But as it affects user experience most of the websites avoid it.

Another option is using AJAX for loading data. This will help avoiding the scrapers which are not built to render JavaScript, but one can make a scraper capable of rendering JavaScript using Selenium WebDriver.
In addition AJAX is also bad for SEO, in case you are into google rankings and all.

A more effiecient and awesome way will be tracking the user behaviour and saving the information into cookies, if something seems suspicious serve a captcha to user. Just how the google captcha works on several sites.

© 2021, All Rights Reserved · Vipin Joshi