User-agent: Scrapy

Stephan Pringle Wednesday, September 7, 2016

Are you trying to find out how User-agent: Scrapy ended up in your Robots.txt for WordPress? It’s caused by a module or plugin that is installed and activated.

User-agent: Scrapy
Allow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: /sitemap.xml

WordPress Plugin
The plugin is most likely the Termly | GDPR/CCPA Cookie Consent Banner. You will decide if you will want to keep or remove the plugin. As a few “low” star reviews are left in the review and support forum, you may end up removing the plugin so plan ahead so that your sites can still provide the information and features that will be lost upon the removal.

Related posts:

  1. Robots.txt for WordPress
  2. Install a WordPress Plugin
  3. User Agent for Exabot
  4. Locate a Browser’s User Agent
  5. SWEBot/1.0

Stephan Pringle

About The Author: Stephan Pringle is an Information Technology Support Specialist. He covers hardware and software and provides tips for you to troubleshoot and repair issues on your own. In his spare time, he writes articles about the State of New York on his Hackintosh and HackBook and that has helped him to become the top contributor of the New York City section of Yahoo! Answers.

Related Posts