Are you trying to find out how User-agent: Scrapy ended up in your Robots.txt for WordPress? It’s caused by a module or plugin that is installed and activated.
User-agent: Scrapy
Allow: /User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpSitemap: https://www.stephanpringle.com/sitemap.xml
WordPress Plugin
The plugin is most likely the Termly | GDPR/CCPA Cookie Consent Banner. You will decide if you will want to keep or remove the plugin. As a few “low” star reviews are left in the review and support forum, you may end up removing the plugin so plan ahead so that your sites can still provide the information and features that will be lost upon the removal.
The following two tabs change content below.
Stephan Pringle
Technology Support Specialist at Sipylus
About The Author: Stephan Pringle is an Information Technology Support Specialist. He covers hardware and software and provides tips for you to troubleshoot and repair issues on your own. In his spare time, he writes articles about the State of New York on his Hackintosh and HackBook and that has helped him to become the top contributor of the New York City section of Yahoo! Answers.
Latest posts by Stephan Pringle (see all)
- Setting Up a New Switch - Monday, April 22, 2024
