Overview
This script helps you build your robots.txt file providing 22 crawler options to better configure the crawling access to your website.
Features
- 22 Crawlers to choose from
- Multiple Sitemaps supported
- Download the file after you finished building it.
- Easy to install
- Easy to use
Requirements
- PHP 5.6
Instructions
Installation :
Unzip the file
Copy the unzipped files to a folder on your histing (create a folder named robots)
Accsess the script like this www.yourwebsite/robots/
Follow the USE INSTRUCTIONS BELOW!
Use :
1. If your website has a sitemap file(it should have one) or multiple files just enter them in the sitemap input box and separate them by comma (,).The comma separation is very important if you have multiple sitemaps.
2. Select the action (allow/disallow),the crawler bot for wich the action is intended and finally enter the filename for your action(if you want the action to be set for all the files on the website just leave it empty).
3. To add more actions for crawler bots just click the add and apply step 2.
4. Now click Generate Text button to preview the file.
5. Click the Download File button(the file will be downloaded to your PC).
6. Copy the file to your hosting and you’re done.
INSTRUCTIONS ARE INCLUDED IN THE ZIP FILE!
