| get_robotstxt | downloading robots.txt file |
| guess_domain | function guessing domain from path |
| parse_robotstxt | function parsing robots.txt |
| paths_allowed | check if a bot has permissions to access page(s) |
| print.robotstxt | printing robotstxt |
| print.robotstxt_text | printing robotstxt_text |
| remove_domain | function to remove domain from path |
| robotstxt | Generate a representations of a robots.txt file |
| rt_cache | get_robotstxt() cache |