p0dalirius/robotstester

This Python script can enumerate all URLs present in robots.txt files, and test whether they can be accessed or not.

语言：Python · GitHub 星标：48 · Fork：4

话题：bugbounty · crawler · pentesting · python · robots · tool

在 GitHub 上查看原仓库