AntoineGagne/robots

A parser for robots.txt with support for wildcards. See also RFC 9309.

语言：Erlang · GitHub 星标：3 · Fork：2

话题：crawling · erlang · erlang-library · parser · parsing · parsing-library · rfc-9309 · robots-exclusion-standard

在 GitHub 上查看原仓库