Fast web spider written in Go.
GoSpider is a fast web crawler that parses sitemap.xml and robots.txt file. It can generate and verify link from JavaScript files, extract URLs from various sources and can detect subdomains from the response source.
1.1.6bin/gospideraarch64-darwinaarch64-freebsdaarch64-linuxarmv5tel-linuxarmv6l-linuxarmv7a-darwinarmv7a-linuxarmv7l-linuxi686-darwini686-freebsdi686-linuxloongarch64-linuxm68k-linuxmicroblaze-linuxmicroblazeel-linuxmips-linuxmips64-linuxmips64el-linuxmipsel-linuxpowerpc64-linuxpowerpc64le-linuxriscv32-linuxriscv64-linuxs390-linuxs390x-linuxwasm32-wasiwasm64-wasix86_64-darwinx86_64-freebsdx86_64-linux