What would a good fediverse crawler have to look like?
Ignore robots.txt
It would have to be separate from the database and easy to set up elsewhere incase someone bans it's IP or even IP range
Fake the user agent and possibly even pretend it's a human
That's a shitton of work though so why even bother