Follow
I made a robots.txt. Am I missing any?
User-agent: CCBot
User-agent: ChatGPT-User
User-agent: FacebookBot
User-agent: Google-Extended
User-agent: GPTBot
User-agent: Omgilibot
Disallow: /
@foxxy ...oh, is this for art-stealer-scrapers and not crawlers in general? Might not be relevant then.
@topaz Oh hehe, am derp and should have added a bit more explanation but yepyep you got it :3 I still want my thingies to be found and indexed.
@foxxy "bytedance" or something. Apparently Tiktok has a crawler.
(I think we just blocked anything that contains "bytedance", but we're using Nginx patternmatching and 403ing them. For robots.txt we just use *, EVERYONE keep out.)