DEV Community

#robots

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
Your robots.txt probably does not say what you think it says about AI crawlers

Your robots.txt probably does not say what you think it says about AI crawlers

Comments
4 min read
The Five Paths Our robots.txt Names: Two Are Not on This Host, and Three Carry No Canonical

The Five Paths Our robots.txt Names: Two Are Not on This Host, and Three Carry No Canonical

Comments
3 min read
Two robots.txt Files, One Product: The App Host Has Rules, the API Host Has 1,248 Bytes of Licence and No Directives

Two robots.txt Files, One Product: The App Host Has Rules, the API Host Has 1,248 Bytes of Licence and No Directives

Comments
3 min read
Should your link checker obey robots.txt? Let's argue about it

Should your link checker obey robots.txt? Let's argue about it

Comments
6 min read
One robots.txt Rule to Opt Out of AI Training: What Developers Need to Know (2027)

One robots.txt Rule to Opt Out of AI Training: What Developers Need to Know (2027)

Comments 3
2 min read
How the top 1,000 .jp sites treat AI bots: weekly robots.txt census (Parallel World census)

How the top 1,000 .jp sites treat AI bots: weekly robots.txt census (Parallel World census)

Comments
5 min read
Humanoid Robots in 2026: How Close Are They to Useful Everyday Work?

Humanoid Robots in 2026: How Close Are They to Useful Everyday Work?

Comments
8 min read
GPTBot in robots.txt: the hosting toggle developers need to check

GPTBot in robots.txt: the hosting toggle developers need to check

1
Comments
3 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.