User avatar
nixCraft 🐧 @nixCraft@mastodon.social
2y
Reddit blocked all search engines including bing, DDG, and others, but because of the deal with Google, it is still making money and is the top destination for many Google searches. Here is robots.txt. They are blocking it because AI bots are scrapping and profiting from user-generated content. If you do this you will be dropped from Google immediately but now they have a deal with Google
Reddit.com robots.txt file. Reddit blocked all search engines including bing, DDG, and others, but because of the deal with Google, it is still making money and is the top destination for many Google searches. Here is robots.txt. They are blocking it because AI bots are scrapping and profiting from user-generated content.
12
1
0
0
User avatar
🪨 @Varpie@peculiar.florist
2y
@nixCraft Seems to me like GoogleBot isn't respecting the robots.txt, I'll open an issue, they should definitely fix that.
⭐4
0
0
0
4