I see bots crawling devise pages in my log: 141.8.143.179 - - [06/Oct/2018:00:35:45 -0400] “GET /users/sign_in HTTP/1.1” 200 4022 “-” “Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)”
Should I disallow those paths in robots.txt ?
I see bots crawling devise pages in my log: 141.8.143.179 - - [06/Oct/2018:00:35:45 -0400] “GET /users/sign_in HTTP/1.1” 200 4022 “-” “Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)”
Should I disallow those paths in robots.txt ?
1) why?
2) do your logs show these bots *reading* robots.txt?
Yes