Commit 0504ace4dee77b083af0bf9f23c80989a3fa09cd
1 parent
3617b81f2d
Exists in
DSpace52++
update robots.txt with new crawlers
Showing
1 changed file
with
10 additions
and
0 deletions
Show diff stats
dspace-jspui/src/main/webapp/robots.txt
| ... | ... | @@ -236,6 +236,16 @@ User-agent: oBot |
| 236 | 236 | Disallow: / |
| 237 | 237 | User-agent: OpenHoseBot |
| 238 | 238 | Disallow: / |
| 239 | +User-Agent: Owlin bot v3 http://owlin.com/bot | |
| 240 | +Disallow: / | |
| 241 | +User-Agent: Owlin Bot v. 3.0 | |
| 242 | +Disallow: / | |
| 243 | +User-Agent: Owlin Bot | |
| 244 | +Disallow: / | |
| 245 | +User-Agent: Owlin - http://www.owlin.com | |
| 246 | +Disallow: / | |
| 247 | +User-Agent: Owlin | |
| 248 | +Disallow: / | |
| 239 | 249 | User-agent: Plista |
| 240 | 250 | Disallow: / |
| 241 | 251 | User-agent: plukkie | ... | ... |