Commit 0504ace4dee77b083af0bf9f23c80989a3fa09cd
1 parent
3617b81f2d
Exists in
DSpace52++
update robots.txt with new crawlers
Showing
1 changed file
with
10 additions
and
0 deletions
Show diff stats
dspace-jspui/src/main/webapp/robots.txt
... | ... | @@ -236,6 +236,16 @@ User-agent: oBot |
236 | 236 | Disallow: / |
237 | 237 | User-agent: OpenHoseBot |
238 | 238 | Disallow: / |
239 | +User-Agent: Owlin bot v3 http://owlin.com/bot | |
240 | +Disallow: / | |
241 | +User-Agent: Owlin Bot v. 3.0 | |
242 | +Disallow: / | |
243 | +User-Agent: Owlin Bot | |
244 | +Disallow: / | |
245 | +User-Agent: Owlin - http://www.owlin.com | |
246 | +Disallow: / | |
247 | +User-Agent: Owlin | |
248 | +Disallow: / | |
239 | 249 | User-agent: Plista |
240 | 250 | Disallow: / |
241 | 251 | User-agent: plukkie | ... | ... |