Commit 0504ace4dee77b083af0bf9f23c80989a3fa09cd

Authored by Paulo Graça
1 parent 3617b81f2d
Exists in DSpace52++

update robots.txt with new crawlers

Showing 1 changed file with 10 additions and 0 deletions   Show diff stats
dspace-jspui/src/main/webapp/robots.txt
... ... @@ -236,6 +236,16 @@ User-agent: oBot
236 236 Disallow: /
237 237 User-agent: OpenHoseBot
238 238 Disallow: /
  239 +User-Agent: Owlin bot v3 http://owlin.com/bot
  240 +Disallow: /
  241 +User-Agent: Owlin Bot v. 3.0
  242 +Disallow: /
  243 +User-Agent: Owlin Bot
  244 +Disallow: /
  245 +User-Agent: Owlin - http://www.owlin.com
  246 +Disallow: /
  247 +User-Agent: Owlin
  248 +Disallow: /
239 249 User-agent: Plista
240 250 Disallow: /
241 251 User-agent: plukkie
... ...