Commit c990d73b41ccdfffc99e81835c0f76e085a8d85e

Authored by Paulo Graça
1 parent 0504ace4de
Exists in DSpace52++

Update robots.txt with new crawlers

Showing 1 changed file with 2 additions and 0 deletions   Show diff stats
dspace-jspui/src/main/webapp/robots.txt
... ... @@ -218,6 +218,8 @@ User-agent: jobs.de-Robot
218 218 Disallow: /
219 219 User-agent: magpie-crawler
220 220 Disallow: /
  221 +User-agent: MauiBot
  222 +Disallow:/
221 223 User-agent: MaxPointCrawler
222 224 Disallow: /
223 225 User-agent: meanpathbot
... ...