Scholar by Blackboard
Request an invitation

ScholarBot

For website owners who have seen ScholarBot in their logs.

ScholarBot is the web reader behind Scholar, a study tool from Blackboard that college students use through their school's Blackboard courses. When an instructor links to a web page from inside a course, ScholarBot reads that page so the course's students can study from it.

How to recognise it

Every request ScholarBot makes carries this user agent:

ScholarBot/1.0 (+https://scholar.com/bot; course content ingestion)

Requests come from cloud hosting (Amazon Web Services). We don't publish a fixed list of IP addresses, so the user agent is the way to identify it.

What it reads

  • Pages an instructor linked as a course item. ScholarBot reads a page only when an instructor made it a course link. A site that is only mentioned in a course's text is not read.
  • One step further, only on your site, only for index pages. If a linked page has little text of its own, like an index page, ScholarBot may read up to three of the pages or files it links to on the same site. It never goes further than that one step.
  • Images on pages it has read. When a page shows an image, such as a diagram, ScholarBot may download it to read any text inside it, the way a browser loads it. These image requests use the same user agent, but they are not checked against robots.txt.
  • Return visits. ScholarBot checks back on a page it has read about every 14 days, so students see your current version.

What it doesn't do

  • It doesn't log in, send cookies, or submit forms.
  • It doesn't run JavaScript. A page that needs a browser to show its content is recorded as unreadable and left alone.
  • It doesn't discover or crawl sites that no instructor linked.
  • It doesn't republish your page. Scholar keeps the text so students in that course can search it and ask questions about it. When Scholar answers from it, it names your page as the source, and students open the page itself on your site.

robots.txt

Before reading a page, ScholarBot fetches your site's /robots.txt and follows its Disallow rules for User-agent: * and User-agent: ScholarBot. It applies the same check to every redirect it follows. To keep it off your whole site:

User-agent: ScholarBot
Disallow: /

A few specifics, so there are no surprises:

  • Disallow always wins. ScholarBot doesn't read Allow lines, so an Allow can't reopen a path that a Disallow closes.
  • The * and $ wildcards work as usual. Crawl-delay is not read.
  • If your robots.txt can't be fetched, ScholarBot treats your site as open to it.
  • A page you block is checked again on the same 14-day cycle, so if you lift a block, it takes effect without you contacting anyone.

Contact us

For a question about ScholarBot, or to ask us to stop reading your site, email support@scholar.com. Include your domain, and a sample log line if you have one. A robots.txt rule applies from ScholarBot's next visit without waiting for us.

For how Scholar handles course and student data, see Trust & Security.