AI2Bot

Academic crawler supporting open AI research initiatives like Semantic Scholar and the Dolma multi-trillion token dataset.

Purpose: Scientific and open-source AI research

Quick Facts

Company: Allen Institute for AI

Respects robots.txt: Yes

Last Updated: 2025-05

Official Documentation

📊 Popularity & Traffic

Smaller volume than commercial bots but highly significant for open-source AI benchmarks.

🤖 User Agent Strings

Use these patterns to identify AI2Bot in your server logs or configure your robots.txt file.

AI2Bot

Respects robots.txt
General research crawler
User Agent Pattern: AI2Bot

AI2Bot-Dolma

Respects robots.txt
Crawler for the Dolma open dataset
User Agent Pattern: AI2Bot-Dolma

🌐 IP Ranges

Source: Allen Institute infrastructure
No specific IP ranges published. Identify this bot using the User Agent strings above.

📝 Robots.txt Configuration

Add the following to your robots.txt file to block AI2Bot:

User-agent: AI2Bot
Disallow: /

💡 Important Notes

• AI2 is a non-profit founded by Paul Allen with a focus on 'AI for the common good'
• Used to build the OLMo (Open Language Model) datasets
• Known to be very respectful of site owner preferences and robots.txt

Other AI Crawlers

OpenAI GPTBot \
Model training data collection
OpenAI OAI-SearchBot \
ChatGPT search indexing and citations
OpenAI ChatGPT-User \
Real-time webpage fetching for user queries
Google Googlebot \
Search indexing and AI model training

Beyond blocking crawlers

See what AI is saying about your brand

Understanding crawlers is step one. With Aiso, you can see the actual conversations happening about your brand inside ChatGPT, Claude, and Perplexity.