Killer Features
Ha! Killer Features in BotScope — most are SEO / webmaster relevant — now it gets exciting:
- Sitemap Matching — BotScope cross-references your XML sitemap against log entries and reveals, for example, which URLs have crawling issues. Which URLs from your sitemap a specific bot — e.g. Googlebot — isn't visiting.
- Sitemap Matching with Google Search Console Data — BotScope combines crawler visits per URL with clicks & impressions — so you can see which URLs are crawled frequently but don't rank well. Those are exactly the low-hanging fruits you can push into performance with minimal effort.
- Google Fake IPs — Every Googlebot hit is immediately verified against 2,000+ official Google networks — refreshed daily. This lets you detect sneaky scrapers impersonating Googlebot to crawl your site behind a robots.txt block under the cover of Google's identity, so you can block them directly on your server. It's usually price grabbers or blocked SEO tools doing the faking — and you'll be surprised how often it happens.
- Google Officials — We sort everything hitting your log from a Google IP. 99% can be attributed to one of Google's many crawlers. The remainder we classify as "Google Officials" — requests from one of the 2,000 Google networks that don't match any known user agent. We believe these could be cloaking-detection bots or other intriguing visitors…
- Block Bots Directly from the Tool — Next to most IPs you'll find a 🚫 icon. Click it and a small popup appears with the deny command for that IP and its range for your server, plus a robots.txt directive where we know one.
- Hidden App Traffic — Many apps have in-app browsers. Sometimes a proxy or server service crawls the content, sometimes the user themselves. We surface the app traffic landing in your logs because it often gets lost in standard analytics or goes completely unnoticed — tracking pixels, ads and similar are filtered out inside apps. Since we can't be certain whether it's a real user IP or a residential proxy, we encrypt the IP on these log entries right from the start to stay GDPR-compliant.
This traffic type reveals:
- Which URLs are actually being shared via WhatsApp?
- What are people discussing about you in Slack?
- Which of your images are being displayed in the Pinterest app?
- Which images are being shown live by Grok in its answers?
- How often and which of your content actually appears in ChatGPT? You can then compare this against real clicks arriving with a UTM parameter — essentially "ChatGPT impressions vs. ChatGPT clicks."
- Which of your URLs are being served by Perplexity?
- How many visitors come to you through the DuckDuckGo app?
- Which of your content is being discussed in Snapchat?
- How often do you appear in Google's NotebookLM?
- How often are your recipes being used in the Recipe maker app?
- Backlinks — We include referrers in the export. Run this over a longer period and you'll surface virtually every active backlink — because many crawlers follow links, and some (especially Chinese crawlers) behave like real browsers and send the referrer header. If a crawler found the link, Google usually has too. This gives you a great up-to-date overview of active inbound links. Believe it or not — I've discovered links this way that no established tool like Ahrefs, SEMrush or MajesticSEO had ever shown me.
- robots.txt Checker — OK, not a killer feature per se — but we validate your robots.txt against RFC 9309 (Garry Illyes had a hand in that one…) so you can always be confident your robots.txt directives are actually correct.
- Google Search Consle (GSC) Referrer Analysis — When you spot a 404 in your data, one click lets you ask Google Search Console whether it knows any referring URLs — a genuine killer feature when you're wondering: how did the crawler (Googlebot in particular) end up on this 404? So you can fix the source. A long-standing SEO headache, especially when the origin is external rather than internal. SEOs are going to love this.