Crawler Session Analysis

The Crawler Session feature displays the chronological sequence of an individual crawl process as a tree diagram.

Activation

• In the timeline a 🕷 icon appears next to every IP address (only when no path filter is active).
• Click 🕷 to analyse the session of that crawler.

What is a Crawler Session?

• All GET requests from the same user agent within the same IP range (IPv4: /24 block, IPv6: exact IP).
• A session ends after 60 minutes of inactivity.
• 304 responses and Google proxy requests (utm_source=gpl) are filtered out.

Tree View

• Requests less than 10 seconds apart appear on the same horizontal level.
• Longer pauses result in a deeper level — with the pause duration displayed.