<< Back to Blog

Web crawler behavior

When you the need to analyse your website popularity arises, the first idea that comes to mind is to use the specialised services provided by the search engine developers. While this is the most common course of action, there exists another approach to website popularity analysis.  You can comprehensively monitor behaviour of all web-crawlers visiting your website instead.

With this in mind, we created a system that allows you to:

  1. Keep track of web-crawler visits to your website. You can use this to: 
    • See how web-crawlers react to the changes in your website content; 
    • Detect anomalous absences of web-crawler visits; 
    • Track both short-term and long term web-crawler visit tendencies.
  2. Detect web-crawler page requests that resulted in error return codes.
  3. Learn how well your pages are indexed by different web-crawlers.
  4. Figure out behavior profiles of each specific web-crawler, such as start and end pages of the indexing process.
  5. Look at web-crawler statistics such as:
    • Amount of time a web-crawler spends indexing your website;
    • Number of pages indexed in a single web-crawler visit;
    • Amount of traffic generated by the web-crawler requests;
    • Average time between the web-crawler visits.

At RST Cloud we make web-crawler behaviour analytics easy to read and understand. We also include detailed metric explanations and suggestions on website optimisation. 

And we do much more for you at RST Cloud!

Posted on October 26, 2015 by Nikolay Arefiev