Webinar Insights: Indexing and Crawling

October 7, 2021
0 minute read

Getting a web page indexed doesn’t guarantee that it will rank well in search engines and bring organic traffic to your site. However, not getting your web pages indexed guarantees that they will not rank. In other words, you’ve got to be in it to win it: your pages need to be discoverable, crawled, and indexed if you want traffic. This article focuses on some of the key insights about how search engines crawl and index pages from our recent webinar Indexing and Crawling: What You Should Know. 

The Finite Resources of Search Engines

Indexing is only possible if your website’s pages are discoverable so that search engines can crawl them. A critical point is that search engines have finite resources—their budgets for crawling are constrained by three factors:

  1. Financial costs, such as electricity for running servers and hiring staff to maintain those resources
  2. Computational costs stemming from the need to increase computing resources, such as servers, to crawl growing volumes of web pages
  3. Environmental expenses: sustainability is a tenet of how companies like Google operate and will operate in the future, so crawling more pages leaves more of an environmental footprint


These three factors result in search engines crawling and indexing websites from the perspective of efficiency versus effectiveness. From an initial fetch of web pages that meet a quality and relevance threshold for a particular search term, the search engine then fine-tunes the top results over multiple phases of crawling. 


Even if a search engine had infinite resources, its value proposition would remain the same: providing customers (anyone searching something online) with the best results. In a world of infinite resources, the search engine would still only select the most relevant, high-quality pages for particular search terms. It’s the job of website owners and content creators to create those high-quality pages.

How Search Engines Determine Importance

Search engines such as Google or Bing ultimately want to answer one question when choosing whether to crawl, index, and rank a page:


Is this URL worth it?


Given the size of the Internet, the volume of pages published each day, and finite resources, search engines need a systematic approach to crawling and indexing. For context, Bing discovers 70 billion new URLs every single day. The system for crawling is based on:


  • Discovery: going out and finding new URLs
  • Schedules (machine learning systems predict when pages meaningfully change and recrawl them)
  • Queues
  • Thresholds for being crawled and indexed
  • Tiers of importance

The prioritization for crawling is mostly driven by importance. This importance is influenced by two factors:


  1. Demand: dynamically figuring out what customers want, and crawling or pruning pages from results as appropriate based on search trends, seasonality, etc
  2. Safe crawling: protecting website owners by not overburdening their site’s performance with excessive crawl requests


When crawling a specific website, search engines use a breadth-first approach, which finds pages along the shortest available paths by following layers downward (e.g. homepage -> pillar page -> sub-page). This approach contrasts with the alternative of going deep, which would mean going straight from the home page to lower-level pages deep within the URL hierarchy. 


The idea of the breadth-first approach is to ensure search engines follow the most important URLs. Many lower-level URLs could be redundant, boilerplate pages, and indexing all of them in search results would worsen the user experience for people searching online. Search engines attempt to detect clusters of very similar pages on your site and find a canonical (master copy) of the page among them to index. 

How To Make Your Pages Discoverable

In order for your web pages to be discoverable in the first place, you need to get the technical basics in place. Some actionable ways to make your content discoverable are:


  • Make sure pages are well connected 
  • Generate an XML sitemap that indicates the structure of your site to search engines and includes all pages you want to be indexed
  • Consistently use intent and query-driven internal linking anchor text to link pages
  • Group topics into strong relatedness clusters
  • Making sure pages are crawlable and renderable
  • Consolidate weaker related pages into one stronger page about a topic
  • Indicating in URLs and file names what those resources are about (e.g. example.com/best-smartwatches)


Consider the search engine as your customer—you want to have a solid information architecture in place that helps your customer easily find and navigate the various pages on your website. In other words, by viewing the search engine as a user trying to digest your content, you can better serve the search engine’s needs and improve your chances of being crawled and indexed. 

Why Does Consistency Matter to Search Engines?

If you can be consistent in applying these practices across your web pages, you’re in a better position to be discovered, crawled, and indexed. The reason that consistency in a site’s structure matters comes down to the machine learning algorithms that underpin how search engines function. 


Search engine machine learning algorithms essentially use judgment to determine which URLs among a set of discoverable pages are good and worth indexing and which aren’t worth indexing for specific queries. The training set for these algorithms to improve their judgment is the entire Internet.


Consistency in site structure and information architecture matters because similar consistencies already exist in currently ranked websites. If you take a random sample of the Internet, patterns emerge across sites, such as the use of internal search, about pages, product pages, blog pages, etc. Machine learning algorithms notice these patterns, which means that consistently applying them to your own site helps the algorithms more easily retrieve and index your pages.


The role of consistency is particularly important when considering whether to use a content management system for your site. You should have a really compelling business reason to use a bespoke, custom-coded website. Search engine algorithms already recognize the site information architecture patterns of popular CMS platforms and website builders, which makes it easier to get indexed out-of-the-box. 

How Search Engines Judge Quality

The advice for getting your indexed pages ranking well in search engines often focuses on providing quality content, but how exactly do search engines algorithmically judge quality as they crawl a website? Here are some pointers:


  • Write original content
  • Don’t stuff keywords into pages
  • Avoid publishing machine-generated content 


Understand that each page matters. If you have a website that is 20 percent great content and 80 percent junk or spun content, the site will ultimately be flagged as low quality and it’ll be very hard to rank well with any page. This brings the discussion full circle to the customer analogy—if you provide customers with low-quality products most of the time and high quality for a small fraction of the time, they won’t return to do business with you. 

The Future of Crawling and Indexing

Over the next 5-10 years, search engines are likely to get far better at supervised and semi-supervised machine learning techniques and more accurate at content extraction through natural language processing techniques. Websites will play a more prominent role in helping search engines crawl and index content through API-driven integrations. 


Ultimately, search engines will aim to become more efficient at crawling and indexing web pages as the web continues to grow exponentially. For website owners, having a solid information architecture and publishing original, high-quality content will only become more important to get crawled and indexed amongst the mass of new content published each day.       


Did you find this article interesting?


Thanks for the feedback!
A screenshot of a plumber's website with a
By Renana Dar May 5, 2025
Many SMBs still hesitate to embrace eCommerce. As the agency partner, you have the opportunity to tear down the perceived walls of eCommerce and show clients how eCommerce can make their business more efficient, accessible, and profitable. Read all about it!
A computer screen with a graph on it and a purple background.
By Santi Clarke April 24, 2025
Learn how platform ecosystems drive revenue and why they are essential for the growth of SaaS businesses.
By Santi Clarke April 24, 2025
One of the greatest challenges for SaaS platforms is keeping users engaged long-term. The term “stickiness” refers to a product's ability to retain users and make them want to return. In the context of SaaS platforms, creating a sticky product means that users consistently find value, experience seamless interactions, and continue using the product over time. The following are 7 practical strategies you can take to improve the stickiness of your SaaS solution. 1. Offer websites that help customers build their digital presence One of the most effective ways to make your SaaS platform sticky is by offering websites to your users. Many businesses today need an online presence, and by providing a platform where your customers can easily build and manage their websites, you increase their reliance on your product. When you offer users a website-building solution, you’re helping them create something foundational to their business. Websites, in this case, aren’t just a tool—they become a part of their identity and brand. This deepens their engagement with your platform, as they need your product to maintain and update their site, ultimately making them less likely to churn. Plus, websites naturally encourage frequent updates, content creation, and customer interactions, which means your users will return to your platform regularly. When you can give your users the tools to create something so essential to their business, you make them more dependent on your platform. This creates a higher barrier to exit, as migrating a fully built website to another service is no small task. In fact, websites are some of the stickiest products you can sell, so adding them to your product portfolio can be one of the best decisions you can to keep your customers using your technology for the long haul. 2. Deliver continuous value through product innovation The key to keeping users coming back to your SaaS platform is ensuring that they consistently see value in it. This means not only meeting their immediate needs but also evolving to address their growing demands. Constant product innovation is essential for keeping your users satisfied and invested in your platform. One way to achieve this is through regular updates that add new features or improvements based on user feedback. A SaaS platform that evolves with its users will keep them engaged longer, making it harder for competitors to steal their attention. Encourage user feedback and prioritize updates that create tangible improvements. This creates an ongoing relationship with your users, which boosts stickiness. 3. Offer a multi-product solution Another powerful way to increase your platform’s stickiness is by offering a suite of products or features that integrate well together. When your users adopt multiple products, they are more likely to stay because they become embedded in your ecosystem. The benefits of this strategy are clear. Research shows that once users adopt more than one product, especially when they integrate >4 tools into their workflow, their likelihood of churn decreases significantly. This happens because the more a user integrates into your suite of products, the harder it is for them to switch to a competitor. These users have invested time in learning your ecosystem and rely on it for their day-to-day operations, making it much harder for them to make the switch. 4. Create a personal connection with your users Human connection is one of the most powerful drivers of user retention. People don’t want to feel like they’re using a cold, faceless platform. By offering exceptional customer support, personalized communication, and community engagement, you build a relationship with your users that goes beyond the product itself. Make sure your support team is responsive, knowledgeable, and empathetic. You can also consider offering tailored onboarding experiences to ensure users understand how to make the most of your platform. When users feel like their success matters to you, they are more likely to remain loyal. 5. Leverage data to personalize the user experience Using data to drive personalization is another strategy that can significantly increase the stickiness of your platform. By tracking user behavior and usage patterns, you can tailor the experience to each individual user’s needs. This could mean recommending features they haven’t yet explored or sending them reminders about tools they may not be fully utilizing. Personalization gives users the feeling that the platform was designed specifically for them, making it harder to walk away from. By demonstrating that you understand their unique needs, you can build a stronger connection and ultimately increase retention rates. 6. Focus on seamless integrations and API capabilities To further increase stickiness, consider expanding your product’s ability to integrate with other tools your users already rely on. Whether it’s email marketing software, CRM systems, or social media management tools, seamless integrations add tremendous value by making it easier for users to incorporate your platform into their existing workflows. The more your product can work in tandem with other popular tools, the more indispensable it becomes. In fact, users who depend on integrations are less likely to churn since their entire ecosystem is tied to your platform’s functionality. 7. Encourage user advocacy and community building User advocacy is another powerful tool in building a sticky product. When users feel a sense of community or even ownership over the platform, they become your most passionate promoters. Encourage your users to share their success stories, join community forums, or contribute to product development through beta testing or feedback loops. A thriving user community not only increases user engagement but also creates a sense of loyalty. When users are part of something larger than themselves, they are more likely to remain committed to your platform, reducing churn and increasing lifetime value. Create deep, lasting customer relationships Making your SaaS platform sticky is all about creating a deep, lasting connection with your users. This requires building a platform that continuously delivers value, creating a seamless and personalized experience, and integrating features that keep users coming back. By focusing on product innovation, offering a multi-product ecosystem, and fostering strong user relationships, you’ll be well on your way to reducing churn and boosting user retention. Stickiness isn’t just a nice-to-have; it’s essential for long-term success. Focus on creating a platform that users can’t imagine living without, and you’ll see them stick around for the long haul.
Show More

Latest posts