There is often a lag time between when you publish content and when search engines discover it and enable searchers to find that content. In a world where many businesses depend on providing timely, relevant, and useful information to their audiences online, crawling and indexing delays are costly. This article describes the new proactive IndexNow crawling approach and Duda’s support of this exciting new initiative.
Crawling and Indexing Time Lags
Traditionally, the discovery, crawling, and indexing of your web pages is a passive process managed by web crawlers. These web crawlers are bots operated by search engines, such as Google or Bing. Web crawlers use software to find, download, and index web content.
When you publish new content on your site, search engines don’t crawl and index that content straight away. There is a delay between publishing and indexing influenced by how search engines operate. Google
cites in its documentation that the main method for finding new web pages is, “following links from pages that we already know about.”
Furthermore, updating existing content also doesn’t result in the most up-to-date version of those pages being crawled and indexed straight away. Search engines have queues and crawling schedules that can result in lag times similar to new pages.
A 2021 episode of
AskGoogleBot featuring John Mueller further clarified this time lag in getting your new or updated indexed. According to Mueller, it can take “several hours to several weeks” for Google to index new content or updated content.
Aside from news publishers, which search engines treat differently, the current state of crawling and indexing is that you can expect to wait an unknown duration of up to several weeks before the content you work hard to produce and refine even gets indexed. For many businesses, this time lag results in missed opportunities to engage with their online audience or even to attract prospective new customers. It’s clear that the web needs a better approach to keep it as relevant as possible. 
A Brief Overview of IndexNow
What if there was a way for search engines to immediately crawl and index your new or updated content as soon as you click the Publish or Update button? This immediate approach would make crawling and indexing a much more efficient and proactive process for all website owners; not just news publishers.
IndexNow provides a new, easy way for website owners to let search engines know about content changes on their web pages. The new open protocol uses an Application Programming Interface (API) to ping search engines and notify them immediately about any content changes. Both Microsoft Bing and Yandex are behind the IndexNow initiative.
IndexNow uses a collaborative approach. When you notify just one search engine that has adopted IndexNow about new or updated content, all other search engines with IndexNow enabled will also get notified about this content. The result is much more relevance and freshness in search results uniformly across the Internet. The more that different search engines adopt IndexNow, the more its impact will become evident in terms of online user experience.
It’s worth concluding this description with an important point about IndexNow: getting indexed faster doesn’t mean you’ll rank better in search results. You still need to create original, high-quality, SEO-optimized content and follow webmaster content guidelines if you want to get the top positions in search engine results pages.
Duda Supports IndexNow for a Fresher Internet
If you wanted to use IndexNow on your own, you’d need some development knowledge and manual work to interact with the API. You’d either need to submit individual or bulk URLs when those web pages were created or received updates. This additional work takes away valuable time better spent designing attractive, valuable websites for clients or for your own business.
At Duda, we want to ensure as many website owners can benefit from this new protocol as possible. We’re proud to support out-of-the-box adoption for IndexNow at Duda, rolling out later this month. In practice, this means that whenever the content changes on your site, we’ll automatically and immediately notify supported search engines of those changes. We’ll make sure that you can benefit from this new approach to crawling and indexing without even needing to do anything.
Closing Thoughts
IndexNow promises to evolve the landscape of search in the following three ways:
Providing users with the freshest content and most relevant results for their searches rather than outdated content.
Website owners can ensure their hard work in creating original content and refining their existing content assets pays off faster by being crawled and indexed swiftly.
From the perspective of search engines, IndexNow enables more efficient use of computing resources by reducing the need for exploratory crawls. A byproduct of this improved efficiency is also a greener Internet with lower environmental impact.
Hopefully, more search engines and platforms will follow the lead of Bing, Yandex, and Duda in pushing for an Internet that works better.
