Duda Supports Proactive Web Crawling with IndexNow Adoption

October 18, 2021
0 minute read

There is often a lag time between when you publish content and when search engines discover it and enable searchers to find that content. In a world where many businesses depend on providing timely, relevant, and useful information to their audiences online, crawling and indexing delays are costly. This article describes the new proactive IndexNow crawling approach and Duda’s support of this exciting new initiative.

Crawling and Indexing Time Lags

Traditionally, the discovery, crawling, and indexing of your web pages is a passive process managed by web crawlers. These web crawlers are bots operated by search engines, such as Google or Bing. Web crawlers use software to find, download, and index web content. 

When you publish new content on your site, search engines don’t crawl and index that content straight away. There is a delay between publishing and indexing influenced by how search engines operate. Google cites in its documentation that the main method for finding new web pages is, “following links from pages that we already know about.” 

Furthermore, updating existing content also doesn’t result in the most up-to-date version of those pages being crawled and indexed straight away. Search engines have queues and crawling schedules that can result in lag times similar to new pages. 

A 2021 episode of AskGoogleBot featuring John Mueller further clarified this time lag in getting your new or updated indexed. According to Mueller, it can take “several hours to several weeks” for Google to index new content or updated content. 

Aside from news publishers, which search engines treat differently, the current state of crawling and indexing is that you can expect to wait an unknown duration of up to several weeks before the content you work hard to produce and refine even gets indexed. For many businesses, this time lag results in missed opportunities to engage with their online audience or even to attract prospective new customers. It’s clear that the web needs a better approach to keep it as relevant as possible. ï»¿

A Brief Overview of IndexNow

What if there was a way for search engines to immediately crawl and index your new or updated content as soon as you click the Publish or Update button? This immediate approach would make crawling and indexing a much more efficient and proactive process for all website owners; not just news publishers. 

IndexNow provides a new, easy way for website owners to let search engines know about content changes on their web pages. The new open protocol uses an Application Programming Interface (API) to ping search engines and notify them immediately about any content changes. Both Microsoft Bing and Yandex are behind the IndexNow initiative. 

IndexNow uses a collaborative approach. When you notify just one search engine that has adopted IndexNow about new or updated content, all other search engines with IndexNow enabled will also get notified about this content. The result is much more relevance and freshness in search results uniformly across the Internet. The more that different search engines adopt IndexNow, the more its impact will become evident in terms of online user experience. 

It’s worth concluding this description with an important point about IndexNow: getting indexed faster doesn’t mean you’ll rank better in search results. You still need to create original, high-quality, SEO-optimized content and follow webmaster content guidelines if you want to get the top positions in search engine results pages.

Duda Supports IndexNow for a Fresher Internet

If you wanted to use IndexNow on your own, you’d need some development knowledge and manual work to interact with the API. You’d either need to submit individual or bulk URLs when those web pages were created or received updates. This additional work takes away valuable time better spent designing attractive, valuable websites for clients or for your own business. 

At Duda, we want to ensure as many website owners can benefit from this new protocol as possible. We’re proud to support out-of-the-box adoption for IndexNow at Duda, rolling out later this month. In practice, this means that whenever the content changes on your site, we’ll automatically and immediately notify supported search engines of those changes. We’ll make sure that you can benefit from this new approach to crawling and indexing without even needing to do anything.

Closing Thoughts

IndexNow promises to evolve the landscape of search in the following three ways:

Providing users with the freshest content and most relevant results for their searches rather than outdated content. 

Website owners can ensure their hard work in creating original content and refining their existing content assets pays off faster by being crawled and indexed swiftly.

From the perspective of search engines, IndexNow enables more efficient use of computing resources by reducing the need for exploratory crawls. A byproduct of this improved efficiency is also a greener Internet with lower environmental impact. 

Hopefully, more search engines and platforms will follow the lead of Bing, Yandex, and Duda in pushing for an Internet that works better. 



Did you find this article interesting?


Thanks for the feedback!
By Shawn Davis April 16, 2026
Website builder analysed 69M AI crawler visits across over 850,000 websites in February 2026 to determine key trends and characteristics that increase local AEO
By Shawn Davis April 1, 2026
Core Web Vitals aren't new, Google introduced them in 2020 and made them a ranking factor in 2021. But the questions keep coming, because the metrics keep changing and the stakes keep rising. Reddit's SEO communities were still debating their impact as recently as January 2026, and for good reason: most agencies still don't have a clear, repeatable way to measure, diagnose, and fix them for clients. This guide cuts through the noise. Here's what Core Web Vitals actually measure, what good scores look like today, and how to improve them—without needing a dedicated performance engineer on every project. What Core Web Vitals measure Google evaluates three user experience signals to determine whether a page feels fast, stable, and responsive: Largest Contentful Paint (LCP) measures how long it takes for the biggest visible element on a page — usually a hero image or headline — to load. Google considers anything under 2.5 seconds good. Above 4 seconds is poor. Interaction to Next Paint (INP) replaced First Input Delay (FID) in March 2024. Where FID measures the delay before a user's first click is registered, INP tracks the full responsiveness of every interaction across the page session. A good INP score is under 200 milliseconds. Cumulative Layout Shift (CLS) measures visual stability — how much page elements unexpectedly move while content loads. A score below 0.1 is good. Higher scores signal that images, ads, or embeds are pushing content around after load, which frustrates users and tanks conversions. These three metrics are a subset of Google's broader Page Experience signals, which also include HTTPS, safe browsing, and mobile usability. Core Web Vitals are the ones you can most directly control and improve. Why your clients' scores may still be poor Core Web Vitals scores vary dramatically by platform, hosting, and how a site was built. Some of the most common culprits agencies encounter: Heavy above-the-fold content . A homepage with an autoplay video, a full-width image slider, and a chat widget loading simultaneously will fail LCP every time. The browser has to resolve all of those resources before it can paint the largest element. Unstable image dimensions . When an image loads without defined width and height attributes, the browser doesn't reserve space for it. It renders the surrounding text, then jumps it down when the image appears. That jump is CLS. Third-party scripts blocking the main thread . Analytics pixels, ad tags, and live chat tools run on the browser's main thread. When they stack up, every click and tap has to wait in line — driving INP scores up. A single slow third-party script can push an otherwise clean site into "needs improvement" territory. Too many web fonts . Each font family and weight is a separate network request. A page loading four font files before rendering any text will fail LCP, especially on mobile connections. Unoptimized images . JPEGs and PNGs served at full resolution, without compression or modern formats like WebP or AVIF, add unnecessary weight to every page load. How to measure them accurately There are two types of Core Web Vitals data you should be looking at for every client: Lab data comes from tools like Google PageSpeed Insights, Lighthouse, and WebPageTest. It simulates page loads in controlled conditions. Lab data is useful for diagnosing specific issues and testing fixes before you deploy them. Field data (also called Real User Monitoring, or RUM) comes from actual users visiting the site. Google collects this through the Chrome User Experience Report (CrUX) and surfaces it in Search Console and PageSpeed Insights. Field data is what Google actually uses as a ranking signal — and it often looks worse than lab data because it reflects real-world device and connection variability. If your client's site has enough traffic, you'll see field data in Search Console under Core Web Vitals. This is your baseline. Lab data helps you understand why the scores are what they are. For clients with low traffic who don't have enough field data to appear in CrUX, you'll be working primarily with lab scores. Set that expectation early so clients understand that improvements may not immediately show up in Search Console. Practical fixes that move the needle Fix LCP: get the hero image loading first The single most effective LCP improvement is adding fetchpriority="high" to the hero image tag. This tells the browser to prioritize that resource over everything else. If you're using a background CSS image for the hero, switch it to anelement — background images aren't discoverable by the browser's preload scanner. Also check whether your hosting serves images through a CDN with caching. Edge delivery dramatically reduces the time-to-first-byte, which feeds directly into LCP. Fix CLS: define dimensions for every media element Every image, video, and ad slot on the page needs explicit width and height attributes in the HTML. If you're using responsive CSS, you can still define the aspect ratio with aspect-ratio in CSS while leaving the actual size fluid. The key is giving the browser enough information to reserve space before the asset loads. Avoid inserting content above existing content after page load. This is common with cookie banners, sticky headers that change height, and dynamically loaded ad units. If you need to show these, anchor them to fixed positions so they don't push content around. Fix INP: reduce what's competing for the main thread Audit third-party scripts and defer or remove anything that isn't essential. Tools like WebPageTest's waterfall view or Chrome DevTools Performance panel show you exactly which scripts are blocking the main thread and for how long. Load chat widgets, analytics, and ad tags asynchronously and after the page's critical path has resolved. For most clients, moving non-essential scripts to load after the DOMContentLoaded event is a meaningful INP improvement with no visible impact on the user experience. For websites with heavy JavaScript — particularly those built on frameworks with large client-side bundles — consider breaking up long tasks into smaller chunks using the browser's Scheduler API or simply splitting components so the main thread isn't locked for more than 50 milliseconds at a stretch. What platforms handle automatically One of the practical advantages of building on a platform optimized for performance is that many of these fixes are applied by default. Duda, for example, automatically serves WebP images, lazy loads below-the-fold content, minifies CSS, and uses efficient cache policies for static assets. As of May 2025, 82% of sites built on Duda pass all three Core Web Vitals metrics — the highest recorded pass rate among major website platforms. That baseline matters when you're managing dozens or hundreds of client sites. It means you're starting each project close to or at a passing score, rather than diagnosing and patching a broken foundation. How much do Core Web Vitals actually affect rankings? Honestly, they're a tiebreaker — not a primary signal. Google has been clear that content quality and relevance still dominate ranking decisions. A well-optimized site with thin, irrelevant content won't outrank a content-rich competitor just because its CLS is 0.05. What Core Web Vitals do affect is the user experience that supports those rankings. Pages with poor LCP scores have measurably higher bounce rates. Sites with high CLS lose users mid-session. Those behavioral signals — time on page, return visits, conversions — are things search engines can observe and incorporate. The practical argument for fixing Core Web Vitals isn't just "because Google said so." It's that faster, more stable pages convert better. Every second of LCP improvement can reduce bounce rates by 15–20% depending on the industry and device mix. For client sites that monetize through leads or eCommerce, that's a revenue argument, not just an SEO argument. A repeatable process for agencies Audit every new site before launch. Run PageSpeed Insights and record LCP, INP, and CLS scores for both mobile and desktop. Flag anything in the "needs improvement" or "poor" range before the client sees the live site. Check Search Console monthly for existing clients. The Core Web Vitals report surfaces issues as they appear in field data. Catching a regression early — before it compounds — is significantly easier than explaining a traffic drop after the fact. Document what you've improved. Clients rarely see Core Web Vitals scores on their own. A monthly one-page performance summary showing before/after scores builds credibility and makes your technical work visible. Prioritize mobile. Google uses mobile-first indexing, and field data shows that mobile CWV scores are almost always worse than desktop. If you only have time to optimize one version, do mobile first. Core Web Vitals aren't a one-time fix. Platforms change, new scripts get added, campaigns bring in new widgets. Build the audit into your workflow and treat it like any other ongoing deliverable, and you'll stay ahead of the issues before they affect your clients' rankings. Duda's platform is built with Core Web Vitals performance in mind. Explore how it handles image optimization, script management, and site speed automatically — so your team spends less time debugging and more time building.
By Ilana Brudo March 31, 2026
Vertical SaaS must transition from tools to an AI-powered Vertical Operating System (vOS). Learn to leverage context, end tech sprawl, and maximize retention.
Show More

Latest posts