Why is my Webflow site blocked for bots and why is the sitemap verification in Google Search Console not working?

TL;DR

Ensure site-wide and individual page SEO settings don’t block indexing in Webflow.
Verify robots.txt doesn’t disallow crawling and correctly includes the sitemap URL.
Check that your sitemap is accessible in the browser and matches your domain.
Confirm your domain is properly connected and DNS settings are correct.
Use Google Search Console to test URLs and resubmit the sitemap if needed.

Your Webflow site may be blocked for bots if it's unintentionally set to block indexing or if there's an issue with your robots.txt or sitemap settings. This can directly affect Google Search Console's ability to verify your sitemap.

1. Check Page and Site-Wide SEO Settings

Go to Project Settings > SEO tab and make sure “Disable Webflow Subdomain Indexing” is not selected if you're using the Webflow subdomain.
Check the individual page settings in the Webflow Designer:
Open the page settings (gear icon on the page).
Ensure “SEO Settings > Hide this page from search engines” is not checked.

2. Review robots.txt Settings

In Project Settings > SEO tab, find the robots.txt field.
Ensure it doesn't contain Disallow: / which would block all bots from crawling your site.
Typical default should allow crawling:

```

User-agent: *

Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

```

Update the sitemap URL to match your actual domain correctly (use https and include the www if your domain uses it).

3. Confirm Sitemap Availability

Visit https://yourdomain.com/sitemap.xml in your browser.
If it returns a 404 or any error, then the sitemap isn't published or the domain is misconfigured.

4. Make Sure Your Domain Is Properly Connected

In Project Settings > Hosting, confirm your domain is connected and published without errors.
DNS A records should point to (a) 75.2.70.75, (b) 99.83.190.102

5. Test and Resubmit Sitemap in Google Search Console

In Google Search Console, use the URL Inspection Tool to test a live URL from your site to confirm that it is crawlable.
Go to the Sitemaps section and resubmit sitemap.xml.
If you still receive errors, check if Google reports a crawl error (e.g., blocked by robots.txt or 403 Forbidden).

Summary

Your site might be blocked for bots due to a misconfigured robots.txt, disabled indexing settings, or a broken sitemap link. Make sure page-level and project-level SEO settings allow indexing, your robots.txt is not overly restrictive, and your sitemap is accessible and correctly linked in Google Search Console.

Webflow Resources

Webflow templates

Webflow inspiration

Rate this answer

Other Webflow Questions

Can I add external fonts to the Webflow library?

Does Webflow provide green hosting options to reduce CO2 emissions for websites?

Can European users and customers of Webflow rely on standard privacy clauses when using Webflow and hosting websites through Webflow, and can these rights also be asserted against subcontractors like AWS and Fastly? Additionally, does Webflow plan to establish an EU-based subsidiary and develop an on-premise solution or restrict hosting to EU servers for users who prefer it?

Can Webflow handle a tool that recommends items based on user answers from a questionnaire, with data filtering done through third-party tools like Zapier or Automate? Where should the data be stored?

How can I copy a perfectly designed section from one page to another in Webflow?

What could be causing my Webflow portfolio website to automatically scroll to a certain point when opening the exported site, without any anchor links attached to the URL in the browser?

How can I force a background video element to autoplay, loop, and mute on Safari and iPhones in Webflow without embedding the video?

How can you achieve a responsive background image in Webflow without sacrificing loading speed for mobile users?

How can I change animations for different devices in Webflow, specifically for mobile? Also, how can I make elements smaller in phone portrait mode while using the same tags as in landscape mode?

How can I make a slider element in Webflow scale according to the elements within it, rather than being cut off on smaller screen sizes?

How can I implement a plugin to invert the colors of my logo and menu button in Webflow based on the background color of my slider, in real time and without manual intervention?

Can I get a refund if I cancel my annual hosting subscription on Webflow? What happens if I downgrade from annual to monthly?

Can I convert regular videos into the Lottie format and have them displayed in Webflow?

How can I fix automatic redirect issues in Webflow when changing page slugs within a folder?

Can I adjust the indent of a bullet list in a rich text element in Webflow?

Can I capture a screenshot of a Webflow page on click and convert it to a downloadable PDF?

How can I alternate the layout of the image and heading for each collection item in a two-column format on Webflow?

How can I execute JavaScript code when clicking a specific button with a given ID in a Webflow project?

How can I disable the Webflow success or failure state for a sign-up form and display a custom thank you page using jQuery and the Webflow form submit state?

How can I create a vertical scrolling menu with Webflow, similar to the one on Apple's website, that switches to horizontal scrolling when the menu doesn't fit on one screen?

How can I change the language of the Google Maps embed from English to Arabic in Webflow?

How can I reduce the vertical space between lines within a bullet point in Webflow? Can I replace the bullet points with icons on the "Services" page?

How can the text color be changed for custom input fields on Webflow?

How can I add conditional visibility in Webflow to prevent a div from appearing on a published page if a CMS field is empty?

How can I stop a YouTube video from playing in the background in audio mode when I close a modal in Webflow?

How can I connect and use my JavaScript and CSS files for special functions and styles in Webflow?

How can I fix the "Ensure text remains visible during webfont load" warning in Webflow?

How can I implement a ThreeJS script as a background for my Webflow project using custom code?

How can I send data from a Webflow Form to ActiveCampaign without using Zapier? I have set the form to POST and input the form's action URL, similar to Mailchimp but it redirects me to the admin area of ActiveCampaign without sending the data. Has anyone had success with this method?

Can Webflow be used to create a "blob" elements effect in the header of a website using custom code or JavaScript?