Has anyone experienced the issue of Googlebot being blocked by robots.txt in Webflow, even though access has been allowed in the file?

TL;DR
  • Ensure robots.txt in Webflow allows Googlebot, verify site is published, and confirm DNS settings point to Webflow's IPs.
  • Check firewall permissions and use Google Search Console tools to test robots.txt and Googlebot access, adjusting settings as needed.

Issues with Googlebot being blocked by robots.txt in Webflow can be frustrating, especially if the file appears to allow access. Here’s how to address this issue:

​

1. Check Robots.txt Settings in Webflow

  • Go to Project Settings in your Webflow site.
  • Click on the SEO tab and locate the Robots.txt section.
  • Ensure your robots.txt file includes directives to allow Googlebot access, such as User-agent: Googlebot Allow: /.

​

2. Review Site's Hosting and DNS

  • Confirm the site is published and live on the Webflow servers.
  • Verify your DNS settings are correct and pointing to Webflow’s IP addresses (a) 75.2.70.75, (b) 99.83.190.102.

​

3. Check for Permissions or Firewall Issues

  • Review any external firewall settings that might restrict Googlebot.
  • Ensure there are no server-side security settings blocking Googlebot’s IP addresses.

​

4. Test Robots.txt Retrieval

  • Use the robots.txt Tester tool available in Google Search Console.
  • Enter your site's URL/robots.txt to ensure Googlebot can access and read the file correctly.

​

5. Verify with Fetch as Google

  • Use the URL Inspection Tool in Google Search Console.
  • Request a live test to see if Googlebot encounters any denied access issues.

​

6. Check for Recent Changes

  • Review any recent changes made in your Webflow settings or DNS configurations that might have affected access.
  • Roll back any changes abruptly preceding the issue to isolate the cause.

​

Summary

Ensure that your robots.txt settings in Webflow explicitly allow Googlebot, check DNS settings and firewall permissions, and verify with Google Search Console tools. Adjustments to configuration or recent changes might hold the key to resolving this blocking issue.

Webflow Resources
Webflow templates
Webflow inspiration
Rate this answer

Other Webflow Questions

Explore Logic tutorials
How can I resolve the 500 (Internal Server Error) I am encountering while trying to upload a file into a form using Webflow with uploadcare integration?
What is the beta release of Webflow Designer Canvas and what does it represent?
Can Webflow include bootstrap icons or similar like Font Awesome?
Will Webflow still support the use of the <code>&lt;meta name="keywords" /&gt;</code> tag for search engine rankings?
How can I add a CSS function recommended by Google Pagespeed Insights in Webflow using custom code?
How can I make a looping animation with a background video component in Webflow that plays on mobile devices?
Is it possible to exclude the blog page and navigation button from the subdomain and prevent them from being translated into English using Webflow?
Can you access Webflow's code to change the value of a parent div with a custom attribute in Udesly's audio player for Webflow based on a CMS field?
How can I create an interactive map on Webflow using SVG code that exceeds the HTML embed character limit?
Can anyone with experience using fullpage.js in Webflow help me troubleshoot issues with left/right navigation arrows, triggering animations, and creating a "current" state for slide links? Thank you!
How can I create a navigation with a sub-menu option in Webflow's navbar component?
How can I change the language of the Google Maps embed from English to Arabic in Webflow?
Can Webflow run a script to detect iOS and only display a 'download now' button on iPads for a site designed for an iPad app?
Is there a more efficient method in Webflow for adding keywords and other Meta tags to my header without duplication? Thank you for any assistance. Mike
How can I stop a Webflow Vimeo video from playing with a button other than the play button, close the video window when the close window button is pressed, and autoplay the video when the window opens?
Can I capture a screenshot of a Webflow page on click and convert it to a downloadable PDF?
How can I alternate the layout of the image and heading for each collection item in a two-column format on Webflow?
How can I execute JavaScript code when clicking a specific button with a given ID in a Webflow project?
How can I disable the Webflow success or failure state for a sign-up form and display a custom thank you page using jQuery and the Webflow form submit state?
How can I create a vertical scrolling menu with Webflow, similar to the one on Apple's website, that switches to horizontal scrolling when the menu doesn't fit on one screen?
How can I change the language of the Google Maps embed from English to Arabic in Webflow?
How can I reduce the vertical space between lines within a bullet point in Webflow? Can I replace the bullet points with icons on the "Services" page?
How can the text color be changed for custom input fields on Webflow?
How can I add conditional visibility in Webflow to prevent a div from appearing on a published page if a CMS field is empty?
How can I stop a YouTube video from playing in the background in audio mode when I close a modal in Webflow?
How can I connect and use my JavaScript and CSS files for special functions and styles in Webflow?
How can I fix the "Ensure text remains visible during webfont load" warning in Webflow?
How can I implement a ThreeJS script as a background for my Webflow project using custom code?
How can I send data from a Webflow Form to ActiveCampaign without using Zapier? I have set the form to POST and input the form's action URL, similar to Mailchimp but it redirects me to the admin area of ActiveCampaign without sending the data. Has anyone had success with this method?
Can Webflow be used to create a "blob" elements effect in the header of a website using custom code or JavaScript?