How can I correctly set up the robots.txt file in Webflow to ensure certain pages are not ignored by search engines?

TL;DR
  • Avoid blocking important pages in robots.txt and ensure they don’t have “exclude from indexing” enabled in SEO settings.  
  • After adjustments, publish the site and verify behavior using Google Search Console tools.

To make sure specific pages are not unintentionally blocked by search engines in Webflow, you need to properly configure your robots.txt and page-level SEO settings.

​

1. Understand robots.txt Purpose and Scope

​

  • The robots.txt file tells search engine crawlers which parts of your site they should or should not access.
  • It applies globally to all bots unless user-agent rules specify otherwise.
  • However, if a page has both a noindex meta tag and is blocked in robots.txt, search engines can't see the noindex and may still index the page based on external links (but without a content preview).

​

2. Access the robots.txt File in Webflow

​

  • Go to Project Settings for your site.
  • Click on the SEO tab.
  • Scroll to the robots.txt section to edit.

​

3. Format the robots.txt to Allow Important Pages

​

  • Make sure you're not using a "Disallow: /" directive, which blocks the entire site.
  • Only block paths you want to hide, such as staging content or utility pages.
  • Example:
  • User-agent: *
  • Disallow: /internal-page/
  • Allow: /services/ (optional, use only if you’ve previously disallowed a broader directory)

​

4. Use Page-Specific Settings for Noindex/Index

​

  • Go to the page in the Webflow Designer.
  • Open the Page Settings (gear icon).
  • Under SEO settings, make sure “Exclude this page from search engine indexing” is unchecked for pages you want indexed.
  • Check this option only for pages you want to prevent from being indexed.

​

5. Publish to Check Changes

​

  • Publish your site after making changes to both robots.txt and page-level SEO settings.
  • Use tools like Google Search Console’s URL inspection or the robots.txt Tester to confirm behavior.

​

Summary

​

To ensure pages are not ignored by search engines, avoid blocking them in robots.txt and make sure their SEO settings don’t have "exclude from indexing" enabled. Confirm and test changes using Google tools after publishing.

Webflow Resources
Webflow templates
Webflow inspiration
Rate this answer

Other Webflow Questions

Explore Logic tutorials
How can I resolve scrolling issues with a dropdown list in Webflow, specifically with a 13" Mac screen that cannot scroll to the bottom of the list?
What is the most cost-effective and user-friendly method to upload multiple images in bulk using a CSV file and CMS in Webflow? With over 700 items in the collection, I need an efficient solution for attaching jpgs, pngs, and glb files. I've come across some older discussions on this topic, so I'm curious if there have been any recent developments or improvements.
How can I delete a hover state's styles in Webflow that I added by mistake?
How can I set up subscriptions and release a steady amount on a Webflow website for a subscription box business?
How can I convert MP4 files to LottieFiles using a tool on Webflow?
Is there a reliable custom code to remove scrollbars on all browsers for elements in a Webflow website?
Is it possible to add a native cookie consent that appears only once on the first page load in Webflow? Can this be done using a modal wrapper in Webflow without any third party tools or integrations? If not, are there any suggestions or advice on adding a cookie consent to comply with European regulations?
What is the process for uploading and hosting CSS and JS files on Github for file management and updates in Webflow, if the desired plugin is not available on Github?
What could be causing the issue with the slider button on my Webflow website, where it is not clickable when the slider or the button itself is hovered?
What is the process for adding HTML and CSS script to a Webflow page using custom code settings?
What is a reliable way to randomize the hero header content in Webflow's CMS collection so that it displays a different collection each time the page is reloaded, and is easily updatable by the client in the future?
How can I make my Webflow navbar responsive when switching to mobile landscape breakpoints?
How can I change the language of the Google Maps embed from English to Arabic in Webflow?
What is the best way to create a side navigation bar in Webflow that avoids overlapping with the main text content when resizing the browser?
Is it possible to hide or remove the slider arrows and navigator in Webflow without deleting the entire slider?
Can I capture a screenshot of a Webflow page on click and convert it to a downloadable PDF?
How can I alternate the layout of the image and heading for each collection item in a two-column format on Webflow?
How can I execute JavaScript code when clicking a specific button with a given ID in a Webflow project?
How can I disable the Webflow success or failure state for a sign-up form and display a custom thank you page using jQuery and the Webflow form submit state?
How can I create a vertical scrolling menu with Webflow, similar to the one on Apple's website, that switches to horizontal scrolling when the menu doesn't fit on one screen?
How can I change the language of the Google Maps embed from English to Arabic in Webflow?
How can I reduce the vertical space between lines within a bullet point in Webflow? Can I replace the bullet points with icons on the "Services" page?
How can the text color be changed for custom input fields on Webflow?
How can I add conditional visibility in Webflow to prevent a div from appearing on a published page if a CMS field is empty?
How can I stop a YouTube video from playing in the background in audio mode when I close a modal in Webflow?
How can I connect and use my JavaScript and CSS files for special functions and styles in Webflow?
How can I fix the "Ensure text remains visible during webfont load" warning in Webflow?
How can I implement a ThreeJS script as a background for my Webflow project using custom code?
How can I send data from a Webflow Form to ActiveCampaign without using Zapier? I have set the form to POST and input the form's action URL, similar to Mailchimp but it redirects me to the admin area of ActiveCampaign without sending the data. Has anyone had success with this method?
Can Webflow be used to create a "blob" elements effect in the header of a website using custom code or JavaScript?