Is anyone experiencing issues with their Robots.txt file in Webflow? I'm having trouble uploading the sitemap to Google and testing the site for rich results on Google due to the site being blocked by the robots.txt file. I've tried editing the robots.txt file and deleting text from the SEO tab, but neither solution has worked. Even manually indexing each URL on Google's sitemap app doesn't resolve the issue. Any suggestions?
TL;DR
- Review and clear restrictive rules from Webflow's Project Settings → SEO robots.txt field, republish the site, and verify the live robots.txt is clean.
- Submit sitemap in Google Search Console, test live URLs for indexing, and ensure you're using your custom domain, not the Webflow.io subdomain.
You’re experiencing issues with Google crawling and indexing your Webflow site due to improper rules in your robots.txt file, even after attempts to edit or delete the file content.
1. Check Current Robots.txt Rules in Webflow
- Go to Project Settings → SEO and review the Robots.txt text area.
- Make sure it doesn't include restrictive rules, like:
User-agent: *
Disallow: /
- If those are present, Google will block your entire site from indexing.
2. Clear the Robots.txt File Completely
- If you’ve attempted to clear it but it’s still showing up, Webflow may be caching it.
- Delete all text from the Robots.txt field, save changes, and republish your site.
- Wait a few minutes, then visit:
https://yoursite.com/robots.txtand verify it’s empty or contains only safe directives (e.g., a sitemap URL).
3. Confirm Sitemap Submission and Access
- Go to Google Search Console → Sitemaps, and submit the sitemap:
- Usually it's
https://yoursite.com/sitemap.xml - After submission, check for errors. If Google reports the sitemap is blocked by robots.txt, then the robots file is still not cleaned.
4. Test with Google’s Live URL Test
- Inside Google Search Console → URL Inspection, enter a live URL and click on “Test Live URL”.
- If the result says “Blocked by robots.txt”, the blocking rule is active.
- If it says “Crawl allowed? No: Disallowed by robots.txt”, you've confirmed the issue.
5. Bypass Robots Blocking Temporarily with Noindex/Delete
- As a temporary fix, try:
- Publishing your site with an empty robots.txt, as mentioned earlier.
- Removing any
<meta name="robots" content="noindex">tags in your page settings.
6. Check Webflow’s Staging Domains
- Sometimes, users mistakenly test on a Webflow.io subdomain, which may be blocked by Webflow for SEO to avoid duplicate indexing.
- Be sure all indexing attempts are made on your connected custom domain (e.g.,
https://yourdomain.com), notyourproject.webflow.io.
Summary
Your site's robots.txt is likely still blocking Google due to lingering restrictive rules or caching. Clear the field in Project Settings → SEO, republish your site, and test the live file. Then re-submit your sitemap in Google Search Console and ensure you’re using the custom domain, not a Webflow.io subdomain.
