How can I resolve the issue with deindexing a subdomain and generating a sitemap in Webflow?
TL;DR
- Disable SEO indexing in Page Settings or use a robots.txt file to block crawlers for Webflow-hosted subdomains.
- Turn off sitemap generation in Project Settings or exclude individual pages; manage deindexing and sitemap updates via Google Search Console.
Deindexing a subdomain and managing sitemap generation in Webflow requires adjusting both SEO settings and domain configurations. Here's how to handle each step properly.
1. Prevent Search Engines from Indexing a Specific Subdomain
- Webflow does not allow granular control per subdomain from within the Webflow UI. If you're hosting multiple subdomains externally (not hosted directly in Webflow), you need to handle this on the server or CMS that serves the subdomain.
- If the subdomain is served by Webflow, go to Page Settings of the specific pages and:
- Toggle “Disable SEO indexing” to Yes.
- This adds a meta noindex tag, which discourages search engine indexing.
- For more aggressive control, consider adding a robots.txt rule that blocks crawlers:
- Go to Project Settings > SEO tab
- Under robots.txt, add:
```
User-agent: *
Disallow: /
```
- Note: This blocks all crawling for the entire published site or subdomain, so only use it if the subdomain is separate from your primary domain.
2. Exclude Subdomains From Your Sitemap
- Webflow automatically generates a sitemap at yourdomain.com/sitemap.xml. However:
- Webflow’s sitemap won’t include subdomains unless they are separate Webflow projects.
- If each subdomain is its own Webflow project, each will generate its own sitemap under that subdomain.
- If you don't want a sitemap generated, turn it off by going to:
- Project Settings > SEO tab
- Set “Auto-generate sitemap” to Off
- To exclude certain pages manually:
- In the Page Settings, switch off “Include in sitemap”
3. Alert Search Engines of Changes (Optional but Recommended)
- After deindexing steps are completed, go to Google Search Console:
- Use the URL Removal tool if the subdomain or pages are still indexed.
- Submit a new sitemap (if applicable) using the Sitemaps tool under your domain property.
Summary
To deindex a Webflow-hosted subdomain, disable SEO indexing on pages or use robots.txt to block crawlers. Webflow’s sitemap is generated per project, so subdomains have their own if hosted separately. Use Search Console to request deindexing or share updated sitemaps when needed.
