If you've never heard of `sitemap.xml` or `robots.txt`, your SEO is flying blind. These two files sit at the root of your domain and control how Google's crawler finds, indexes, and prioritizes your pages. Forget either one, and Google might miss entire sections of your site. Mess them up, and you'll silently block the pages you're trying to rank. They're foundational. And most sites get them spectacularly wrong.
Velocity X ships a complete pattern. Astro's `@astrojs/sitemap` auto-generates an XML sitemap. A hand-written `robots.txt` controls crawl access. Together, they tell Google: "Here are all my pages. Here's what you can crawl. Here's what you shouldn't index." No surprises. No missing routes. No accidental blocks.
Sitemap.xml: The Crawler's Treasure Map
A sitemap is a machine-readable list of every public page on your site. It lives at `/sitemap.xml` and contains URLs, last-modified dates, change frequency, and priority hints for Google's crawler. Google doesn't require a sitemap—it can discover pages by following links—but a sitemap dramatically speeds up crawl discovery. Instead of following breadcrumbs from your homepage, Google gets a complete manifest. New pages get indexed faster. Large sites with deep hierarchies get indexed completely instead of partially.
Most static site generators don't include sitemap support by default. Astro does—via `@astrojs/sitemap`. It generates your sitemap.xml at build time from your actual routes. If you add a new page and deploy, the sitemap updates automatically. If you have 500 blog posts, they all appear. Dynamic routes work. It's effortless.
Robots.txt: Your Crawler Control Panel
If sitemap.xml says "here are my pages," robots.txt says "here's what you can crawl." It's a simple text file that tells crawlers (Google, Bing, etc.) which directories and files they're allowed to request. You can allow everything, block specific paths, or set crawl delays. Most critically, you can tell crawlers to respect your budget—don't waste time on admin pages, dashboards, or login flows that aren't meant for public indexing.
A typical robots.txt looks like this:
{`User-agent: *
Allow: /
Disallow: /admin/
Disallow: /dashboard/
Disallow: /*.json$
Sitemap: https://yourdomain.com/sitemap.xml`}That says: "All crawlers can access everything except `/admin/`, `/dashboard/`, and `.json` files. And by the way, here's my sitemap." Simple. Explicit. Works.
Four Traps Most Sites Fall Into
1. Forgetting Noindex on Dashboard Pages
If you have a logged-in dashboard (orders, account settings, analytics), those pages are almost always in your sitemap—because they're real routes in your codebase. Google's crawler can reach them if they're not blocked. The solution: add `noindex` meta tags to dashboard templates. That tells crawlers "crawl if you want, but don't index this." A hand-written robots.txt alone won't catch this—you need noindex in the page's `
`.2. Accidentally Blocking Essential Paths in Robots.txt
A typo in robots.txt—`Disallow: /` instead of `Disallow: /admin/`—and your entire site becomes uncrawlable. Google sees the blanket block and stops indexing everything. Even worse: disallowing `/api/` or `/assets/` can break crawlers' ability to render your pages if they rely on those endpoints. Always test robots.txt changes in Google Search Console before deploying.
3. Missing Dynamic Routes in Your Sitemap
If you're using Astro's dynamic routes—like `[category]/[item].astro`—and you're loading those routes from JSON or a database, your sitemap won't know about them unless you tell it. Astro's `@astrojs/sitemap` needs explicit route generation to include dynamic pages. Without that, 90% of your content might be invisible to the sitemap—and slower to get indexed by Google.
4. Not Using Sitemap Index for Large Sites
Google limits each sitemap file to 50,000 URLs. If you have more than that (unlikely for most marketing sites, but standard for ecommerce), you need a sitemap index—a file that points to multiple sitemaps. Astro handles this automatically if you hit the limit, but knowing it exists means you won't panic if you see `/sitemap-0.xml`, `/sitemap-1.xml`, etc.
The Astro Pattern That Works
Astro's `@astrojs/sitemap` integration ships in `astro.config.mjs`. It generates your sitemap.xml at build time. If you're using dynamic routes, pass a `canonicalURL` and ensure your collection loaders are running at build time—otherwise Astro won't know what routes to generate. For a stock marketing site (static blog, static service pages, static pricing), the default config works out of the box.
Your robots.txt lives at `public/robots.txt` and is copied to your root on deployment. Keep it minimal: allow everything except dashboards, admin panels, and internal endpoints. Reference your sitemap. Done.
Six Questions (and Answers) Everyone Gets Wrong
Q: Do I need a sitemap if I have a robots.txt?
A: No. They're independent. A sitemap is a manifest. Robots.txt is access control. You want both, but they do different things.
Q: Will Google crawl my site if I don't submit the sitemap?
A: Yes, if your site is discoverable (public links, social mentions, manual submission). But it crawls much slower and might miss new pages for weeks. Submit your sitemap in Search Console to speed up discovery.
Q: What's the difference between noindex and Disallow in robots.txt?
A: Disallow prevents crawling. Noindex prevents indexing. If a page is Disallowed, crawlers never fetch it, so they can't read the noindex tag. Use robots.txt for admin/dashboard. Use noindex for public pages you don't want ranked (like internal docs).
Q: Should I block `/admin/` in robots.txt or use a different domain?
A: Both are fine. Robots.txt is lighter if you control the domain. A subdomain (like `admin.yourdomain.com`) is clearer if you need different security rules.
Q: Can robots.txt prevent Google from indexing a page?
A: No. Robots.txt can only prevent crawling. If another site links to your page, Google might index it even if robots.txt blocks it. Use noindex if you need guaranteed non-indexing.
Q: How often should I update my sitemap?
A: Automatically at every deploy if you're using Astro. Manual updates aren't needed—the build generates a fresh sitemap each time.
The Verdict
Sitemap.xml and robots.txt are unglamorous, but they're foundational crawl infrastructure. A sitemap tells Google where you are. Robots.txt tells it what to respect. Together, they cut the time from "page published" to "page ranked" by weeks. Most sites leave them on autopilot or ignore them entirely—which is why most sites' new content moves slowly. Get them right once, and you're done. Your sitemap regenerates at build time. Your robots.txt stays the same until you add a new section worth blocking. No ongoing work. Just faster crawl discovery and better indexing. For more SEO patterns, see the guide on structured data and rich snippets, or let's discuss your full crawl strategy in a site audit.