Google Search Console, sitemaps, indexing

A site nobody can find is a site that only exists for you. This is the free, mandatory step between “it’s deployed” and “it’s visible” - and it takes about 20 minutes.

1. Verify your domain

Go to search.google.com/search-console, add a Domain property (not a URL-prefix property - domain properties cover http, https, www, and non-www in one shot).

Verification is a DNS TXT record. If you’re on Cloudflare DNS (see domain, DNS, Cloudflare), add it there:

Type: TXT
Name: @
Value: google-site-verification=xxxxxxxxxxxx

Verification usually completes within minutes.

2. Generate a real sitemap

Don’t hand-write one. If you’re on Astro, @astrojs/sitemap generates it at build time from your actual routes:

// astro.config.mjs
import sitemap from '@astrojs/sitemap';

export default defineConfig({
  site: 'https://my-app.com',
  integrations: [
    sitemap({
      filter: (page) => {
        // exclude auth pages, admin pages, anything behind a login
        return !page.includes('/auth/') && !page.includes('/admin/');
      },
    }),
  ],
});

The filter callback matters as much as the sitemap itself: it’s how you keep low-value or duplicate pages out of what you ask Google to crawl. A sitemap that lists 1,000 thin pages signals low quality; a sitemap that lists your 50 strongest pages signals focus.

Build, then confirm it exists:

npm run build
curl -s https://my-app.com/sitemap-index.xml | head -20

3. Submit the sitemap

In Search Console: Sitemaps (left nav) -> enter sitemap-index.xml (or sitemap.xml) -> Submit. Google will re-crawl it periodically on its own after this - you don’t resubmit after every deploy.

4. Request indexing on your top pages

For pages you want indexed now rather than waiting for Google’s crawl schedule, use the URL Inspection tool (top search bar in Search Console): paste the URL, wait for the check, click “Request indexing.” This is worth doing for:

• Your homepage
• 5-10 pages that represent your core value (pricing/comparison pages, your best content)

Don’t do this for every page you own - it’s rate-limited and meant for a handful of priority URLs, not a bulk-indexing trick.

5. The 3 reports to check weekly

• Performance (Search results): impressions, clicks, average position, per-query and per-page. This is your only real signal of what’s actually being searched for and found - use it to double down on what works, not what you assumed would work.
• Coverage / Pages: shows what’s indexed vs excluded, and why (noindex, duplicate content, crawl error). Catch problems here before they show up as “why is nobody visiting.”
• Core Web Vitals: not urgent for a new site, but worth a glance if pages feel slow - it flags real user-experience data, not synthetic benchmarks.

6. Noindex on purpose

Not every page deserves to rank. Thin, generic, or duplicate pages hurt your site’s overall quality signal more than they help traffic. If you have hundreds of similar low-value pages (a lesson library, a tag archive, auto-generated variants), noindex the weak ones and keep only the pages with real signal - impressions, clicks, or genuine uniqueness - indexed. Pair the noindex meta tag with the sitemap filter above so you’re not asking Google to crawl pages you’ve told it not to index.

7. llms.txt, the newer half of visibility

Search Console covers classic Google indexing. Increasingly, traffic also comes through AI answers (ChatGPT, Perplexity, AI Overviews) that read your site directly rather than ranking it. See a future guide on llms.txt and GEO for that side - for now, a clean sitemap and clear, factual pages help both.