Deployment & Indexability

The decisive SEO question is not “Which framework?” It is:

Does an application server receive requests after deployment?

That determines whether Commerce Engine should generate discovery responses on demand or write them as files at build time.

Decision matrix

Production shape	Use	Typical examples
Server receives requests	framework server adapter / portable request handler	Next.js server, TanStack Start, Astro server output, SvelteKit node/edge adapters
Files/CDN only	`writeCommerceSeoAssets()` prebuild	Vite SPA, Astro static output, SvelteKit `adapter-static`, fully exported Next output

Both modes produce the same discovery model. The difference is only where it is materialized.

Server mode

Use one framework handler and enable robots + sitemap:

createNextjsSeoProxy(seo, { robots: true, sitemap: true });

createTanStackStartSeoMiddleware(seo, { robots: true, sitemap: true });

createAstroSeoMiddleware(seo, { robots: true, sitemap: true });

createSvelteKitSeoHandle(seo, { robots: true, sitemap: true });

The portable createCommerceSeoRequestHandler() from @commercengine/seo/server provides the same contract for other runtimes that use standard Web Request and Response objects. A request handler always owns:

HTML/Markdown content negotiation for product/category/search routes
explicit .md mirrors
/llms.txt
/sitemap.md

It owns /robots.txt only when robots: true is enabled, and /sitemap.xml plus /sitemap/{id}.xml shards only when sitemap: true is enabled. Both server-mode options default to false so the package cannot silently shadow existing application routes.

Why `Vary: Accept` matters

When the same URL can return HTML or Markdown, the HTML response must vary on Accept; otherwise a shared cache can serve the wrong representation to the next client. Commerce Engine adapters scope this header only to URLs that actually negotiate Markdown so unrelated CDN cache keys do not fragment. Explicit .md URLs remain the primary discovery links and work even when an AI client cannot set custom Accept headers.

Static mode

A static host cannot inspect Accept, run catalog queries, or build arbitrary search results after deployment. Generate real files before the framework build: Keep site identity and public routes in the same pure commerceSeo config the application and AI tools already import. A TypeScript prebuild can reuse that module directly:

pnpm add -D tsx dotenv

scripts/generate-seo-assets.ts

import { existsSync } from "node:fs";
import { createCommerceSeo } from "@commercengine/seo";
import { writeCommerceSeoAssets } from "@commercengine/seo/build";
import { createStorefront, Environment } from "@commercengine/storefront";
import { config as loadEnvFiles } from "dotenv";
import { commerceSeo } from "../src/lib/commerce-seo.config";

// Prebuilds run before the framework loads its env files. Read local production
// files explicitly; dotenv does not overwrite variables already injected by CI.
const envFiles = [
  ".env.production.local",
  ".env.local",
  ".env.production",
  ".env",
].filter(existsSync);
if (envFiles.length) loadEnvFiles({ path: envFiles });

const storeId = process.env.CE_STORE_ID;
const apiKey = process.env.CE_API_KEY;
if (!storeId || !apiKey) {
  throw new Error("[seo] missing CE_STORE_ID / CE_API_KEY");
}

const seo = createCommerceSeo({
  ...commerceSeo,
  storefront: createStorefront({
    storeId,
    apiKey,
    environment:
      process.env.CE_ENV === "production"
        ? Environment.Production
        : Environment.Staging,
  }),
});

await writeCommerceSeoAssets(seo, { outDir: "public" });

Leave indexable unset in the shared build script. The package then uses Vercel, Netlify, Cloudflare, or generic Node deployment signals: preview builds remain non-indexable, while proven production builds become indexable automatically. Run that script before the framework build, for example:

package.json

{
  "scripts": {
    "build": "tsx scripts/generate-seo-assets.ts && vite build"
  }
}

Use the framework’s normal build command after && (astro build, next build, and so on).

Output directory

Framework/deployment	Directory
Vite / React SPA	`public/`
Astro static	`public/`
Static Next.js	`public/`
SvelteKit `adapter-static`	`static/`

Generated files are artifacts and should generally be gitignored.

Why physical files instead of catch-all routes

Framework routers prioritize concrete product routes. A /product/[slug] route can claim /product/shoe.md before a catch-all Markdown route sees it. Files placed in the publish directory bypass the application router entirely.

Static assets generated

By default the static generator can produce:

robots.txt
sitemap.xml
sitemap/{id}.xml        # only when sharding is needed
llms.txt
sitemap.md
<productBase>/*.md
<categoryBase>/*.md

The package defaults are /products and /category; the examples in these docs configure /product explicitly to match the Commerce Engine starter route convention. Optional variant mirrors can be generated when variant public URLs use distinct paths. Query-string variants usually reuse the base product .md file rather than multiplying the file count. The static generator builds sitemap.md and XML sitemap output from the assets it actually emits, so intentionally unroutable catalog entities are not advertised as dead links.

Production versus preview indexability

createCommerceSeo() resolves one indexable value and applies it consistently across head metadata, robots behavior, dynamic responses, and static output. indexable is an explicit override:

indexable: true   // force indexability
indexable: false  // force noindex
// omitted        // detect the deployment; only proven production is indexable

If it is omitted, the package detects common deployment environments:

Platform signal	Production	Preview/non-production
Vercel `VERCEL_ENV`	`production`	`preview`, `development`
Netlify `CONTEXT`	`production`	`deploy-preview`, `branch-deploy`, `dev`
Cloudflare Pages/Workers	configured production branch, default `main`/`master`	other branches
Generic Node	`NODE_ENV=production`	`NODE_ENV=development`
No reliable signal	not proven production	safe non-indexable default

Preview platforms often build with NODE_ENV=production. Do not use NODE_ENV alone to decide whether a branch deployment should be indexed.

Static noindex protection

HTML pages can carry robots metadata. Markdown and text files cannot. On a non-indexable static build, the package can emit a _headers file with a site-wide X-Robots-Tag: noindex, nofollow rule for hosts such as Netlify and Cloudflare Pages. If your host does not understand _headers, apply equivalent response headers in that host’s configuration.

Robots policy

Production robots should make public storefront pages crawlable while keeping operational/session URLs out of discovery. Use package policy unless your application has a specific reason to override it. A non-indexable deployment intentionally emits an allow-all robots policy with no sitemap while its pages/responses carry noindex, nofollow. Do not replace that with Disallow: /: a crawler blocked from fetching the page cannot read the noindex directive. Never use robots.txt as the only preview protection. robots.txt controls crawling; page/header directives control indexability.

Sitemap limits

XML sitemaps are limited to 50,000 URLs per file. Commerce Engine automatically shards larger catalogs and produces a sitemap index. You can reduce the maximum during testing:

createNextjsSeoProxy(seo, {
  robots: true,
  sitemap: true,
  maxUrlsPerSitemap: 10_000,
});

Do not silently truncate the catalog to keep one sitemap file small.

Build credentials

Static SEO generation runs before the app build and needs public catalog credentials. Local development may use .env.local; CI usually injects environment variables directly and has no local env file. A robust prebuild script:

reads the local file only when it exists
overlays/accepts CI environment variables
fails explicitly if Store ID or API key is missing
never emits an empty “valid-looking” sitemap after an authentication failure

The Commerce Engine starters implement this pattern.

Existing-file conflicts

Before adding the package, remove overlapping files/routes:

static robots.txt
static or hand-built sitemap.xml
custom llms.txt
old Markdown route handlers
duplicate JSON-LD generators

A physical public/robots.txt can shadow a perfectly correct dynamic robots.txt with no warning.

Deep-link hosting checks

SEO creates links to real pages, so production host routing must resolve them.

SPA

Configure a catch-all rewrite to index.html after static-file matching. Hard-refresh a product/category URL as part of deployment validation.

SvelteKit adapter-static on Vercel

Use:

vercel.json

{
  "cleanUrls": true
}

If /privacy-policy.html works but /privacy-policy 404s, the problem is host routing rather than the Svelte build.

Validate the deployed site

Check the deployed production origin, not only localhost:

curl -I https://acme.example/product/example
curl -H 'Accept: text/markdown' https://acme.example/product/example
curl https://acme.example/product/example.md
curl https://acme.example/robots.txt
curl https://acme.example/sitemap.xml
curl https://acme.example/llms.txt
curl https://acme.example/sitemap.md

Then verify:

canonical URLs use the real production domain
sitemap URLs hard-load successfully
preview builds return noindex protection
product/category Markdown links resolve
only one product structured-data entity is present per page
social preview tags use product/category values rather than generic site defaults

SEO package

Structured data and discovery API.

Framework guides

Framework-specific wiring.

Getting Started

Framework Guides

Reference

Decision matrix

Server mode

Why `Vary: Accept` matters

Static mode

Output directory

Why physical files instead of catch-all routes

Static assets generated

Production versus preview indexability

Static noindex protection

Robots policy

Sitemap limits

Build credentials

Existing-file conflicts

Deep-link hosting checks

SPA

SvelteKit adapter-static on Vercel

Validate the deployed site

SEO package

Framework guides

​Decision matrix

​Server mode

​Why Vary: Accept matters

​Static mode

​Output directory

​Why physical files instead of catch-all routes

​Static assets generated

​Production versus preview indexability

​Static noindex protection

​Robots policy

​Sitemap limits

​Build credentials

​Existing-file conflicts

​Deep-link hosting checks

​SPA

​SvelteKit adapter-static on Vercel

​Validate the deployed site

SEO package

Framework guides

Decision matrix

Server mode

Why `Vary: Accept` matters

Static mode

Output directory

Why physical files instead of catch-all routes

Static assets generated

Production versus preview indexability

Static noindex protection

Robots policy

Sitemap limits

Build credentials

Existing-file conflicts

Deep-link hosting checks

SPA

SvelteKit adapter-static on Vercel

Validate the deployed site