SEO Infrastructure for a New Cross-Border Website: robots.txt, Sitemaps, Canonical Tags, and the Submission Workflow

Many B2B export websites go live with pages that look fine to buyers, but confusing to search engines. The homepage loads, the inquiry form sends mail, and the product page looks complete, yet Google or Baidu ends up crawling parameter URLs, preview paths, duplicated filters, or the wrong language version. That is usually not a content problem. It is an infrastructure problem.

If your team has already drafted the homepage, category page, product page, about page, and contact page, finish the basic layout first and then tighten the SEO layer. This launch checklist for page structure can help: core page structure checklist. This article focuses on what to do next: robots.txt, sitemaps, canonical tags, and a practical submission workflow that supports indexing without making promises you cannot control.

When to use this workflow

Use this workflow when:

  • your site already has a live domain with HTTPS;
  • your main page types are ready for visitors and inquiry capture;
  • you want Google and Baidu to discover the right URLs instead of test paths or search-result pages;
  • your team is small, so you need rules that are easy to review and maintain.

This is especially useful for the first week after launch, or right before you start publishing multilingual product, FAQ, or solution content.

What to prepare before touching any file

Prepare four lists first:

  1. Your single preferred site host, such as https://example.com, including whether you standardize on www or non-www.
  2. Every page that should be indexable: homepage, key category pages, core product pages, about, contact, privacy, terms, and published content.
  3. Every page that should not compete in search: admin paths, login, internal search, preview URLs, filtered duplicates, thank-you pages, and temporary campaign pages.
  4. Access to Search Console, Baidu submission tools, and the deployment path that generates your sitemap.

This preparation prevents a common launch mistake: teams carefully polish the product page, but forget that search pages, inquiry success pages, or UTM parameter URLs are still visible to crawlers and may become the first discovered version of the content.

Steps 1: define the final URLs that deserve indexing

Before writing configuration, decide what the final URL is for each page type.

  • The homepage should have one canonical URL.
  • Each category page should represent a real buyer-facing topic, not a filtered duplicate.
  • Each product page should keep one durable public URL, not multiple preview or printable variants.
  • Each article, FAQ page, or solution page should point to its final public version.
  • Trust pages such as about, contact, privacy, and terms should stay accessible and stable if they are part of the public site.

If you run Chinese and English versions, do not canonicalize the Chinese page to the English page or the other way around. Different languages are different pages. The goal is to remove duplicates within the same language, not to collapse every localized page into one URL.

Steps 2: use robots.txt for crawl boundaries, not as a hiding tool

robots.txt is for crawl management. It is not a reliable way to hide pages from Google, and it is not a replacement for canonical or noindex. A lean export site usually needs to block only obvious non-public areas such as admin, login, register, and internal search. Do not block core product assets, public category pages, or files needed to render the page correctly.

User-agent: *
Allow: /
Disallow: /admin
Disallow: /login
Disallow: /register
Disallow: /search

User-agent: Baiduspider
Allow: /
Disallow: /admin
Disallow: /login
Disallow: /register
Disallow: /search

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/baidu_urls.txt

Two review rules matter here:

  • never block a page that you still expect search engines to understand, validate, and index properly;
  • never let staging or preview environments share the same open crawl rules as production.

If a page should be inaccessible, protect it. If it should stay public but out of search, handle that on the page with noindex.

Steps 3: generate a sitemap that contains only canonical URLs

A sitemap is not a database export. It is a curated list of URLs that you want search engines to treat as legitimate public destinations. Every URL in the file should return 200, be accessible, and represent a page that passed human review.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-07-07</lastmod>
  </url>
  <url>
    <loc>https://example.com/category/stainless-connectors</loc>
    <lastmod>2026-07-07</lastmod>
  </url>
  <url>
    <loc>https://example.com/product/stainless-hose-clamp</loc>
    <lastmod>2026-07-06</lastmod>
  </url>
  <url>
    <loc>https://example.com/article/how-to-choose-hose-clamps-en</loc>
    <lastmod>2026-07-07</lastmod>
  </url>
</urlset>

Keep these URLs out of the sitemap:

  • redirects and broken pages;
  • noindex pages;
  • parameter URLs such as ?utm=, ?sort=, or duplicate filter combinations;
  • internal search pages, preview pages, or draft product pages;
  • public pages that are still waiting for human review.

Only update lastmod when a meaningful change happened on the page: main content, structured data, buyer-facing details, or important internal links. If you update it for every trivial edit, the file becomes noisy and less useful for review.

Steps 4: add self-referencing canonical tags to every public page

For most export websites, the default answer is simple: every public product page, category page, article page, FAQ page, and solution page should point to itself as canonical.

<link rel="canonical" href="https://example.com/product/stainless-hose-clamp">
<meta name="robots" content="index,follow">

This matters when the same page can be opened through sorting parameters, campaign tags, multiple navigation paths, or temporary language switches. Your canonical tag should always point back to the final URL you want buyers and search engines to keep.

For pages that users may visit but you do not want in search, such as internal search results, inquiry thank-you pages, or short-lived campaign pages, page-level noindex is usually the cleaner control. Also avoid a lazy pagination shortcut: do not canonicalize every page 2, 3, and 4 back to page 1 if those pages are legitimate list pages with unique items.

Steps 5: submission is a notification process, not a ranking trick

Once the files and tags are ready, handle submission in a controlled order:

  1. Open robots.txt and sitemap.xml in the browser and confirm 200 responses.
  2. Submit the root sitemap in Search Console.
  3. Use URL Inspection to validate the homepage, one category page, one product page, and one new article page.
  4. Send newly published or meaningfully updated canonical URLs to your Baidu submission flow.
  5. Record the date, the submitted URLs, and the returned status so the next review is evidence-based.

For ordinary blog posts and product pages, do not treat the Google Indexing API as a generic fast lane. Its supported use cases are limited, and a normal export content site should rely on crawlable links, clean sitemaps, proper canonical signals, and occasional spot checks through URL Inspection.

Human review points before and after launch

Do a short human review before publishing new URLs:

  • confirm that each public page has exactly one stable canonical target;
  • confirm that robots rules do not block key product or category resources;
  • confirm that the sitemap contains only approved public pages;
  • confirm that language versions keep their own canonical URLs;
  • confirm that contact and inquiry-related success pages are not competing in search;
  • confirm that navigation and internal links can actually lead crawlers from the homepage to important commercial pages.

This human review is what keeps a small team from amplifying mistakes through automation.

Pitfall boundaries to keep in mind

The most common pitfall patterns are:

  1. using robots.txt as if it were a privacy wall;
  2. exporting every record into the sitemap, including duplicates, drafts, and redirects;
  3. sending conflicting signals between internal links, canonical tags, and the sitemap;
  4. allowing search-result pages or inquiry confirmation pages to compete with commercial pages;
  5. submitting URLs repeatedly without checking whether the underlying page is actually valid.

If you already have older articles or older product pages, pair this setup with a maintenance pass later. This guide on old-content maintenance is useful for that second review cycle: old content maintenance workflow.

How to validate results and review weekly

Track three result windows:

  • Within 48 hours: confirm that robots.txt, sitemap.xml, and canonical markup are accessible and syntactically clean.
  • Within 7 days: confirm that Search Console can read the sitemap and that important page types are being discovered.
  • Within 2 to 4 weeks: compare indexed URLs, search queries, and the reduction of bad parameter pages or duplicate entries.

Use this short weekly review:

  1. Which newly published page URLs are the canonical ones?
  2. Which important pages are still missing from the sitemap or internal link graph?
  3. Which pages should stay public for users but leave search through noindex?
  4. Which inquiry-related pages need rule changes before they create noise?

Summary

For a new cross-border website, robots.txt, sitemaps, canonical tags, and submission are not isolated technical chores. They are the operating rules that tell search engines which page is real, which page is secondary, and which page should stay out of the index. Once those rules line up, later SEO work, content expansion, and inquiry conversion improvements become easier to measure and safer to scale.