Sitemap crawl: how Fynd indexes your pages

Crawl a sitemap into Fynd

What a sitemap crawl does

A sitemap crawl gives Fynd a repeatable starting point for collecting knowledge from a public website. Instead of adding every URL by hand, you provide the sitemap URL and let Fynd inspect the pages listed there. This is useful when a shop has product pages, editorial articles, buying guides, help pages, or a blog that should be available to the assistant.

The crawl is not a copy of your entire website without rules. Fynd follows the URLs made available by the sitemap, fetches eligible pages, and classifies useful content. During this process it can detect product data exposed as Product JSON-LD. It also recognizes editorial structured data exposed as BlogPosting or Article JSON-LD. Pages identified as blog content receive page_type=blog, which allows the knowledge system and sync interface to treat them as articles rather than products.

Use a sitemap crawl when the content is already published and you want Fynd to keep using the same public source. It is especially suitable for stores that manage content in a CMS and do not want a second, manually maintained product catalogue.

Before you start

Check the sitemap in a browser before you add it to Fynd. A valid sitemap normally has a URL such as:

https://example.com/sitemap.xml
https://example.com/sitemap_index.xml
https://example.com/wp-sitemap.xml

Make sure that:

• the sitemap and its child sitemaps are publicly reachable;
• the relevant product and article URLs are included;
• pages do not require a customer login;
• your site does not block the crawler through access controls;
• canonical URLs point to the public version of each page;
• product and article data is rendered or exposed in the HTML response.

Do not use a staging sitemap that contains test products, password-protected pages, or temporary URLs. A crawl can only produce dependable answers when its source is dependable. If your production site has several languages, use the sitemap for the language you want the assistant to answer from, or configure separate sources deliberately.

Add the sitemap source

Open the Fynd knowledge area and choose the option to add website knowledge. Select a sitemap-based source, then paste the complete sitemap URL. Include https:// and avoid adding a page URL where a sitemap URL is expected.

Give the source a recognizable name. For example, use Storefront sitemap – English rather than a generic label such as Website. Clear names matter later when several sources need review.

Next, choose the crawl limit. The max_pages configuration sets the maximum number of pages Fynd may process for this source in one sync. It is a guardrail, not a promise that exactly that many pages will be imported. A crawl can end below the limit when the sitemap contains fewer eligible pages or when some URLs cannot be fetched.

Start with a limit that covers the part of the site you actually need. A catalogue with 300 products and 40 articles should not necessarily use an unrestricted site-wide sitemap with thousands of archive, filter, and duplicate URLs. Increase max_pages only after confirming that the selected URLs are useful.

How Fynd identifies content

Fynd reads the page response and looks for meaningful structured information. Product pages are recognized through Product JSON-LD. This commonly contains a product name, description, image, price, availability, brand, or offer information. Structured product data is preferable to guessing details from a page layout because it gives the crawler explicit fields to work with.

For editorial pages, Fynd recognizes BlogPosting and Article JSON-LD. When either type is found, the page is classified with:

page_type=blog

That classification is important when reviewing a sync. A product question and an article question are not always answered from the same kind of source. Blog classification also makes it easier to spot whether the crawl is finding guides and posts as expected.

Some pages contain both product references and editorial text. Fynd uses the structured data it finds; make sure your markup accurately describes the primary purpose of the page. Do not mark ordinary category or marketing pages as products merely to influence crawling. Incorrect JSON-LD leads to incorrect downstream knowledge.

Read the sync results

After the crawl finishes, open the source details or sync UI. Review the total processed pages, any skipped or failed URLs, product counts, and blog counts. The sync UI shows blog counts so you can verify that BlogPosting and Article pages were discovered rather than silently treated as generic pages.

Use the counts as a diagnostic tool:

• Zero products can mean product pages are missing from the sitemap or lack Product JSON-LD.
• Zero blog pages can mean your posts are absent from the sitemap or do not expose BlogPosting or Article JSON-LD.
• Unexpectedly high counts can indicate duplicate URLs, archives, pagination, or language variants.
• A crawl stopping at the limit means max_pages may be too low for the intended scope.
• Fetch failures often point to redirects, blocked requests, expired URLs, or server errors.

Do not judge a source only by its total page count. Open representative product and article records and check whether titles, descriptions, prices, and article text match the public page. A smaller, clean source is more useful than a large source full of repeated navigation and low-value archive pages.

Improve the source when results are incomplete

If a product page is missing, first search the sitemap itself. If it is absent there, update the CMS or sitemap plugin rather than increasing the Fynd crawl limit. If it is present but not classified as a product, inspect the page source for valid Product JSON-LD. Correct malformed structured data and run another sync.

For missing articles, check that the post uses BlogPosting or Article JSON-LD and that it appears in a crawlable sitemap. A page can look like a blog post to a person but still be unclassified if it only contains visual content without editorial structured data.

Reduce noise at the source whenever possible. Exclude internal search pages, faceted filters, tag archives, and thin duplicate pages from the sitemap. These pages consume max_pages without improving answers. If your platform cannot generate a focused sitemap, use a separate sitemap URL for the knowledge-worthy section of the site.

Run ongoing syncs safely

Re-run the source after publishing products, revising prices, adding guides, or changing important editorial content. Treat the sitemap as the authoritative route list: when a page is removed from the sitemap, review whether it should remain in knowledge.

Make changes one at a time when diagnosing a problem. For example, first fix JSON-LD, then sync, then inspect the blog count. This makes the result attributable to the change you made. Keep max_pages documented with the reason for its value, especially if the source grows over time.

Troubleshooting checklist

The sitemap cannot be read

Confirm the exact URL, HTTPS certificate, redirects, and public accessibility. A browser test is helpful, but also ensure the response is not customized only for logged-in users.

Products appear as generic pages

Validate that product detail pages expose valid Product JSON-LD in the HTML available to the crawler. Check that the correct product URLs are listed, rather than only collection pages.

Articles do not appear in the blog count

Confirm that article URLs are present and expose BlogPosting or Article JSON-LD. Then re-run the sync and inspect the reported blog count.

The crawl is incomplete

Compare processed pages with max_pages. If the limit was reached, raise it carefully. If it was not reached, inspect skipped and failed URLs and correct the sitemap or page access issue.

Summary

A sitemap crawl is the right choice when your public site already contains the product and editorial knowledge Fynd needs. Keep the sitemap focused, expose accurate Product, BlogPosting, and Article JSON-LD, set a deliberate max_pages limit, and use the sync UI’s product and blog counts to validate each run.