Find Products from a Website

Find Products from a Website lets you enter a supplier or manufacturer domain and have merchi.ai automatically discover all products on that site. Rather than building a spreadsheet of URLs manually, the discovery process reads the site’s structure, finds every product page, extracts the key information, and presents you with a list to review before importing.

This is an Enterprise feature. You must be on an Enterprise plan to access it.

How discovery works

When you start a discovery run, merchi.ai:

1. Checks robots.txt — reads the site’s robots.txt to identify disallowed paths and any crawl delay. Discovery respects these rules throughout the run.
2. Finds sitemaps — locates the site’s XML sitemap (including nested sitemaps) to get a complete map of product URLs without having to crawl every page.
3. Identifies product pages — filters the sitemap to URLs that look like product pages based on URL patterns and path structure.
4. Extracts product data — visits each product page and pulls the title, images, product code, and collection using structured data (JSON-LD), page content, and AI extraction as a fallback.
5. Stores images — downloads a thumbnail of each product image and stores it in your account so the discovery results display quickly without relying on the supplier’s servers.

Discovery runs as a background task and updates in real time as products are found. You can leave the page and come back — the run continues.

Credits

Discovery is billed on two components:

• Base charge: 1 credit per run, regardless of how many products are found
• Per product: 0.01 credits per discovered product (rounded up to the nearest whole credit)

Credits are reserved when the run starts. If your account does not have sufficient credits, the discovery will not begin.

Importing discovered products into your account triggers separate scrape and content generation jobs, each billed at the standard rate of 1 credit per product.

Starting a discovery

1. Go to Add Products and select Find Products from a Website
2. Enter the domain you want to discover products from (e.g. supplier.com)
3. Click Start Discovery

merchi.ai will validate the domain, check robots.txt, and begin the discovery process. You will be taken to the discovery results page where products appear as they are found.

The discovery results page

The results page shows each discovered product with:

• Image — a thumbnail extracted from the product page
• Title — the product name
• Product Code — SKU or product identifier if found on the page
• Collection — category or range the product belongs to
• Your Product ID — an optional field where you can assign your own internal identifier before importing

You can switch between a grid view and a list view using the toggle at the top right.

Assigning your own product ID

The Your Product ID column in list view (or the inline field in grid view) lets you enter your own SKU or internal code before importing. If you enter a value here, it takes precedence over any product code found on the supplier’s page during scraping.

This is useful when your internal identifiers do not match the supplier’s codes.

Selecting products to import

Check the products you want to import, then click Import Selected Products. This triggers a scrape and content generation job for each selected product. You can track progress on the Runs page.

Past discoveries

All previous discovery runs are listed on the Discoveries page, accessible from the main navigation. Each entry shows the domain, the number of products found, the run status, and when it was started. Click any entry to view the full results and import products from a previous run.

Handling blocked sites

Some suppliers use bot detection that blocks automated access. If a site is blocking the discovery:

• Ask the supplier’s technical team to whitelist the user agent merchi-ai-bot/1.0
• Ask them to configure their robots.txt to allow merchi-ai-bot on their product paths

See URL Scraping for details on how to request whitelisting and what to ask for.

Multi-language sites

If a supplier site runs in multiple languages, discovery deduplicates product URLs using hreflang signals and URL patterns. Products that exist only to serve a localised version of an already-discovered URL are not added as separate entries.

Troubleshooting

Discovery finds 0 products The site may not have an XML sitemap, or the sitemap may not include product URLs. Sites that rely entirely on JavaScript rendering for their catalogues can also return empty results. In these cases, the Scrape Spreadsheet is a better option — build the URL list manually.

Discovery shows “blocked” status The site’s robots.txt is disallowing access, or the site is actively blocking the crawler. Contact the supplier to request a whitelist.

Images not showing Image extraction can fail if the supplier’s server blocks hotlinking. merchi.ai stores a thumbnail copy of each image during discovery, so if an image still does not appear after the run completes, it could not be fetched from that site.

“A discovery is already running” Only one discovery can run per account at a time. Wait for the current run to complete or fail before starting a new one. You can view active runs on the Discoveries page.