URL Scraping

merchi.ai can scrape product pages directly from supplier or manufacturer websites and use the extracted content — images, descriptions, documents, and technical data — to generate enriched product content.

How scraping works

When a scrape job is triggered, merchi.ai visits the product URL, fetches the page HTML, and extracts the information defined in your scrape prompt. Images are downloaded, PDF documents (if configured in Assets) are stored, and all of this is passed to the AI model alongside your schema and writing knowledge.

The scraper uses a cascading fetch strategy to handle different site configurations:

A lightweight direct request is tried first
If the page is dynamic or blocked, a headless browser is used
If the browser attempt is also blocked, Bright Data proxies are used as a fallback

Our user agent

merchi.ai identifies itself to web servers using the user agent string:

merchi-ai-bot/1.0

This is sent with every request so that server operators can identify our traffic clearly.

What this means for you

If you intend to scrape supplier or manufacturer websites, we recommend contacting those sites and asking them to whitelist the merchi-ai-bot/1.0 user agent. This:

Reduces the chance of your scrape jobs being blocked or rate-limited
Ensures the site operator is aware of and consents to automated access
Protects your account from potential access restrictions

Most supplier technical or web teams can add a user agent whitelist in a few minutes. Sharing the exact string merchi-ai-bot/1.0 gives them everything they need.

robots.txt compliance

merchi.ai always reads and respects a site’s robots.txt file before scraping. This is the standard mechanism web server operators use to signal which paths automated agents are allowed to access.

What we respect

Disallow rules: If a path is disallowed for all agents (*) or specifically for merchi-ai-bot, we will not attempt to scrape that URL.
Crawl-delay: If a site specifies a crawl delay in their robots.txt, merchi.ai will wait that number of seconds between requests. This prevents your scraping activity from placing excessive load on the target server.

Asking a supplier to configure robots.txt

If you want to give merchi.ai controlled access to a supplier site, you can ask their team to add an entry to their robots.txt file:

User-agent: merchi-ai-bot
Allow: /products/
Crawl-delay: 2

This example allows our bot to access anything under /products/ with a 2-second pause between requests. They can restrict this to specific paths and set whatever crawl delay suits their infrastructure.

If a supplier wants to block our bot entirely they can add:

User-agent: merchi-ai-bot
Disallow: /

We will honour this and the scrape job will not attempt to fetch any pages from that domain.

Troubleshooting failed scrapes

If a scrape job fails or returns incomplete data, common causes include:

Bot detection: The site is blocking automated requests. Ask the supplier to whitelist merchi-ai-bot/1.0.
JavaScript rendering: Some pages only load product data after JavaScript executes. Our headless browser fallback handles most cases, but some sites use aggressive anti-bot measures.
Disallowed by robots.txt: Check whether the product URL path is disallowed. If so, contact the site operator.
PDF upload failures: If PDFs are not storing, check your Assets configuration and verify the product-pdfs storage bucket is set up correctly.

For persistent failures, contact support with the run ID from the Runs page and we can investigate further.