curatebot — the ZETO crawler

ZETO shows people the sale items, in their size, from the online stores they follow. To do that it reads those stores' public product pages with a crawler that identifies itself, follows robots.txt, and goes slowly.

Identity

Product token
curatebot
User-Agent
curatebot/<version> (+https://zeto.app/crawler)
Operator
ZETO
Contact
support@zeto.app
Egress IPs
Not published yet. Write to the address above and we will confirm whether a specific request was ours.

One User-Agent, always over HTTPS, never varied per site. We never rotate addresses or disguise curatebot as a browser to get around a block.

Former addresses

This page has moved. The crawler is the same crawler, run by the same operator, under the same product token — only the address in its User-Agent changed. A log line from before the move reads like one of these, and all of them are ours:

curatebot/<version> (+https://shopcur8.com/crawler)
curatebot/<version> (+https://zeto.app/crawler)

What it fetches

  • Public product pages, product JSON (such as Shopify's /products.json), sitemaps and robots.txt of online retailers that a ZETO user has chosen to follow.
  • Nothing behind a login, no forms, no carts or checkout, no search boxes.
  • Never marketplaces or classifieds (eBay, Facebook Marketplace and the like) — those are reached only through their own APIs and data providers. Facebook Marketplace listings come from a paid listing-data provider, Apify, whose own network reads Facebook’s pages — ZETO’s crawler never does.
  • Conditional requests wherever the site supports them (ETag / If-Modified-Since), so an unchanged page costs the site a 304 and no body.
  • Product images, from wherever the store serves them — usually a platform image CDN rather than the store's own site. They are read to work out what a garment looks like, so a shopper can find it from a photo, and — for stores ZETO reads directly — to keep the copies described under “What it stores”. That CDN's own robots.txt governs those requests.
  • A store’s own icon, once every thirty days: the home page’s <head>, its web-app manifest if it links one, and at most four of the icons they name (the manifest icon, the Apple touch icon, the favicon, or the logo in the page’s structured data). It is shown beside the store’s name on its card, linking to the store, exactly as published — never altered, and never copied: ZETO keeps the icon’s address, its size and a fingerprint, and the icon itself is served from wherever the store serves it. These reads honour robots.txt and happen for every store a member follows — including stores ZETO does not yet index. Where a store publishes no usable icon, ZETO may instead show the profile picture of the Instagram account the store links from its own home page. That picture is obtained through data providers, not by ZETO’s crawler; ZETO keeps a small copy and removes it when the store leaves ZETO.
  • A brand’s own site icon, for a brand ZETO shows without a store of its own: from the brand’s home page and its web-app manifest, on the brand’s own domain and its www. only, once every thirty days. These reads honor robots.txt, and nothing is stored: ZETO keeps the icon’s address, its size and a fingerprint, never the picture.
  • A product page a member’s price check turned up — from a search for an item, or from an item’s own page — on a site that may not be a store ZETO follows: one read of that page for its price and, on a Shopify page that names no sizes, one read of the same product’s /products/<handle>.js on the same site. At most 5 such reads per price check, at least 2 seconds apart on any one site, no more than 2,000 a day to any one site, and a page read in the last 6 hours is not read again. These reads honor robots.txt, its home page / rule included, and its Crawl-delay; a 403, a challenge, or an email to the address above stops them for that site. Nothing of the page is kept but its price and sale price, whether it can be bought, its brand, name, product codes, sizes and address. Never a marketplace, never behind a login.
  • A store’s public returns and shipping policy pages about once a week, its size guide about once a month, its home page about once a day, its own locations or store-locator page about once a month, and, for a Shopify store, the pickup-locations list it shows for one product about once a month, to record what the store announced and where its shops are — kept as dated copies, never republished — and, where a store’s terms restrict reuse of its pages, its returns policy page alone, about once a week, read for the return window it states and not kept: nothing of the page is stored but the one sentence that states the window; robots.txt governs every one of these reads.
  • Where a store lists its shops through a store-locator service on its own page, the list that service sends every visitor for that store, about once a month — read at the address the store’s page uses, honouring that service’s robots.txt, only for a store whose terms have been reviewed, and used only to show where the store’s shops are.

What it stores

  • The product facts a shopper compares: name, brand, price and sale price, currency, sizes and their availability, image URLs, and the page's structured product markup.
  • Never reviews, reviewer names or ratings — those are stripped before anything is written.
  • Prices are kept as a history so a follower can see when an item went on sale.
  • Product photos, for stores ZETO reads directly: a private copy at the size the store itself publishes (never larger than 2048 pixels on the long side), and smaller copies made from it that are shown only to signed-in ZETO members, always beside the link to the product on the store’s own site. A small copy may also be kept to recognise the same item again; it is never shown. Copies carry no metadata from the original.
  • Never photos from stores ZETO reaches through an affiliate product feed, and never photos from marketplaces.
  • Besides those copies, a product photo is turned into a list of numbers that describes how it looks, and ZETO keeps that description and two fingerprints of the original. (One exception, recorded in our legal notes: while we are measuring how well that description works, a copy is held on an engineer’s own machine and deleted when the measurement ends.)

To keep your photos out of ZETO, disallow curatebot in the robots.txt of the host that serves your images, mark your product pages noimageindex, or write to support@zeto.app. Copies already made are removed from our servers and our edge the same day; a copy already in a member’s own browser or app cache is outside any removal.

Opting out

Add this to your robots.txt and the crawler stops within 24 hours (it re-reads robots.txt at least that often):

User-agent: curatebot
Disallow: /

Disallow rules for specific paths are honoured the same way, as is Crawl-delay. For an immediate stop, or to have already-collected product data removed, email support@zeto.app.

A text-and-data-mining reservation — tdm-reservation: 1 (TDMRep) or a Content-Signal of ai-train=no or ai-input=no — is read and recorded for every store, and for a store under UK or EU law it is honoured: ZETO does not compute image vectors from that store's photos or show them to an AI model.

Two exceptions, stated here so they are never a surprise. The first: before ZETO reads anything else from a store, and about once a month after that, it reads the store's public terms page (on Shopify, /policies/terms-of-service) even where a store's robots.txt still disallows that path, because ZETO cannot honour terms it is not allowed to read. What that page says is read and recorded; a reading the policy admits is applied by the system under a switch an operator enabled, and a store that says no in any form stops the next read.

The second is outside any store: to learn brand names, ZETO asks Wikidata's public query service (query.wikidata.org/sparql) a handful of questions a month, under Wikimedia's API and User-Agent rules — one question at a time, waiting whenever it is asked to — although that host's robots.txt asks general crawlers to stay off it. What the crawler obeys is robots.txt. A robots rule, a 403, a challenge, or an email to the address above stops it — even these two reads (on Wikidata's host, a robots rule that names curatebot). The crawl is disclosed here because a crawl found later is the exposure, not the read itself.

For retailers

What a store can do to be listed on ZETO — a product feed, or a robots.txt allowance plus a written yes — and what ZETO reads first and honours, is on the retailers page.

How fast it goes

  • One request at a time per site, at least 2 seconds apart by default.
  • A Crawl-delay in robots.txt is honoured up to 60 seconds; above that the site is treated as opted out.
  • Retry-After is honoured, and the crawler backs off on 429 and 5xx responses; repeated blocks or an anti-bot challenge stop it for that site without any retry storm.
  • Images come from a CDN shared by thousands of stores rather than from your own servers, so they go a little faster — at most 4 per second, and never more than one at a time to any one host. A Crawl-delay on that CDN still wins. Your own site keeps the 2-second pace above.
  • Response bodies are capped by type, and feeds are read once, streamed, never re-pulled unchanged.