Research & field notes
Technical pilot: making ShelfSignal research discoverable
A documented crawlability change on our own site: what changed, what HTTP checks confirmed and which commercial effects remain unmeasured.
Vladimir Fedorov · Published 26 October 2026
The problem was discovery, not a measured sales decline
In an HTTP review on 26 October 2026, ShelfSignal’s public research pages were accessible, but /robots.txt and /sitemap.xml returned 404. Some canonical links pointed to the no-slash URL while the final served page used a trailing slash. A Dataset licence link pointed to a missing terms page.
A missing robots file is not a crawl ban. These observations identified incomplete discovery and metadata, not a proven reason for lost rankings or simulated shopping choices.
What was changed
- Added robots.txt with a sitemap location and a sitemap containing seven existing useful public pages.
- Aligned canonical links with the final public page addresses, keeping all existing slugs.
- Removed the unverified broken licence assertion from Dataset markup rather than inventing terms.
- Left private reports, the external application and contact-service delivery outside the change.
The work was committed and deployed with a backup. Public file hashes were checked against the tested version. The current research directory and service pages are a later content extension, not part of that original seven-URL snapshot.
Observed result
| Check | Before the technical change | After deployment |
|---|---|---|
| robots.txt | HTTP 404 | HTTP 200 |
| sitemap.xml | HTTP 404 | HTTP 200 with validated public URLs |
| Canonical links | Selected URLs used the redirecting form | Aligned to the final address |
| Traffic, enquiries or choice-share lift | No comparable 28-day baseline | Not established by this pilot |
What this demonstrates
This is a technical pilot on our own research site, not a customer storefront success story. It demonstrates scoped implementation and verification. It does not demonstrate more AI recommendations, more enquiries or more revenue.
For a commercial storefront audit, we agree the briefs and observation window first, implement the approved change, rerun the same measurement and report outcomes including a flat or negative result. A completed real outcome case will require those observations; it cannot be manufactured from a working sitemap.
Request your report
Tell us about your website and goals. We will reply with the appropriate scope and next steps.
From findings to approved improvements
Use the measurement on its own, or ask us to turn it into website recommendations and implementation. Optional managed hosting adds chat-based editing through an agreed messenger, with you in control of publication.