SourceVouchBot
SourceVouchBot is the crawler behind SourceVouch, a directory of manufacturers and service providers. It visits company websites to build and check public company pages. It reads the company's name, description, general contact details, logo, and what the site says about certificates and sustainability ratings.
What it does and does not do
- It reads a few pages per website: the home page and pages such as About, Contact, Quality and Sustainability.
- It reads robots.txt first and follows it, including Crawl-delay. If it may not read your home page, it reads nothing at all. It waits between requests.
- It collects company facts only. It does not collect names, email addresses or phone numbers of individual people.
- It does not submit forms, sign in or download files other than logos.
How it finds companies
From open data that lists companies with their own website: Wikidata, and OpenStreetMap (© OpenStreetMap contributors, available under the Open Database Licence), read through the University of Freiburg's QLever service. Those sources give the website's address, and often the company's name, country and category; the rest of the page comes from the company's own website.
From OpenStreetMap we take places inside each country that have a website and are tagged as a factory (man_made=works), an industrial site (industrial=*) or a maker's workshop (a few craft=* values), with their name, operator, country and a category guessed from their tags. We send this OpenStreetMap-derived list to anyone who asks at hello@sourcevouch.com.
How to stop it
Add these lines to your website's robots.txt:
User-agent: SourceVouchBot Disallow: /
If your company already has a page on SourceVouch, you can claim it to correct or complete it, or ask for it to be removed. Write to hello@sourcevouch.com.