Skip to content

Journal

·6 min read

GPTBot never runs your analytics script

Training and search crawlers fetch HTML and leave. If your only collector is a browser snippet, those fetches never existed — and most privacy-first tools still miss them.

AI crawlersMiddleware

A dark server corridor with amber status lights receding into shadow
Crawlers visit the HTML. They never reach the script.

GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot do not execute JavaScript. They request a URL, read the HTML, and move on. A 1 KB cookieless tracker — or a 45 KB Google tag — never loads. Browser-only analytics therefore report a quieter internet than the one that is actually reading your pages.

That gap matters now. Assistants cite what their crawlers have fetched. If /pricing is fetched 340 times and later refers two humans, you have a page that models know and people do not click. If /docs/middleware is fetched less and refers more, that is the page to copy. You cannot see either ratio from a script that never ran.

Macro photograph of a network switch with a single faintly glowing cable
The hit exists on the wire. Collection has to happen there, or not at all.

Record the request, not the pageview

Vector 5 splits collection in two. The MIT tracker still counts people in browsers. A server middleware — Next.js, Express, Hono, Cloudflare Workers, or plain Node — inspects the user agent on the way in and forwards non-human hits to your instance. Crawlers are stored with the exact path they fetched and tagged ai_crawler. Headless agents land as ai_agent. Uptime monitors stay in bot and never inflate human metrics.

middleware.ts
import { vector5 } from '@vector5/middleware/next'

export const middleware = vector5({
  ingestUrl: process.env.V5_INGEST_URL!,
})

By default only non-human actors are recorded this way, so you do not double-count people who also run the script. Turn on recordHumans if the site itself has no JavaScript.

What you can do with the list

  • See which URLs each model family has actually read, not which ones you hoped they would.
  • Pair crawls with later human_via_ai referrals to find pages that get cited versus pages that get clicked.
  • Keep robots.txt honest: if you disallow a path, the middleware should go quiet there too.
If a crawler never runs JavaScript, a JavaScript collector cannot tell you it was there. The request log can.