Skip to content

Journal

·5 min read

Headless agents are not crawlers

GPTBot reads. Playwright acts. Mixing them into “bots” hides the thing that is already walking your checkout.

AgentsActors

A headless mannequin in profile facing a camera in a dark studio
One fetches the page. The other uses it. Different actors.

Crawlers retrieve documents for an index or a training corpus. Headless agents retrieve documents to do a job: compare prices, file a form, summarise a dashboard for someone who will never visit. Their user agents look like Chrome until they do not — Playwright, Puppeteer, crawl4ai, explicit agent strings. Vector 5 keeps them in ai_agent, next to — not inside — ai_crawler.

That split changes what you do on Monday. A crawler spike on /docs is a citation opportunity. An agent spike on /checkout is a product question: is your flow usable by software acting for a customer, and do you want it to be?

Human and robot hands reaching for the same cup
If they complete a goal, it is a goal. It is not a human goal until you say so.
  • ai_crawler — fetch, no JS, mapped to a model family.
  • ai_agent — automation, often headless, often completing events.
  • bot — Pingdom, Lighthouse, link previews. Noise, unless you study noise.
A crawler is a reader. An agent is a user you have not decided to welcome.