---
title: "AI crawler logs"
description: "Connect Cloudflare, Vercel or WordPress, or upload your server's access log, and see which AI crawlers read your site, which pages they fetch, and where they hit errors."
url: https://help.rankprompt.com/en/articles/122-ai-crawler-logs
collection: "Site health"
updated: 2026-11-19
language: en
---

# AI crawler logs

Connect Cloudflare, Vercel or WordPress, or upload your server's access log, and see which AI crawlers read your site, which pages they fetch, and where they hit errors.

An AI platform can only cite a page its crawler has read. **AI crawlers** reads your server logs and shows what GPTBot, ClaudeBot, PerplexityBot, Googlebot and the other AI crawlers actually fetched from your site.

Open it from **Website > AI crawlers** in the sidebar (`/analytics/ai-crawlers`). It is included in **Pro** and above and costs no credits.

## Which source fits your site

Rank Prompt looks at how your site is served and offers that way first; the others stay one click away:

-   **Cloudflare** when Cloudflare sits in front of the site. It sees every request, even when Vercel or WordPress run behind it.
    
-   **Vercel** when Vercel serves the site.
    
-   **WordPress** when the site runs WordPress and nothing in front of it reports visits.
    
-   **Upload a log** for everything else. A Netlify site uploads a log too: Netlify's log drains need its Enterprise plan.
    

A **Rank Prompt hosted blog** reports its own AI crawlers: the source appears by itself once the blog is live, on the plan that has the blog (Starter and above), with nothing to connect. Its history starts the day it started reporting. A blog in a folder of your site whose Cloudflare or Vercel source already reads that host is counted once, by that source.

## Uploading a log

1.  Click **Upload a log**
    
2.  Choose a file or drop it in the box
    
3.  Click **Upload**
    

The file is read in the background and the page updates when it is done. The **Uploaded logs** table shows each file with its status, the requests read and the bot hits found.

**What you can upload** (up to 100 MB per file, gzipped files are fine):

<table style="min-width: 50px;"><colgroup><col style="min-width: 25px;"><col style="min-width: 25px;"></colgroup><tbody><tr><th colspan="1" rowspan="1"><p>Source</p></th><th colspan="1" rowspan="1"><p>Format</p></th></tr><tr><td colspan="1" rowspan="1"><p>nginx, Apache, LiteSpeed</p></td><td colspan="1" rowspan="1"><p>The access log in the Combined Log Format, the default on most servers</p></td></tr><tr><td colspan="1" rowspan="1"><p>Cloudflare</p></td><td colspan="1" rowspan="1"><p>Logpush in JSON lines</p></td></tr><tr><td colspan="1" rowspan="1"><p>Netlify, other hosts and CDNs</p></td><td colspan="1" rowspan="1"><p>Any export as JSON lines or CSV with a time, a URL or path, a status and a user agent column</p></td></tr></tbody></table>

A file over 100 MB can be split and uploaded in parts. Uploading the same period twice counts those visits twice, so upload each period once.

## Connecting Cloudflare

If your site runs behind Cloudflare, Rank Prompt can read its AI crawler visits every hour with a token that can only read. Any Cloudflare plan works, the free one included.

1.  On the AI crawlers page, click **Connect Cloudflare** (in **Sources**, or in **Integrations > Measure and track**)
    
2.  In Cloudflare, open **My Profile > API Tokens** and choose **Create Token**, then **Create Custom Token**
    
3.  Add two permissions: **Zone, Zone, Read** and **Zone, Analytics, Read**
    
4.  Under **Zone Resources**, include your site's zone, then create the token and copy it
    
5.  Back in Rank Prompt, check the **Domain** (it starts as the brand's site), paste the token in **API token** and click **Connect**
    

A Cloudflare zone serves every subdomain under it, so Rank Prompt reads only your site's domain and its `www.`. If your zone serves other hosts, Rank Prompt lists them after you connect: tick the ones that are your site, type any other host of the zone under **Add** (for example `blog.yourbrand.com`), and click **Save hosts**. Up to 20 hosts can be read. You can change them later with **Choose hosts** on the source. Visits from addresses a crawler's company does not publish still show, marked as not verified.

The first read brings the last 30 days Cloudflare keeps, and the source shows **Reading the last month** until it is done, usually a few minutes. A host you add later gets its last month read too. After that each hour is read once it has settled, so a visit shows up one to two hours after it happened. Cloudflare samples busy sites, and Rank Prompt counts each sampled visit as the visits it stands for. If Cloudflare stops accepting the token, the source shows **Error** with the reason and a **Connect Cloudflare again** button: **Cloudflare refused the token** means it was deleted or expired, and **The token cannot read this domain** means its Zone Resources leave your site's zone out. Connecting again replaces the token. The token is kept encrypted.

Cloudflare only sees the requests that pass through it. If your site's DNS record is **DNS only** (grey cloud) and another host answers it, Netlify for example, Cloudflare counts nothing. Rank Prompt checks how your site answers and then says **Cloudflare is not in front** on the page and on the source, offers the source that fits instead, and leaves Cloudflare's days out of the chart. Turn on the proxy (orange cloud) on the record, or use another source.

## Connecting Vercel

If Vercel serves your site, Rank Prompt creates a log drain on your project that sends each AI crawler visit as it happens, usually within a minute. Log drains need a **Pro** or **Enterprise** team on Vercel, and Vercel bills the drained volume to that team (see Vercel's drains pricing). Rank Prompt keeps only the AI crawler visits.

1.  On the AI crawlers page, click **Connect Vercel** (in **Sources**, or in **Integrations > Measure and track**)
    
2.  In Vercel, open **Account Settings > Tokens** and choose **Create Token**. Set its **Scope** to the team that owns the site's project and pick an expiration
    
3.  Paste the token in Rank Prompt and click **Find projects**
    
4.  Choose the project. The one serving your site is marked **Serves your site**
    
5.  Click **Connect**
    

The drain sends only production traffic, signed with a secret only Rank Prompt and Vercel hold; anything not signed with it is refused. The token is kept encrypted and used to create the drain, check it once a day and remove it when you disconnect. Until the first crawler visit arrives the source shows **Waiting for the first delivery**.

If the drain stops, the source shows **Error** with the reason and a **Connect Vercel again** button: the token was refused, the team left Pro, the drain was deleted or paused in Vercel, or the token cannot reach the project anymore. Connecting again makes a new drain and removes the old one. **Disconnect** removes the drain from Vercel; if Vercel cannot be reached, delete it in your team's **Settings > Drains**.

## Connecting a WordPress site

A WordPress site can send its AI crawler visits on its own, every five minutes, through the free Rank Prompt Connector plugin (version 1.1.0 or later).

1.  Connect the site in **Integrations > Publish content** with a WordPress user who is an administrator
    
2.  On the AI crawlers page, click **Connect WordPress** (in **Sources**)
    
3.  In WordPress, open **Settings > Rank Prompt** and click **Send AI crawler visits to Rank Prompt**
    

Nothing is noted or sent until step 3. Pages a cache serves before WordPress loads (WP Rocket, Varnish, a CDN) are not seen, so counts can run low; an access log from the server or CDN counts everything. **Disconnect** in **Sources** stops sending.

## What the page shows

-   **Bot hits**: requests from AI crawlers in the period
    
-   **Verified**: the share of hits that came from an address the crawler's company publishes. Someone can send requests with GPTBot's name; only OpenAI's own addresses count as verified. Crawlers whose companies publish no addresses (Meta, Amazon, ByteDance) are never verified
    
-   **Pages crawled**: distinct pages AI crawlers fetched
    
-   **Error rate**: hits that got a 4xx or 5xx answer. A crawler that hits an error cannot read the page
    
-   **Hits per day** for the six most active crawlers
    
-   **Crawlers**: each crawler with its company, what it fetches for (**Training**, **Search index**, **User request**, **Ads**), its hits, verified share, pages, errors and last visit
    
-   **Pages crawled**: each page with the crawlers that fetched it, its hits, errors, last status and last visit. Filter it by crawler
    

Pick the period (7, 30 or 90 days) at the top of the page.

## Reading the results

-   **Search index** and **User request** crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot) are the ones behind citations. A page they never fetch cannot be cited in their answers.
    
-   **Errors** on a page a crawler keeps fetching are the first thing to fix.
    
-   **Training** crawlers (GPTBot, ClaudeBot, CCBot) gather data for future models; their volume says nothing about citations today.
    
-   Few or no AI crawlers at all usually means robots.txt or a firewall blocks them: run a [Technical SEO audit](https://help.rankprompt.com/en/articles/63-technical-seo-audit.md) on your home page.
    

## Privacy

Only daily counts per crawler and page are kept. IP addresses are used to verify each hit and then dropped, query strings are removed from every path, and the uploaded file is deleted as soon as it is read. Counts are kept for 13 months.

## Plans

AI crawlers is part of **Pro**, **Agency** and **Agency Plus**. If a brand's plan drops below Pro, its uploads and connected sources stop being read and the page shows the upgrade; a Vercel drain is switched off so Vercel stops billing its volume. Back on Pro, the sources pick up where they stopped.
