AI
Brave's Unblockable Crawler Means Robots.txt Alone Won't Protect Clients
Edward Ilin12 min read
Brave sells access to its search index to AI companies, and a New York Times lawsuit alleges that index holds more than 30 billion pages gathered by a crawler site owners cannot block by name. Brave has not yet responded in court, so the claims are unproven. For agencies, the practical risk is that client content can feed AI answers without sending a click.
Your robots.txt file may not protect anyone.
You manage the rules for every client site. You assume each AI crawler has a name you can block. One crawler apparently does not. And your clients' pages are probably already in its index.
Here is what changed, what it means for your accounts, and three moves for this week.
On Nov. 2, the New York Times sued Brave, the company behind the Brave browser and Brave Search. Search Engine Journal put it plainly: Brave sells its search index to AI, and your website is probably in it (read the analysis).
The complaint says Brave copied pages and resold them to AI firms. You will see below why you cannot simply opt out.
The uncomfortable part is not the lawsuit itself, but the plumbing underneath it.
Brave's search index is now a product sold to AI companies, and your clients' pages are likely inside it
What changed: a lawsuit alleges Brave's roughly 30 billion page index was built with a crawler sites couldn't block, then licensed to AI firms including Anthropic, Perplexity, Mistral and xAI
Start with the facts, because the details matter. TechCrunch reported that the complaint says Brave built an index of more than 30 billion web pages using an unidentified crawler, leaving publishers unable to block it. The Times says Brave copied those pages without identifying itself and then sold the content to AI companies.
The product is called the Brave Search API. Per Search Engine Journal's analysis of the filing, the complaint names five AI companies it says Brave supplies: Anthropic, Perplexity, Mistral, xAI and Cohere. None of them is being sued. Brave's own About page, quoted in the same analysis, says the API is used by some of the biggest names in AI.
The reason the crawler cannot be blocked by name is unusual. Brave's help page says its crawler does not advertise a differentiated user agent. Brave's stated reason: some sites let Google in and turn everyone else away, so a crawler with Brave's name on it would be refused. Brave says it follows Google's rules instead, so whatever Googlebot may not read, Brave will not read either.
- You can allow Google and block OpenAI's or Anthropic's named crawlers, because those names are published.
- You cannot allow Google and block Brave. It is both or neither.
- The one exit Brave points to is the noindex tag, which also tells Google to drop the page.
Two limits on all of this. Brave has not yet answered the lawsuit, so these are allegations. And the Times's own phrase for the missing crawler name is a "refusal to release the name of its web crawler bot."
Why this hits your workflow: Brave Search answers are generated from other websites' content, so your clients' pages can feed AI answers without sending anyone a click
This is where it lands on your desk. Search Engine Journal's analysis quotes the complaint describing an offering called "LLM Context" that hands over extracted page chunks and source metadata in a shape and length that satisfies the needs of an LLM without need for further scraping. In plain terms, the AI company receives your client's page already chopped up and writes an answer from it.
Whether that answer links back is the AI company's call. It is not the client's call, and it is not something Brave is selling. That breaks the old deal every account manager explains to a client: someone searches, sees your page, clicks, and lands on your site.
The scale of blocking effort already in the market shows how little protection robots.txt gives. Axios found that more than 40% of top U.S. publishers block AI training crawlers, compared with 25% that bar AI search crawlers. Those publishers are doing the work crawler by crawler. A crawler with no name sits outside that whole approach.
My judgment: plan as if robots.txt cannot stop AI reuse of client content
The lawsuit is unproven, and Brave has fought this fight before: when News Corp sent it a cease-and-desist, Brave sued first, saying indexing the web is something "all search engine operators must do to exist," per Search Engine Journal's account. News Corp then sued back, claiming Brave hides its crawlers. The court will sort out the allegations. You have work to do now.
Plan around the part nobody disputes. Brave itself says it has no separate crawler name. That means the standard control, a named line in robots.txt, does not exist for this index. Even a win for the Times would not automatically cover anyone else's pages. A ruling will say a lot about every site in that index, but it will not rewrite your clients' contracts or their exposure.
My position: assume any indexable page can be reused by AI systems, whatever the robots.txt says. Build your client advice on that today, and revise it only if the case changes the facts.
Do this week, step #1: Audit which client sites rely on robots.txt as their only defense against AI reuse and flag them in your next account review
Open each priority client's robots.txt. If the AI protection is a list of named bots and nothing else, flag the account. It is exposed to any crawler that does not announce itself.
You do not need a full technical audit to start. Write down three things per client: which AI crawlers are blocked by name, whether any pages sit behind a paywall or login, and whether anyone has ever decided that AI reuse of this content is acceptable. The third answer is usually "nobody decided," and that is the finding to bring to the account review.
- Flag clients with premium, licensed or regulated content first.
- Note that the only exit Brave offers is noindex, which also removes the page from Google. That is rarely an acceptable trade for a client who depends on search.
Do this week, step #2: Run each priority client's top queries in Brave Search and record where their content shows up in generated answers
Checking takes about 30 seconds per domain: go to search.brave.com and type site: followed by the client's domain. Everything that appears is in the index Brave sells. In the author's own check cited by Search Engine Journal, 16 pages showed on the first results page, and Brave served a stored copy of an updated description rather than a live look at the site. Expect the same lag on your clients.
Then run each client's five or six highest-value queries and capture what the results and any generated answers say. A simple sheet is enough:
- Query, date, and whether the client appears in results.
- Whether a generated answer draws on the client's content, and whether it links to them.
- Whether the stored copy is stale. Outdated claims in an AI answer are a client-facing risk of their own.
This is evidence you can show a client in minutes, and it turns an abstract lawsuit into their own pages on screen.
Do this week, step #3: Put one line in your client reporting template explaining that visibility in AI answers is now a separate metric from rankings and clicks
One sentence is enough: rankings and clicks measure the old deal, while presence inside AI answers measures a new one, and the two can move in opposite directions. Add it now, before a client asks why traffic dipped while their content is being quoted everywhere.
If your team produces content across many clients, the volume makes this harder, not easier. A platform like Brainpercent, which helps agencies publish branded posts and articles across up to ten channels from one idea, only raises the stakes of knowing where that content ends up. The more you publish, the more of it flows into indexes you cannot see. Keep the new reporting line simple, and revisit it when the court filings give you something firmer to say.
The bottom line: you cannot name this crawler, but you can measure its reach.
A Lawsuit Says Brave Sold Its 30-Billion-Page Index: Your Questions Answered
What do I tell a client who asks, "Is our content being used by AI because of this?"
Give them a short, honest answer: it might be, we can check, and we are checking. Nothing in the complaint has been proven, so avoid saying their pages were taken or sold. What you can say with confidence is that any page open to search crawlers may be reusable by AI systems, and that you are pulling evidence for their specific domain.
A good line for the call: "We ran your top queries in Brave Search and here is what shows up. If your content appears in generated answers, we will decide together whether that is acceptable or needs action." That moves the conversation from fear to a decision they own. If the content shows up stale or misquoted, that becomes the concrete problem you solve first.
Does the lawsuit only affect big publishers like the New York Times, or do small client sites count too?
The plaintiff is a large publisher, but the mechanics described in the complaint do not depend on size. A crawler that follows Google's rules and carries no separate name does not check whether a site is famous before indexing it. A regional dental practice and a national newspaper face the same lack of a named opt-out. The difference is leverage. Big outlets can sue or negotiate licensing deals, while a small client has only the controls on its own site.
That is why smaller accounts deserve a quick look too. They often have thinner legal protection and less awareness of where their pages end up. A ten-minute site: check in Brave Search tells you whether the domain is indexed at all, and that is enough to decide if the account needs a deeper conversation.
If I use noindex on a client's pages, will that fully keep them out of AI answers?
Noindex is the one exit Brave points to. Use it on a narrow set of pages: members-only resources, internal landing pages, or drafts that were never meant to be public. It also asks Google to drop the page, so a client who relies on organic search pays for that exit with rankings. That makes it the wrong choice for a whole site. Do not offer it as a guarantee.
Timing matters too. Search engines work from stored copies, and the article's own check showed Brave serving an older description. Removal takes time to show up, and anything already collected can stay in the index for a while. If a client has pages that must stay out of reach, put them behind a login. That is the fix that holds, and it does not depend on a tag being honored quickly.
How is Brave Search's AI answer feature different from the index that is the subject of the lawsuit?
They are related but not the same thing, and mixing them up will confuse a client. Brave's own search results can include a generated answer that pulls information from other websites and answers your query. That is Brave using its index in its own product. The lawsuit is about the separate step where the index is licensed through an API to outside AI companies.
For your reporting, that means two places to look. Check what Brave's own answers say about the client, and remember that other AI products may be drawing on the same underlying pages without ever mentioning Brave. A clean result in Brave's interface does not prove the content is absent from answers elsewhere, so record what you see and label it as one sample, not the full picture.
What should I watch for in the next few weeks so I do not overreact or miss something?
Watch for three things: Brave's formal response in court, any change to its crawler policy or the help page describing its user agent, and whether the other AI companies named in the filing comment. A new crawler name would change your advice immediately, because it would restore the standard robots.txt control. A court order on disclosure would do the same.
This week, run the audit, save the screenshots, and add the reporting line to your client updates. Do not rewrite contracts or pull content yet. Set a calendar reminder for two weeks out to reread the filings and your notes. Panicked site-wide changes cost agencies more than waiting for the facts, so make the three small moves now and hold the rest.
Key Takeaways on A Lawsuit Says Brave Sold Its 30-Billion-Page Index
The core issue is simple. A lawsuit alleges Brave built a huge index with a crawler that has no public name, then licensed it to AI companies. Brave has not answered in court, so treat every claim as unproven. But the missing crawler name is not in dispute: Brave says it has none, which means a named robots.txt rule cannot target it.
For your accounts, that changes the working assumption. Any indexable client page may end up inside an AI answer, and that answer may never send a click. You cannot name this crawler, but you can measure its reach: check what each client's index footprint looks like, log where their content appears in generated answers, and give clients one reporting line that separates AI visibility from rankings and clicks. That is three small jobs, and together they turn a vague legal story into evidence you can put in front of a client in a single meeting.
Do the robots.txt check and the Brave search sweep for your top clients this week, then add the reporting line before the next monthly report goes out. If you want less of your week spent producing the content that all this tracking is about, Brainpercent can write, design, and publish branded posts and articles across up to ten channels from one idea, so you can put the time back into protecting what you publish.
Ready to automate all this? Brainpercent is the all-in-one content platform that generates SEO articles, social posts, and videos for you, on autopilot. Start your free trial or see pricing.
Frequently Asked Questions
Make content like this for your brand.
Posts, images and videos from your own site, in minutes.
Try it free