GPTBot
OpenAI says GPTBot crawls content that may be used to train generative AI foundation models. Disallowing GPTBot means the site's content should not be used for that training. OpenAI says this setting is independent of OAI-SearchBot.
Cited robots.txt tokens for AI crawlers, so an agent can tell training, search, a person-triggered fetch, and an open crawl apart.
Google-Extended is a robots.txt control for Gemini training and grounding. It has no HTTP user-agent of its own, and Google says it does not change whether a page stays in Google Search. The search crawler token is Googlebot. Card checked 2026-10-17 against Google's common crawlers list (page updated 2026-10-15).
Applebot is Apple's search crawler for Spotlight, Siri, and Safari. Apple says it respects robots.txt on general search crawls aimed at that token, does not follow crawl-delay, and follows a Googlebot group only when the file does not mention Applebot. The checker reports that fallback and does not apply it. Applebot-Extended is a different token: a control for foundation-model training that does not crawl, and disallowing it does not remove a page from Apple search. Applebot was checked 2026-10-18 and Applebot-Extended 2026-10-17 against About Applebot (page published 2026-09-04).
PerplexityBot surfaces sites in Perplexity search. Perplexity says it is not used to train foundation models, and recommends allowing the token in robots.txt so the site appears in search results. Perplexity-User is a different token: a person-triggered fetch that generally ignores robots.txt. Both cards were checked 2026-10-17 against Perplexity Crawlers (page metadata dateModified 2026-01-29).
CCBot is Common Crawl's crawler for an open web-crawl repository. The CCBot page says publishing User-agent: CCBot and Disallow: / prevents the crawl, and that some crawlers falsely use that name. A post dated 2025-09-17 says the Opt-Out Registry is now the Opt-Out Ledger: Common Crawl encourages robots.txt and also publishes legal opt-out requests. The checker reports that ledger and does not apply it. The CCBot page was checked 2026-10-18 and shows no article date. The opt-out post was read the same day.
Amazon's page lists three independent tokens, checked 2026-10-18. Amazonbot may train Amazon AI models, and automated crawls honor allow and disallow. Amzn-SearchBot is for search such as Alexa, does not train generative models, and follows other search bots only when the file does not mention Amzn-SearchBot. The checker does not apply that fallback. Amzn-User fetches for a person's request and may not follow every robots.txt directive. Meta-ExternalAgent trains foundation models or indexes content. Meta-WebIndexer is for Meta AI search. Meta-ExternalFetcher fetches at a person's request and may bypass robots.txt. DuckAssistBot crawls for DuckDuckGo AI-assisted answers and is not used to train models; a Disallow takes effect after 72 hours and does not change search rankings. DuckDuckBot is the search crawler and respects WWW::RobotRules.
curl -s https://whosthatbot.com/v1/crawlers/perplexitybotThe registry is /v1/crawlers. 125 cards, 24 vendors, no key. Botcard does not fetch the site you are asking about.
Paste the file and a path. The checker answers allow or disallow for each token under RFC 9309, and it repeats that token's vendor stance. It does not fetch a host.
curl -s https://whosthatbot.com/v1/robots-check -H 'content-type: application/json' -d '{"robots_txt":"User-agent: GPTBot\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs"}'tools/call with name check_robots is the same check at POST /mcp. POST tools/call check_robots to /mcp. Send Content-Type: application/json and Accept: application/json, text/event-stream. Tools: check_robots, list_crawlers, get_crawler, suggest_robots. All four are read-only. The discovery card is /mcp/server-card.
The pasted file disallows GPTBot on /docs and allows *. The call does not fetch a host.
curl -s https://whosthatbot.com/mcp -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"check_robots","arguments":{"robots_txt":"User-agent: GPTBot\nDisallow: /\n\nUser-agent: *\nAllow: /\n","path":"/docs","ids":["gptbot"]}}}'That call returns fetched_target false. This row is disallow via group GPTBot, rule Disallow: /.
Name the kinds to allow and the kinds to block. The file's groups are the matching cards. Each group cites the vendor page. effect request_only means the vendor does not say the crawler obeys that line. A control_token such as Google-Extended stays out of a block-training file unless you name control_token. The file has no User-agent: * group. The call does not fetch a host.
curl -s https://whosthatbot.com/v1/robots-suggest -H 'content-type: application/json' -d '{"policy":"allow search, block training"}'tools/call with name suggest_robots is the same call at POST /mcp.
GPTBotOpenAI says GPTBot crawls content that may be used to train generative AI foundation models. Disallowing GPTBot means the site's content should not be used for that training. OpenAI says this setting is independent of OAI-SearchBot.
OAI-SearchBotOpenAI says OAI-SearchBot is used to surface sites in ChatGPT search. Sites that opt out are not shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI says this setting is independent of GPTBot, and that a robots.txt change can take about 24 hours to affect search.
OAI-AdsBotOpenAI says OAI-AdsBot visits landing pages submitted as ChatGPT ads, to check policy compliance and relevance. It says those visits are not used to train generative AI foundation models. The same page does not say whether robots.txt controls OAI-AdsBot.
ChatGPT-UserOpenAI says ChatGPT-User fetches pages when a person uses ChatGPT, a Custom GPT, or a GPT Action. It is not an automatic crawl, and OpenAI says robots.txt rules may not apply. It is not the token for ChatGPT search opt-out; that token is OAI-SearchBot.
GooglebotGoogle says Googlebot is a common crawler, so it obeys robots.txt on automatic crawls. Preferences for this token affect Google Search, including Discover and Search features, plus products such as Google Images, video mode, and Google News. Google-Extended is a different token and does not change Search inclusion.
Google-ExtendedGoogle says publishers use the Google-Extended robots.txt token to manage whether content Google has crawled may train future Gemini models (Gemini Apps and Vertex AI API for Gemini) and whether it may ground answers in Gemini Apps and in Grounding with Google Search on Vertex AI. Google says this token does not change inclusion or ranking in Google Search.
Google-CloudVertexBotGoogle says Google-CloudVertexBot performs crawls a site owner requested in order to build Vertex AI Agents. It does not affect Google Search. On this crawler, a robots.txt rule matches if it names Google-CloudVertexBot or Googlebot; Google says matching one of a crawler's tokens is enough.
ClaudeBotAnthropic says ClaudeBot collects public web content that may be used to train generative models. Restricting it signals that the site's future material should be left out of those training sets. The article says Anthropic's bots honor robots.txt, including a non-standard Crawl-delay.
Claude-UserAnthropic says Claude-User fetches sites when a person asks Claude something. Disabling the token stops Anthropic from retrieving that content for the user query, which can reduce visibility in user-directed web search. This article treats the token as a robots.txt control. OpenAI's own ChatGPT-User page says robots.txt may not apply to that different token.
Claude-SearchBotAnthropic says Claude-SearchBot crawls to improve search-result quality. Disabling it stops indexing for that search use and may reduce how visible or accurate the site is in search answers. It is a different token from ClaudeBot (training) and Claude-User (a person asked).
ApplebotApple says Applebot crawls the web for search in Spotlight, Siri, and Safari. Enabling the token in robots.txt allows site content to appear in those results. The same page says crawled data may also train Apple foundation models, and that opting out of that training is the Applebot-Extended token, not this one. A nosnippet tag, not this token, is how the page says to opt out of broad world-knowledge answers. Apple says this crawler respects robots.txt on general search crawls aimed at Applebot, does not follow crawl-delay, and follows the Googlebot group when the file does not mention Applebot but does mention Googlebot. The page does not say that a Disallow removes the site from search. The iTMS user agent on the same page is not this token. Applebot-Extended is a different token and is not this card.
Applebot-ExtendedApple says Applebot-Extended does not crawl pages. It only decides how Apple may use pages that Applebot already crawled: disallowing the token opts the site out of training Apple's general-purpose foundation models that power generative features in Apple Intelligence, Services, and Developer Tools. Apple says those pages can still be included in search results, and that this token is not a Search ranking signal. A nosnippet tag, not this token, is how the same page says to opt out of broad world-knowledge answers.
Perplexity-UserPerplexity says Perplexity-User visits a page when a person asks a question, so it can answer and link that page. It says this token is not used to crawl the web or to collect content for training foundation models. The same section says the token controls which sites those requests can access, and that the fetcher generally ignores robots.txt because a person requested the fetch. Perplexity says each robots.txt setting works independently. PerplexityBot is a different token and is not this card.
PerplexityBotPerplexity says PerplexityBot surfaces and links sites in Perplexity search results. It says this token is not used to crawl content for foundation models. The same section recommends allowing the token in robots.txt, and allowing requests from the published IP ranges, so the site appears in those results. Perplexity says each robots.txt setting works independently and that a change can take up to 24 hours. The page's machine-readable article metadata lists dateModified 2026-01-29 and no datePublished. Perplexity-User is a different token and generally ignores robots.txt. This card does not.
CCBotCommon Crawl's CCBot page says the crawler identifies itself as CCBot/2.0 with the FAQ URL inside the user-agent. The page description says it is a Nutch-based crawler that uses the crawl database to pick candidates. The body says the crawl supports an open repository of web crawl data, and that publishing User-agent: CCBot and Disallow: / prevents Common Crawl from crawling the site. The page says some crawlers falsely use the CCBot name, and that the real crawler uses dedicated IP ranges with reverse DNS, except IPv6 where reverse DNS is not yet supported. The CCBot page shows no article date. A separate post dated September 17, 2025 says the Opt-Out Registry is now the Opt-Out Ledger, because the list is immutable. That post says Common Crawl publishes every legal opt-out request it has received, encourages robots.txt to control crawling, and still processes legal requests. It points CCBot configuration back at the CCBot page and does not say the ledger replaces the Disallow block.
AmazonbotAmazon says Amazonbot is used to improve products and services, to give customers more accurate information, and that it may be used to train Amazon AI models. The page says each of Amazonbot, Amzn-SearchBot, and Amzn-User is an independent setting, and that a change may take about 24 hours. Automated crawling by these user agents honors allow and disallow, does not support crawl-delay, and respects the noarchive, noindex, and none robots meta tags. Amzn-SearchBot and Amzn-User are different tokens.
Amzn-SearchBotAmazon says Amzn-SearchBot improves search in Amazon products and services. Permitting it makes a site eligible to appear in search experiences such as Alexa. Amazon says this token does not crawl content for generative AI model training. If robots.txt does not mention Amzn-SearchBot but allows other search bots, Amazon says this crawler follows those bots' directives. This check does not apply that fallback. Automated crawling honors allow and disallow and does not support crawl-delay. Each Amazon user-agent setting is independent. Amazonbot and Amzn-User are different tokens.
Amzn-UserAmazon says Amzn-User supports user actions, such as an Alexa query that needs current information, and may fetch a live page on that person's behalf. Because a person can start the action, Amazon says it may not follow all robots.txt directives. Amazon says this token does not crawl content for generative AI model training. The same page says automated crawling by the listed user agents honors allow and disallow. This card follows the Amzn-User paragraph. The checker still reports the RFC 9309 match and does not fetch a host. Amazonbot and Amzn-SearchBot are different tokens.
Meta-ExternalAgentMeta says Meta-ExternalAgent crawls for uses such as training foundation AI models or improving products by indexing content directly. The page says to block a crawler by adding a Disallow for it, and that a robots.txt change can take up to 24 hours because crawlers may cache the file. The robots.txt example on the page uses User-agent: meta-externalagent. Meta-ExternalFetcher is a different token and may bypass robots.txt. Meta-WebIndexer is a different token.
Meta-ExternalFetcherMeta says Meta-ExternalFetcher fetches individual links at a person's request. It supports product functions such as evaluating and improving agentic AI, including helping AI navigate a site to complete tasks. Meta says this crawler may bypass robots.txt because a person requested the fetch. The checker still reports the RFC 9309 match and does not fetch a host. Meta-ExternalAgent and Meta-WebIndexer are different tokens.
Meta-WebIndexerMeta says Meta-WebIndexer crawls to improve Meta AI search result quality, and that allowing the token helps Meta cite and link the site in Meta AI responses. The page says to block a crawler by adding a Disallow, and that a robots.txt change can take up to 24 hours. Meta-ExternalAgent is the training token on the same page. Meta-ExternalFetcher may bypass robots.txt and is not this card.
DuckAssistBotDuckDuckGo says DuckAssistBot crawls pages in real time for AI-assisted answers that cite their sources, and that this data is not used to train AI models. A publisher opts out by disallowing the DuckAssistBot user agent. DuckDuckGo says that change takes effect after 72 hours and does not change organic search rankings or whether the site appears in search results. The page lists crawling@duckduckgo.com for opt-out problems. DuckDuckBot is a different token.
DuckDuckBotDuckDuckGo says DuckDuckBot improves search results. The page says the user agent will appear as DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html) and that it respects WWW::RobotRules, linking that name to the MetaCPAN module page. DuckAssistBot is a different token: a real-time crawl for AI-assisted answers, not this card.
BytespiderToutiao Search's webmaster page says Bytespider crawls pages, builds an index, and answers search queries. The page does not mention generative-model training or robots.txt. A separate Robots matching page (https://zhanzhang.toutiao.com/docs/intro/520) describes * and $ wildcards, path rules it says match Google's, and unordered query-parameter matching. That page does not name Bytespider and does not say the crawler honors robots.txt. The About page says a reverse-DNS hostname ending in .bytedance.com identifies the crawler, and that a crawl-frequency limit set in the webmaster tool takes effect within one day. Questions go to zhanzhang@bytedance.com.
MistralAI-UserMistral says MistralAI-User is for user actions in Vibe. When a person asks Vibe a question, it may visit a page and include a link to the source. Mistral says this token governs which sites those requests can be made to. It is not an automatic crawl and is not used to crawl content for generative AI training. MistralAI-Index and MistralAI-Training are different tokens.
MistralAI-IndexMistral says MistralAI-Index automatically crawls the web for indexing only, so Mistral search can help answer questions in Vibe. Mistral says that content is not used for generative AI training of any kind. The page lists this token as a robots.txt tag and does not say that the crawler obeys the file or that a Disallow stops indexing. MistralAI-User and MistralAI-Training are different tokens.
MistralAI-TrainingMistral says MistralAI-Training crawls web content to help build datasets for training Mistral generative AI models, and that webmasters can disallow this user agent in robots.txt. Mistral says this crawler is not used for search indexing or to answer live user queries in Vibe. MistralAI-User and MistralAI-Index are different tokens.
DiffbotDiffbot says its web crawls follow a site's robots.txt by default, including Disallow and crawl-delay, and that a partnership or agreement with the site can override that. This check does not apply that override and does not apply crawl-delay. Diffbot says this user agent is general proactive crawling so sites can be discovered and cited from the Diffbot Knowledge Graph and web search, and that it is not used for AI training. Diffbot-User is a different token. The page's note on Extract and Crawlbot is about a customer's own user agent, not this card.
Diffbot-UserDiffbot says Diffbot-User is for requests on behalf of a person browsing a URL in Diffbot software, in response to that person's input. The page does not say whether this token obeys robots.txt. The default adherence sentence on the same page is about Diffbot web crawls, which are the Diffbot token, not this one. The page's note on Extract and Crawlbot is about a customer's own user agent, not this card.
AI2BotAi2 says the AI2 Bot explores certain domains to find web content used to train open language models. The page publishes one HTTP user-agent and says that string can be used to filter or reject the crawler's traffic. It does not say whether robots.txt controls this token.
ImagesiftBotThe Imagesift by Hive page says ImageSiftBot scrapes publicly available images for its web intelligence products, and that those products use the saved index for search and retrieval of similar images. Standard robots.txt directives that target ImagesiftBot are respected. If there is no ImagesiftBot rule and there is a Googlebot rule, the page says ImagesiftBot follows the Googlebot directives. This check does not apply that fallback. The page says ImagesiftBot supports crawl-delay as a minimum number of seconds between the start of consecutive requests. This check does not apply crawl-delay.
MojeekBotMojeek says MojeekBot is the web crawler for the Mojeek search engine. It obeys the Robot Exclusion Standard by taking the first record whose User-Agent contains MojeekBot, otherwise the first User-Agent * record, and it does not support the crawl-delay directive. This check matches the whole token and merges every group that names it, which is RFC 9309, and it does not apply crawl-delay. The page also says a noindex meta tag is fetched but not added to the search database. This check does not read meta tags.
Google-GeminiNotebookGoogle says the Gemini Notebook fetcher requests individual URLs that Gemini Notebook users have provided as sources for their projects. The user-triggered fetchers section says these fetchers generally ignore robots.txt rules because a user requested the fetch. The page publishes mobile and desktop HTTP user-agent strings and does not print a separate robots.txt User-agent line. The same section lists Google-NotebookLM as a former agent supported until August 2026. This card does not include that string.
Google-AgentGoogle says Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request. The user-triggered fetchers section says these fetchers generally ignore robots.txt rules because a user requested the fetch. The page publishes mobile and desktop HTTP user-agent strings and does not print a separate robots.txt User-agent line. That section says Google is also experimenting with Web Bot Auth, using the https://agent.bot.goog identity. This check does not verify that signature.
YandexBotYandex says YandexBot is the main indexing robot and that it takes the general rules in robots.txt into account. The MirrorDetector string is this token, not a separate card. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexImagesYandex says YandexImages indexes images to display them in Yandex Images and takes the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexVideoYandex says YandexVideo indexes videos for Yandex video search and takes the general rules in robots.txt into account. YandexVideoParser is a different token and does not take those general rules into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexMediaYandex says YandexMedia indexes multimedia data and takes the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexBlogsYandex says YandexBlogs is the blog search robot that indexes post comments and takes the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexVideoParserYandex says YandexVideoParser indexes videos for Yandex video search and does not take the general rules in robots.txt into account. YandexVideo is a different token and does take those general rules into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexComBotYandex says YandexComBot indexes content for search in languages other than Russian. It can index that content if there is no explicit robot-specific restriction, and it does not take the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexAdditionalBotYandex's Search with Yandex AI page says a Disallow for YandexAdditional or YandexAdditionalBot keeps that page out of Yandex AI answers. The example disallows one path under User-agent: YandexAdditionalBot, and the page says the change applies after a recrawl, typically 2–14 days. The robot list says these tokens help process robots.txt for that purpose, apply to pages the primary crawler has already indexed, do not make indexing requests, and do not take the general rules into account. https://yandex.com/support/webmaster/en/yandex-ai The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexAdditionalYandex's Search with Yandex AI page says a Disallow for YandexAdditional or YandexAdditionalBot keeps that page out of Yandex AI answers. The example disallows one path under User-agent: YandexAdditionalBot, and the page says the change applies after a recrawl, typically 2–14 days. The robot list says these tokens help process robots.txt for that purpose, apply to pages the primary crawler has already indexed, do not make indexing requests, and do not take the general rules into account. https://yandex.com/support/webmaster/en/yandex-ai The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexVertisYandex says YandexVertis is the search verticals robot and that it takes the general rules in robots.txt into account. YandexVerticals is a different token. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexVerticalsYandex says YandexVerticals is the robot of Yandex Verticals (Classifieds): Auto.ru, Yandex Realty, Yandex Jobs, and Yandex Reviews, and that it takes the general rules in robots.txt into account. YandexVertis is a different token. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexOntoDBYandex says YandexOntoDB is the information card robot and that it takes the general rules in robots.txt into account. YandexOntoDBAPI is a different token that downloads dynamic data and does not take those general rules into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexSitelinksYandex says YandexSitelinks checks the availability of pages used as sitelinks and takes the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexSpravBotYandex says YandexSpravBot is the Yandex Business robot and that it takes the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexWebmasterYandex says YandexWebmaster is the Yandex Webmaster robot and that it takes the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexAdNetYandex says YandexAdNet is the Yandex Advertising Network robot and that it takes the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexAccessibilityBotYandex says YandexAccessibilityBot downloads pages to check their accessibility for users. It sends up to 3 requests to the site per second and ignores the crawl-rate setting in Yandex Webmaster. The robot list says it does not take the general rules in robots.txt into account. This check does not apply that crawl-rate setting. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexCalendarYandex says YandexCalendar is the Yandex Calendar robot. It downloads calendar files by users' requests, and those files are often located in directories prohibited from indexing. The robot list says it does not take the general rules in robots.txt into account. The page's example is User-agent: YandexCalendar and Disallow: /. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexCheckBotYandex says YandexCheckBot downloads pages from websites in different regions of the world to check their availability to users and does not make indexing requests. The robot list says it does not take the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexDirectYandex says YandexDirect downloads information about the content of Yandex Advertising network partner sites to identify their topic categories to match relevant advertising. The robot list says it does not take the general rules in robots.txt into account. YaDirectFetcher is a different token: the page says that robot does not use robots.txt. YandexDirectDyn is a different token. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexDirectDynYandex says YandexDirectDyn generates dynamic banners. The robot list says it does not take the general rules in robots.txt into account. YandexDirect is a different token. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexFaviconsYandex says YandexFavicons downloads the site's favicon file to display in search results. The robot list says it does not take the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexFeedYandex says YandexFeed downloads structured data via links that website owners submitted to Yandex services and does not make indexing requests. The robot list says it does not take the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexImageResizerYandex says YandexImageResizer is the mobile devices robot and that it takes the general rules in robots.txt into account. YandexImages is a different token. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexMobileBotYandex says YandexMobileBot defines pages with layout suitable for mobile devices. The robot list says it does not take the general rules in robots.txt into account. The page's example is User-agent: YandexMobileBot and Disallow: /private/*.txt$. YandexMobileScreenShotBot is a different token. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexMobileScreenShotBotYandex says YandexMobileScreenShotBot takes a screenshot of the mobile page. The robot list says it does not take the general rules in robots.txt into account. YandexMobileBot and YandexScreenshotBot are different tokens. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexOntoDBAPIYandex says YandexOntoDBAPI is the information card robot that downloads dynamic data. The robot list says it does not take the general rules in robots.txt into account. YandexOntoDB is a different token and does take those general rules into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexPagecheckerYandex says YandexPagechecker accesses the page for validating structured data via the Structured data validator and that it takes the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexPartnerYandex says YandexPartner downloads information about the content of Yandex partner sites. The robot list says it does not take the general rules in robots.txt into account. The page does not name an advertising product for this token. YandexDirect is a different token. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexRCAYandex says YandexRCA collects data for generating previews. The robot list says it does not take the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexSearchShopYandex says YandexSearchShop downloads product catalogs in YML files by users' requests. These files are often placed in directories prohibited for indexing. The robot list says it does not take the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexTrackerYandex says YandexTracker is the Yandex Tracker robot. The robot list says it does not take the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexUserproxyYandex says YandexUserproxy proxies user actions on Yandex services. It sends requests in response to button clicks and downloads pages for online translation. The robot list says it does not take the general rules in robots.txt into account. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
YandexScreenshotBotYandex says YandexScreenshotBot takes a screenshot of the page. The robot list says it does not take the general rules in robots.txt into account. YandexMobileScreenShotBot is a different token. The robot list says an agreement between Yandex and the site owner can override some robots.txt restrictions. This check does not apply that agreement.
BaiduspiderBaidu says Baiduspider is the search-engine program that visits pages and builds them into the Baidu index. PC search, mobile search, and other search share this token. The FAQ says Baiduspider works on the robots.txt protocol, and that disallowing it keeps the site's pages out of Baidu search results and out of other search results Baidu provides. An updated robots.txt usually takes 48 hours to stop new indexing. Pages already indexed can take several months to leave the results. The same FAQ says a block of this token blocks all crawling from Baidu. This check matches the whole token and does not treat Baiduspider as a prefix. Baiduspider-image, Baiduspider-news, and Baiduspider-favo are different tokens. Baiduspider-video does not support the rules currently. Baiduspider-cpro and Baiduspider-ads do not follow robots.txt records. The FAQ says a meta robots tag can ask Baidu to index a page without showing a cached link. This check does not read meta tags.
Baiduspider-imageBaidu's FAQ lists Baiduspider-image as the image-search user-agent. It shows a robots.txt that disallows Baiduspider and allows Baiduspider-image on /image/. The FAQ says different user-agents can have different rules, and it does not put this token in the exceptions for Baiduspider-video, Baiduspider-cpro, or Baiduspider-ads. The same page says blocking Baiduspider blocks all crawling from Baidu. This check matches the whole token and does not treat Baiduspider as a prefix.
Baiduspider-videoBaidu's English FAQ lists Baiduspider-video as the video-search user-agent and says it does not support the rules currently. The Chinese FAQ at https://help.baidu.com/question?prod_id=99&class=476&id=2996 lists this token and does not say that. This check still reports the RFC 9309 decision for a group that names Baiduspider-video. The English FAQ says blocking Baiduspider blocks all crawling from Baidu. This check matches the whole token and does not treat Baiduspider as a prefix.
Baiduspider-newsBaidu's FAQ lists Baiduspider-news as the News search user-agent. It says different user-agents can have different rules, and the user-agent it names as not supporting the rules is Baiduspider-video, not this token. Baiduspider-cpro and Baiduspider-ads are the tokens that do not follow robots.txt records. The page says blocking Baiduspider blocks all crawling from Baidu. This check matches the whole token and does not treat Baiduspider as a prefix.
Baiduspider-favoBaidu's English FAQ lists Baiduspider-favo as the Baidu bookmark user-agent. The Chinese FAQ calls that product 百度搜藏. The page lists it as a product user-agent of the search program and does not describe a separate crawl. It is not in the exceptions for Baiduspider-video, Baiduspider-cpro, or Baiduspider-ads. The page says blocking Baiduspider blocks all crawling from Baidu. This check matches the whole token and does not treat Baiduspider as a prefix.
Baiduspider-cproBaidu's English FAQ lists Baiduspider-cpro as Union baidu. It says pages this crawler fetches are not built into the index. The crawler works on the agreement set with customers and will not work on the records set by robots.txt. The page gives union1@baidu.com for problems with this crawler. The Chinese FAQ at https://help.baidu.com/question?prod_id=99&class=476&id=2996 calls the product 百度联盟 and says it does not obey the robots protocol. This check does not apply that agreement and still reports the RFC 9309 decision. The page says blocking Baiduspider blocks all crawling from Baidu. This check matches the whole token and does not treat Baiduspider as a prefix.
Baiduspider-adsBaidu's English FAQ lists Baiduspider-ads as business search. It says pages this crawler fetches are not built into the index. The crawler works on the agreement set with customers and will not work on the records set by robots.txt. The page says to contact a customer service representative. The Chinese FAQ calls the product 商务搜索 and says it does not obey the robots protocol. This check does not apply that agreement and still reports the RFC 9309 decision. The page says blocking Baiduspider blocks all crawling from Baidu. This check matches the whole token and does not treat Baiduspider as a prefix.
YetiNaver's robots.txt guide says the Naver search robot follows the rules written in robots.txt. Its examples use User-agent: Yeti. If the site root has no robots.txt file, the robot treats every page as collectable. The same guide says a search robot may still request a favicon, or a JavaScript or CSS file, when the page that includes that file is allowed to be collected. The identification guide names Yeti as the robot for general Naver search. Ads-Naver and Blueno are separate tokens. This check matches the whole token.
Ads-NaverNaver Search Ads says Ads-Naver visits final landing URLs registered in the ads center and collects the public content of those pages so ads can be matched and ranked. Registering the URL is treated as permission to collect, and the robot does not consult robots.txt. The FAQ shows that a file with User-agent: * and Disallow: / still does not stop collection of a registered landing page. To stop collection, the advertiser sends the URL to the ads customer center. That request is not a robots.txt rule. This check still reports the RFC 9309 decision and does not apply the ads-center exception. The identification guide names this token separately from Yeti. This check matches the whole token.
BluenoNaver's help page names Blueno as the summary robot that runs when a person attaches or shares a link in Blog, Cafe, or Smart Editor. It prefers Open Graph or Twitter Card markup and otherwise tries to summarize the page. A summary is cached so later shares of the same link do not all fetch the page. The robots.txt guide at https://searchadvisor.naver.com/guide/seo-basic-robots says special-purpose robots that generate link previews may not consult robots.txt, or may not follow those rules completely. This check still reports the RFC 9309 decision. The identification guide names this token separately from Yeti. This check matches the whole token.
coccocbotCốc Cốc's robots.txt guide says its robots support the robots exclusion protocol. The robot list at https://coccoc.com/search/console/coc-coc-robots calls coccocbot the common name for every Cốc Cốc robot and publishes this HTTP user-agent. Each robot looks for the best-matching name, and a less specific match is ignored. This token is that less specific name. It is not a prefix of coccocbot-web. The guide says the most specific Allow or Disallow wins, and equal length uses Allow. An empty Disallow allows every page, and an empty Allow is ignored. This check skips a blank Allow or Disallow, so an empty Disallow does not block the site. This check matches the whole token. A non-200 response is treated as unrestricted access, and no response skips the site for a while. This check does not fetch. The guide says Crawl-delay is a whole number of seconds and a value above 10 is treated as 10. This check does not apply Crawl-delay.
coccocbot-webCốc Cốc's robots.txt guide says its robots support the robots exclusion protocol. The robot list at https://coccoc.com/search/console/coc-coc-robots calls coccocbot-web the main web robot and publishes two HTTP user-agent strings. Each robot looks for the best-matching name, and a less specific match is ignored. If the file does not mention this token and does mention coccocbot, this robot follows that group. This check reports that fallback and does not apply that fallback. This check matches the whole token. A non-200 response is treated as unrestricted access, and no response skips the site for a while. This check does not fetch. The guide says Crawl-delay is a whole number of seconds and a value above 10 is treated as 10. This check does not apply Crawl-delay.
coccocbot-imageCốc Cốc's robots.txt guide says its robots support the robots exclusion protocol. The robot list at https://coccoc.com/search/console/coc-coc-robots calls coccocbot-image the image robot. Each robot looks for the best-matching name, and a less specific match is ignored. If the file does not mention this token and does mention coccocbot, this robot follows that group. This check reports that fallback and does not apply that fallback. This check matches the whole token. A non-200 response is treated as unrestricted access, and no response skips the site for a while. This check does not fetch. The guide says Crawl-delay is a whole number of seconds and a value above 10 is treated as 10. This check does not apply Crawl-delay.
coccocbot-fastCốc Cốc's robots.txt guide says its robots support the robots exclusion protocol. The robot list at https://coccoc.com/search/console/coc-coc-robots calls coccocbot-fast the fast robot. Each robot looks for the best-matching name, and a less specific match is ignored. If the file does not mention this token and does mention coccocbot, this robot follows that group. This check reports that fallback and does not apply that fallback. This check matches the whole token. A non-200 response is treated as unrestricted access, and no response skips the site for a while. This check does not fetch. The guide says Crawl-delay is a whole number of seconds and a value above 10 is treated as 10. This check does not apply Crawl-delay.
coccocbot-adsCốc Cốc's robots.txt guide says its robots support the robots exclusion protocol. The robot list at https://coccoc.com/search/console/coc-coc-robots calls coccocbot-ads the advertising robot. Each robot looks for the best-matching name, and a less specific match is ignored. If the file does not mention this token and does mention coccocbot, this robot follows that group. This check reports that fallback and does not apply that fallback. This check matches the whole token. A non-200 response is treated as unrestricted access, and no response skips the site for a while. This check does not fetch. The guide says Crawl-delay is a whole number of seconds and a value above 10 is treated as 10. This check does not apply Crawl-delay.
coccocbot-shoppingCốc Cốc's robots.txt guide says its robots support the robots exclusion protocol. The robot list at https://coccoc.com/search/console/coc-coc-robots calls coccocbot-shopping the e-commerce robot. Each robot looks for the best-matching name, and a less specific match is ignored. If the file does not mention this token and does mention coccocbot, this robot follows that group. This check reports that fallback and does not apply that fallback. This check matches the whole token. A non-200 response is treated as unrestricted access, and no response skips the site for a while. This check does not fetch. The guide says Crawl-delay is a whole number of seconds and a value above 10 is treated as 10. This check does not apply Crawl-delay.
APIs-GoogleGoogle's special-case crawlers page says these crawlers are used where there is an agreement between the site and the product. The robots.txt token is APIs-Google. The page says the global user agent (*) is ignored. Crawling preferences for this token affect delivery of push notification messages by Google APIs. The list is not exhaustive. This check matches the whole token and still reports the RFC 9309 decision.
AdsBot-Google-MobileGoogle's special-case crawlers page names AdsBot-Google-Mobile as the robots.txt token for AdsBot Mobile Web. The page says the global user agent (*) is ignored. Crawling preferences for this token affect Google Ads' ability to check web page ad quality. A retired iPhone string on the same page used this same token and is no longer in use. AdsBot-Google is a different token. This check matches the whole token and still reports the RFC 9309 decision.
AdsBot-GoogleGoogle's special-case crawlers page names AdsBot-Google as the robots.txt token for AdsBot. The page says AdsBot ignores the global robots.txt user agent (*) with the ad publisher's permission. Crawling preferences for this token affect Google Ads' ability to check web page ad quality. This check does not treat AdsBot-Google as a prefix of AdsBot-Google-Mobile. This check matches the whole token and still reports the RFC 9309 decision.
Google-SafetyGoogle's special-case crawlers page says the Google-Safety user agent ignores robots.txt rules. It handles abuse-specific crawling, such as malware discovery for publicly posted links on Google properties, and is unaffected by crawling preferences. The page does not print a separate robots.txt User-agent line. This check matches the whole token and still reports the RFC 9309 decision.
Mediapartners-GoogleGoogle's special-case crawlers page says Mediapartners-Google visits participating sites to provide them with relevant ads. The robots.txt token is Mediapartners-Google. The page says the global user agent (*) is ignored. Preferences for this token affect ad-related Google products, for example Google AdSense and Google Ad Manager. This check matches the whole token and still reports the RFC 9309 decision.
Google-CWSGoogle says the Chrome Web Store fetcher requests URLs that developers provide in the metadata of their Chrome extensions and themes. The user-triggered fetchers section says these fetchers generally ignore robots.txt rules because a user requested the fetch. The page does not print a separate robots.txt User-agent line. This check matches the whole token and still reports the RFC 9309 decision.
FeedFetcher-GoogleGoogle's Feedfetcher article, updated 2026-02-11, says Feedfetcher crawls RSS or Atom feeds for Google News and WebSub after a person starts an app or service that asks for the feed. It says Feedfetcher ignores robots.txt entries because it behaves as a direct agent of that person, not as a robot. The article says a site can serve 404 or 410 to the Feedfetcher-Google user agent. This check does not apply that server response. The user-triggered list publishes the HTTP string FeedFetcher-Google. This check matches the whole token, so a group named only Feedfetcher does not match, and it still reports the RFC 9309 decision.
GoogleMessagesGoogle says the Google Messages fetcher generates link previews for URLs sent in chat messages. The user-triggered fetchers section says these fetchers generally ignore robots.txt rules because a user requested the fetch. The page does not print a separate robots.txt User-agent line. This check matches the whole token and still reports the RFC 9309 decision.
Google-PinpointGoogle says the Google-Pinpoint fetcher requests individual URLs that Pinpoint users specified as sources for their personal collections of documents. The user-triggered fetchers section says these fetchers generally ignore robots.txt rules because a user requested the fetch. The page does not print a separate robots.txt User-agent line. This check matches the whole token and still reports the RFC 9309 decision.
GoogleProducerGoogle says Google Publisher Center fetches and processes feeds that publishers explicitly supplied for use in Google News landing pages. The user-triggered fetchers section says these fetchers generally ignore robots.txt rules because a user requested the fetch. The page does not print a separate robots.txt User-agent line. This check matches the whole token and still reports the RFC 9309 decision.
Google-Read-AloudThe Google Read Aloud article, updated 2025-12-02, says Google-Read-Aloud reads web pages with text-to-speech when a person turns that on, and that it is not a web crawler and does not follow links. The article says a site cannot opt out with a robots.txt file. The opt-out it names is a meta tag, name google and content nopagereadaloud. This check does not read meta tags. The user-triggered list publishes mobile and desktop HTTP strings and lists google-speakr as a deprecated former agent. This card does not include that string. This check matches the whole token and still reports the RFC 9309 decision.
Google-Site-VerificationGoogle says Google Site Verifier fetches Search Console verification tokens. The user-triggered fetchers section says these fetchers generally ignore robots.txt rules because a user requested the fetch. The page does not print a separate robots.txt User-agent line. This check matches the whole token and still reports the RFC 9309 decision.
bingbotBing says Bingbot is the standard crawler and handles most crawling each day. The row title is Bingbot. The published strings contain bingbot/2.0. BingPreview's published strings on the same page also contain bingbot, so BingPreview is not a separate token. The help URL HTML is an application shell. The article was read from the help API that page loads, ArticleId 8c184ec0. The page says robots.txt files can be configured to tell Bing crawlers how to interact with the site. It does not say that this crawler obeys every directive. This check matches the whole token and still reports the RFC 9309 decision.
adidxbotBing says AdIdxBot is the crawler used by Bing Ads. It crawls ads and follows the websites from those ads for quality control, with desktop and mobile variants. The row title is AdIdxBot. The published strings contain adidxbot/2.0. The help URL HTML is an application shell. The article was read from the help API that page loads, ArticleId 8c184ec0. The page says robots.txt files can be configured to tell Bing crawlers how to interact with the site. It does not say that this crawler obeys every directive. This check matches the whole token and still reports the RFC 9309 decision.
MicrosoftPreviewBing says MicrosoftPreview generates page snapshots for Microsoft products, with desktop and mobile variants. The published strings contain MicrosoftPreview/2.0. The help URL HTML is an application shell. The article was read from the help API that page loads, ArticleId 8c184ec0. The page says robots.txt files can be configured to tell Bing crawlers how to interact with the site. It does not say that this crawler obeys every directive. This check matches the whole token and still reports the RFC 9309 decision.
BingVideoPreviewBing says BingVideoPreview provides previews of videos only in Bing, with desktop and mobile variants. The published strings contain BingVideoPreview/1.0. A group named BingPreview does not match this token. BingPreview is not a card. The help URL HTML is an application shell. The article was read from the help API that page loads, ArticleId 8c184ec0. The page says robots.txt files can be configured to tell Bing crawlers how to interact with the site. It does not say that this crawler obeys every directive. This check matches the whole token and still reports the RFC 9309 decision.
Slackbot-LinkExpandingSlack says Slackbot-LinkExpanding responds to links that people post in channels. It fetches as little of the page as it can, using HTTP Range headers, to read oEmbed and Twitter Card or Open Graph tags, and it fetches a referred image, video, or audio file to check it. Slack says those responses are cached for about 30 minutes. The page says Slack does not currently honor robots.txt, because it does not follow links and acts on behalf of a person. This check matches the whole token, so a group named Slackbot does not match, and it still reports the RFC 9309 decision. This check does not contact Slack.
Slack-ImgProxySlack says Slack-ImgProxy fetches and caches images posted into channels, so the image is served over HTTPS and the detailed referrer is hidden. The published string is Slack-ImgProxy 0.19. That string does not contain Slackbot. The page says Slack does not currently honor robots.txt, because it does not follow links and acts on behalf of a person. This check matches the whole token, so a group named Slackbot does not match, and it still reports the RFC 9309 decision. This check does not contact Slack.
SlackbotSlack says Slackbot is the default robot for any HTTP request the other two do not cover, including API requests for integrated services and Outgoing Webhooks a team configured. The published string is Slackbot 1.0. The page says Slack does not currently honor robots.txt, because it does not follow links and acts on behalf of a person. This check matches the whole token, so a group named Slackbot does not match Slackbot-LinkExpanding, and it still reports the RFC 9309 decision. This check does not contact Slack.
CloudFlare-AlwaysOnlineCloudflare says it crawls a site when Always Online is enabled. The published user agent contains CloudFlare-AlwaysOnline/1.0. The page does not say whether robots.txt controls this token. This check matches the whole token, so a group named Cloudflare does not match, and it still reports the RFC 9309 decision.
Cloudflare-HealthchecksCloudflare says it crawls a site when Health checks are enabled. The published user agent contains Cloudflare-Healthchecks/1.0 and a 16-character health check ID. The page does not say whether robots.txt controls this token. This check matches the whole token, so a group named Cloudflare does not match, and it still reports the RFC 9309 decision.
Cloudflare-Traffic-ManagerCloudflare says load balancing monitors use Cloudflare-Traffic-Manager/1.0 and a 16-character pool ID. The page does not say whether robots.txt controls this token. This check matches the whole token, so a group named Cloudflare does not match, and it still reports the RFC 9309 decision.
CloudFlare-PrefetchCloudflare says it crawls a site when Prefetch URLs is enabled. The published user agent contains CloudFlare-Prefetch/0.1. The page does not say whether robots.txt controls this token. This check matches the whole token, so a group named CloudFlare-AlwaysOnline does not match, and it still reports the RFC 9309 decision.
Cloudflare-SSLDetectorCloudflare says the SSL/TLS recommender uses the user agent Cloudflare-SSLDetector. The page says this crawler ignores your robots.txt file unless there are rules explicitly targeting the user agent. This check matches the whole token, so a group named Cloudflare does not match, and it still reports the RFC 9309 decision.
Cloudflare-diagnosticsCloudflare says Cloudflare-diagnostics is used when Support Engineers perform error checks and when continuous monitoring raises alerts in the Cloudflare dashboard. The page does not say whether robots.txt controls this token. This check matches the whole token, so a group named Cloudflare does not match, and it still reports the RFC 9309 decision.
Cloudflare Custom Hostname VerificationCloudflare says the user agent Cloudflare Custom Hostname Verification is used when you validate a custom hostname with an HTTP ownership token. The page does not say whether robots.txt controls this token. This check matches the whole token, including the spaces, so a group named Cloudflare does not match, and it still reports the RFC 9309 decision.
CloudflareBrowserRenderingCrawlerCloudflare says Browser Run can crawl third-party websites on behalf of Cloudflare customers. It does not crawl your site as part of a Cloudflare service you have enabled. The Browser Run /crawl endpoint uses CloudflareBrowserRenderingCrawler/1.0. The page points to bot detection IDs and cryptographic verification for blocking that traffic, and does not say whether robots.txt controls this token. This check matches the whole token, so a group named Cloudflare or CloudflareBrowserRendering does not match, and it still reports the RFC 9309 decision.
Cloudflare-AI-SearchCloudflare says AI Search indexes a website you own, in the same Cloudflare account, that you select as a data source. The crawler uses the user agent Cloudflare-AI-Search. The page says you can allow or block it with WAF rules using its bot detection ID, and does not say whether robots.txt controls this token. This check matches the whole token, so a group named Cloudflare or Cloudflare-AI does not match, and it still reports the RFC 9309 decision.
KimiBotKimi says KimiBot crawls content potentially used to train Kimi's foundation models. Disallowing KimiBot signals that the content should not be used for model training. The page says each listed crawler is configured independently, and it shows Crawl-delay in seconds. This check does not apply Crawl-delay. It matches the whole token, so a group named Kimi does not match KimiBot, and it still reports the RFC 9309 decision.
Kimi-UserKimi says Kimi-User handles actions a person starts in Kimi, such as summarizing one article or answering a question that needs live web retrieval. It is not an automatic bulk crawl. Kimi says robots.txt rules may not directly apply, and that disallowing this token may reduce the site's visibility in user-directed web searches. The page shows Crawl-delay in seconds. This check does not apply Crawl-delay. It matches the whole token, so a group named Kimi or KimiBot does not match Kimi-User, and it still reports the RFC 9309 decision.
Kimi-SearchBotKimi says Kimi-SearchBot powers Kimi search: it analyzes pages for relevance and builds the search index. Disallowing Kimi-SearchBot prevents the site from appearing in Kimi search results. The page shows Crawl-delay in seconds. This check does not apply Crawl-delay. It matches the whole token, so a group named Kimi or Kimi-Search does not match Kimi-SearchBot, and it still reports the RFC 9309 decision.
SBIntuitionsBotSB Intuitions says SBIntuitionsBot is used for AI development and information analysis. The page says its crawlers comply with the Robots Exclusion Protocol in RFC 9309. It does not publish a full HTTP user-agent. This check matches the whole token, so a group named SBIntuitions does not match SBIntuitionsBot, and it still reports the RFC 9309 decision.
SBIntuitions-SearchBotSB Intuitions says that when a person asks Sarashina a question, SBIntuitions-SearchBot browses relevant sites on that person's behalf and makes inferences to improve search results. It may include links to those sites. The page says the information from the browsed sites is not used in AI development. The crawlers comply with RFC 9309. The page does not publish a full HTTP user-agent. This check matches the whole token, so a group named SBIntuitions or SBIntuitionsBot does not match SBIntuitions-SearchBot, and it still reports the RFC 9309 decision.
Googlebot-ImageGoogle says preferences for Googlebot-Image affect Google Images, Discover, Video mode, and Search features that show images, logos, and favicons. The page also lists Googlebot as a robots.txt token and says matching one of a crawler's tokens is enough. This check matches the whole token, so a group named Googlebot does not match Googlebot-Image, and still reports the RFC 9309 decision. Common crawlers obey robots.txt on automatic crawls.
Googlebot-VideoGoogle says preferences for Googlebot-Video affect video-related Google Search features and other products that depend on videos. The page also lists Googlebot as a robots.txt token and says matching one of a crawler's tokens is enough. This check matches the whole token, so a group named Googlebot does not match Googlebot-Video, and still reports the RFC 9309 decision. Common crawlers obey robots.txt on automatic crawls.
Googlebot-NewsGoogle says Googlebot News has no separate HTTP user-agent. Crawling uses various Googlebot strings. Preferences for Googlebot-News affect Google News, including news.google.com and the Google News app. The page also lists Googlebot as a robots.txt token and says matching one of a crawler's tokens is enough. This check matches the whole token, so a group named Googlebot does not match Googlebot-News, and still reports the RFC 9309 decision. Common crawlers obey robots.txt on automatic crawls.
Storebot-GoogleGoogle says preferences for Storebot-Google affect all surfaces of Google Shopping, including the Shopping tab in Google Search and Google Shopping. The page lists only the token Storebot-Google. A group named Storebot does not match. This check matches the whole token and still reports the RFC 9309 decision. Common crawlers obey robots.txt on automatic crawls.
Google-InspectionToolGoogle says Google-InspectionTool fetches for Search testing tools, including the Rich Result Test and URL inspection in Search Console. It has no effect on Google Search or other products. The page also lists Googlebot as a robots.txt token and says matching one of a crawler's tokens is enough. This check matches the whole token, so a group named Googlebot does not match Google-InspectionTool, and still reports the RFC 9309 decision. Common crawlers obey robots.txt on automatic crawls.
GoogleOtherGoogle says GoogleOther does not affect a specific product. It is the generic crawler product teams may use to fetch publicly accessible content, including one-off crawls for internal research and development. The page lists only the token GoogleOther. A group named Google does not match. This check matches the whole token and still reports the RFC 9309 decision. Common crawlers obey robots.txt on automatic crawls.
GoogleOther-ImageGoogle says GoogleOther-Image fetches publicly accessible image URLs and, like GoogleOther, does not affect a specific product. The page also lists GoogleOther as a robots.txt token and says matching one of a crawler's tokens is enough. This check matches the whole token, so a group named GoogleOther does not match GoogleOther-Image, and still reports the RFC 9309 decision. Common crawlers obey robots.txt on automatic crawls.
GoogleOther-VideoGoogle says GoogleOther-Video fetches publicly accessible video URLs and, like GoogleOther, does not affect a specific product. The page also lists GoogleOther as a robots.txt token and says matching one of a crawler's tokens is enough. This check matches the whole token, so a group named GoogleOther does not match GoogleOther-Video, and still reports the RFC 9309 decision. Common crawlers obey robots.txt on automatic crawls.