Declared allow · Scan admission
- GoodForBotsBot/1.0 (+https://goodforbots.com/bot)
The recorded robots admission decision allowed our crawler to proceed.
GoodForBotsBot admission along the homepage request chain
https://listingsaas.com/
robots.txt is present and valid
Declared allow · robots.txt
- goodforbotsbot (Site diagnostics)
- * (Default robots group)
- oai-searchbot (ai-search)
- claude-searchbot (ai-search)
- mistralai-index (ai-search)
- kimi-searchbot (ai-search)
- amzn-searchbot (ai-search)
- meta-webindexer (ai-search)
- exasearchbot (ai-search)
- aiwebindex (ai-search)
- duckassistbot (ai-search)
- shapbot (ai-search)
- yandexadditional (ai-search)
- yandexadditionalbot (ai-search)
- ai2bot (Unknown)
- webzio-extended (ai-training)
- claude-user (user-fetch)
- perplexity-user (user-fetch)
- mistralai-user (user-fetch)
- kimi-user (user-fetch)
- amzn-user (user-fetch)
- diffbot-user (user-fetch)
- firecrawlagent (extraction)
- webzio (extraction)
- meta-externalfetcher (user-fetch, agent)
- googlebot (search)
- bingbot (search)
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: *
allow: /
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
- perplexitybot (ai-search)
- chatgpt-user (user-fetch)
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: gptbot
User-agent: chatgpt-user
User-agent: perplexitybot
User-agent: claudebot
User-agent: anthropic-ai
User-agent: ccbot
User-agent: google-extended
User-agent: applebot-extended
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: gptbot
User-agent: chatgpt-user
User-agent: perplexitybot
User-agent: claudebot
User-agent: anthropic-ai
User-agent: ccbot
User-agent: google-extended
User-agent: applebot-extended
disallow: /
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: claudebot
User-agent: gptbot
User-agent: chatgpt-user
User-agent: perplexitybot
User-agent: anthropic-ai
User-agent: ccbot
User-agent: google-extended
User-agent: applebot-extended
disallow: /
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
- mistralai-training (ai-training)
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: mistralai-training
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: kimibot
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
- applebot-extended (ai-training)
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: applebot-extended
User-agent: gptbot
User-agent: chatgpt-user
User-agent: perplexitybot
User-agent: claudebot
User-agent: anthropic-ai
User-agent: ccbot
User-agent: google-extended
disallow: /
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
- google-extended (ai-training, grounding)
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: google-extended
User-agent: gptbot
User-agent: chatgpt-user
User-agent: perplexitybot
User-agent: claudebot
User-agent: anthropic-ai
User-agent: ccbot
User-agent: applebot-extended
disallow: /
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
- meta-externalagent (ai-training, product-improvement)
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: meta-externalagent
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
- amazonbot (ai-training, product-improvement)
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: amazonbot
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
- ccbot (open-dataset, ai-training)
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: ccbot
User-agent: gptbot
User-agent: chatgpt-user
User-agent: perplexitybot
User-agent: claudebot
User-agent: anthropic-ai
User-agent: google-extended
User-agent: applebot-extended
disallow: /
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: bytespider
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: diffbot
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: omgili
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: anthropic-ai
User-agent: gptbot
User-agent: chatgpt-user
User-agent: perplexitybot
User-agent: claudebot
User-agent: ccbot
User-agent: google-extended
User-agent: applebot-extended
disallow: /
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: claude-web
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: cohere-ai
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: googleother
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: baiduspider
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: petalbot
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
- awariosmartbot (Unknown)
- awariorssbot (Unknown)
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: awariosmartbot
User-agent: awariorssbot
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
- google-cloudvertexbot (extraction)
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: google-cloudvertexbot
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: qualifiedbot
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: cotoyogi
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: icc-crawler
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: atlassian-bot
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: fishbot
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: borderxbot
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: navubot
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: semrushbot-swa
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: wardbot
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: magpie-crawler
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
- citibotsitecrawler (Unknown)
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://listingsaas.com/
https://listingsaas.com/robots.txt
User-agent: citibotsitecrawler
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers