Declared allow · Scan admission
- GoodForBotsBot/1.0 (+https://goodforbots.com/bot)
The recorded robots admission decision allowed our crawler to proceed.
GoodForBotsBot admission along the homepage request chain
https://hytale-wiki.fun/
robots.txt is present and valid
Declared allow · robots.txt
- goodforbotsbot (Site diagnostics)
- * (Default robots group)
- mistralai-index (ai-search)
- kimi-searchbot (ai-search)
- amzn-searchbot (ai-search)
- meta-webindexer (ai-search)
- exasearchbot (ai-search)
- aiwebindex (ai-search)
- shapbot (ai-search)
- yandexadditional (ai-search)
- yandexadditionalbot (ai-search)
- mistralai-training (ai-training)
- kimibot (ai-training)
- ai2bot (Unknown)
- webzio-extended (ai-training)
- kimi-user (user-fetch)
- amzn-user (user-fetch)
- diffbot-user (user-fetch)
- firecrawlagent (extraction)
- webzio (extraction)
- meta-externalfetcher (user-fetch, agent)
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: *
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
- oai-searchbot (ai-search)
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: oai-searchbot
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
- claude-searchbot (ai-search)
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: claude-searchbot
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
- perplexitybot (ai-search)
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: perplexitybot
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
- duckassistbot (ai-search)
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: duckassistbot
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: gptbot
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: claudebot
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
- applebot-extended (ai-training)
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: applebot-extended
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
- chatgpt-user (user-fetch)
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: chatgpt-user
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: claude-user
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
- perplexity-user (user-fetch)
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: perplexity-user
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
- mistralai-user (user-fetch)
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: mistralai-user
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
- google-extended (ai-training, grounding)
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: google-extended
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
- meta-externalagent (ai-training, product-improvement)
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: meta-externalagent
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
- amazonbot (ai-training, product-improvement)
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: amazonbot
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
- ccbot (open-dataset, ai-training)
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: ccbot
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: googlebot
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: bingbot
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
- googlebot-image (Unknown)
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: googlebot-image
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: baiduspider
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared allow · robots.txt
No matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: duckduckbot
allow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: bytespider
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: petalbot
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: tiktok spider
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: timpibot
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers
Declared restriction · robots.txt
- google-cloudvertexbot (extraction)
A matching refusal for this URL in the applicable robots group. This is a declaration, not an access test.
Acquisition of https://hytale-wiki.fun/en
https://hytale-wiki.fun/robots.txt
User-agent: google-cloudvertexbot
disallow: /
robots.txt is present and validStates a position on AI crawlersAllows search and user-fetch crawlers