# ============================================================================ # JYNX GAMING PLATFORM - ROBOTS.TXT # ============================================================================ # Last Updated: 2026-10-11 (G4.4.6: Disallow: /ab/ experiment-variant family # added to the wildcard group + every engine group that carries the # protected-directories trio — pinned in scripts/validate-robots.js per the # G7.F.1 matrix convention; family contract docs/reference/ab-family-contract.md) # 2026-10-10 (2026 census refresh, G7.F.1: search/referral # agents added, Google AdsBot named groups pinned — see # docs/guides/robots-txt-strategy.md §Strategic Decisions 5) # Site: https://jynx.app # Languages: 16 (en, fr, es, pt, de, it, ru, ar, hi, id, pl, uk, ko, tr, bn, vi) # Total Pages: 230+ (6 main pages + 119 blog posts × 16 languages) # Sitemaps: 32 language-separated sitemaps (16 main + 16 blog) via sitemap-index.xml # Policy: Open platform - Allow search engines and AI crawlers # Strategy: Gaming platforms benefit from AI recommendations and maximum discoverability # Contact: support@jynx.app # ============================================================================ # ============================================================================ # DEFAULT RULES - ALL BOTS # ============================================================================ # Open by default - gaming social platforms benefit from discoverability User-agent: * Allow: / # --- Protected Directories --- Disallow: /admin/ Disallow: /api/ Disallow: /private/ # --- Experiment Variant Family (G4.4.6) --- # /ab/{id}/{a|b}/ are A/B variant copies of canonical pages (worker serves # variant bytes AT the canonical URL; these direct URLs are the debug/crawl # surface). Direct-URL noindex = worker X-Robots-Tag + this Disallow — # canonical pages stay allowed. Mirrored in each engine UA group below # (named groups never inherit the wildcard block) and pinned by # `npm run seo:robots` (G7.F.1 matrix convention). Disallow: /ab/ # --- Duplicate Content Prevention --- # Block search result pages Disallow: /search?q= Disallow: /*/search?q= # --- Crawl-Trap Query Parameters (Phase 1 / Block P1.3) --- # Temporary defensive block while query-param category/search URLs exist. # Once Block P2.4 ships static /blog/category/{slug}/ pages, these stay # as belt-and-suspenders. Mirrored under each major-engine UA block below. Disallow: /*?category= Disallow: /*?q= Disallow: /*?ref= Disallow: /*?utm_* # --- Assets (Critical for Google Rendering) --- Allow: /*.css$ Allow: /*.js$ Allow: /*.webp$ Allow: /*.png$ Allow: /*.jpg$ Allow: /*.svg$ Allow: /assets/ # --- Translation Files (Protect from Indexing) --- Disallow: /*.json$ # --- AI Standard Files (llmstxt.org) --- # Explicitly allow llms.txt and llms-full.txt for AI ingestion (Phase 2 / Block P2.1) Allow: /llms.txt Allow: /llms-full.txt # ============================================================================ # SITEMAPS # ============================================================================ # Master sitemap index (references all 32 language-separated sitemaps) # This is the primary entry point - it includes: # - English sitemaps: sitemap.xml, blog-sitemap.xml (at root) # - Localized sitemaps: {lang}/sitemap.xml, {lang}/blog-sitemap.xml (in subdirectories) # - 16 languages total: en, fr, es, pt, de, it, ru, ar, hi, id, pl, uk, ko, tr, bn, vi Sitemap: https://jynx.app/sitemap-index.xml # ============================================================================ # MAJOR SEARCH ENGINES # ============================================================================ # --- Google (Googlebot) --- # No crawl-delay for Google - they have sophisticated rate limiting # Note: Protected directories still apply User-agent: Googlebot Disallow: /admin/ Disallow: /api/ Disallow: /private/ # Experiment variant family (G4.4.6 — see the default block's /ab/ note) Disallow: /ab/ Disallow: /*.json$ # Crawl-trap query parameters (P1.3 — see comment in default block) Disallow: /*?category= Disallow: /*?q= Disallow: /*?ref= Disallow: /*?utm_* Allow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-Video Allow: / # --- Google Ads crawlers (F-09: NAMED groups, not wildcard inheritance) --- # Google's AdsBot crawlers are excluded from User-agent: * wildcard matching # (Google Search Central docs) — a * group never reaches them; they must be # named explicitly. Their crawl feeds Google Ads landing-page quality, and # this program runs a live paid denominator, so the stance is Allow. # Matrix-pinned in scripts/validate-robots.js (named-group existence, not # wildcard fallback). Mediapartners-Google (AdSense) keeps its block below — # no AdSense runs on this site. User-agent: AdsBot-Google Allow: / User-agent: AdsBot-Google-Mobile Allow: / # --- Microsoft Bing (Bingbot) --- User-agent: Bingbot Disallow: /admin/ Disallow: /api/ Disallow: /private/ Disallow: /ab/ Disallow: /*.json$ # Crawl-trap query parameters (P1.3) Disallow: /*?category= Disallow: /*?q= Disallow: /*?ref= Disallow: /*?utm_* Allow: / Crawl-delay: 1 # --- Yahoo (Slurp) --- User-agent: Slurp Disallow: /admin/ Disallow: /api/ Disallow: /private/ Disallow: /ab/ Disallow: /*.json$ # Crawl-trap query parameters (P1.3) Disallow: /*?category= Disallow: /*?q= Disallow: /*?ref= Disallow: /*?utm_* Allow: / Crawl-delay: 1 # --- DuckDuckGo (DuckDuckBot) --- User-agent: DuckDuckBot Disallow: /admin/ Disallow: /api/ Disallow: /private/ Disallow: /ab/ Disallow: /*.json$ # Crawl-trap query parameters (P1.3) Disallow: /*?category= Disallow: /*?q= Disallow: /*?ref= Disallow: /*?utm_* Allow: / # --- Brave Search --- User-agent: Brave-Indexer Disallow: /admin/ Disallow: /api/ Disallow: /private/ Disallow: /ab/ Disallow: /*.json$ Allow: / # ============================================================================ # INTERNATIONAL SEARCH ENGINES # ============================================================================ # Critical for multilingual SEO (16 languages) # --- Yandex (Russia) - For Russian language content --- User-agent: Yandex Disallow: /admin/ Disallow: /api/ Disallow: /private/ Disallow: /ab/ Disallow: /*.json$ Allow: / Crawl-delay: 5 User-agent: YandexBot Disallow: /admin/ Disallow: /api/ Disallow: /private/ Disallow: /ab/ Disallow: /*.json$ Allow: / Crawl-delay: 5 User-agent: YandexMobileBot Disallow: /admin/ Disallow: /api/ Disallow: /private/ Disallow: /ab/ Disallow: /*.json$ Allow: / Crawl-delay: 5 # --- Baidu (China) - Future expansion --- User-agent: Baiduspider Disallow: /admin/ Disallow: /api/ Disallow: /private/ Disallow: /ab/ Disallow: /*.json$ Allow: / Crawl-delay: 5 User-agent: Baiduspider-mobile Disallow: /admin/ Disallow: /api/ Disallow: /private/ Disallow: /ab/ Disallow: /*.json$ Allow: / Crawl-delay: 5 # --- Naver (South Korea) - For Korean language content --- User-agent: NaverBot Disallow: /admin/ Disallow: /api/ Disallow: /private/ Disallow: /ab/ Disallow: /*.json$ Allow: / Crawl-delay: 5 User-agent: Yeti Disallow: /admin/ Disallow: /api/ Disallow: /private/ Disallow: /ab/ Disallow: /*.json$ Allow: / Crawl-delay: 5 # --- Seznam (Czech Republic) --- User-agent: SeznamBot Disallow: /admin/ Disallow: /api/ Disallow: /private/ Disallow: /ab/ Disallow: /*.json$ Allow: / Crawl-delay: 5 # ============================================================================ # AI CRAWLERS - ALLOWED WITH RATE LIMITING # ============================================================================ # Strategy: Gaming platforms benefit from AI recommendations and discovery # Rate limiting protects server resources while allowing full access # --- OpenAI (ChatGPT) --- User-agent: GPTBot Allow: / Crawl-delay: 10 User-agent: ChatGPT-User Allow: / Crawl-delay: 10 # --- OpenAI search/referral (2026 census: sends traffic → allowed) --- # OAI-SearchBot surfaces the site in ChatGPT search answers # (developers.openai.com/api/docs/bots) User-agent: OAI-SearchBot Allow: / Crawl-delay: 10 # --- Anthropic (Claude) --- # 2026 census (support.claude.com/en/articles/8896518): ClaudeBot = training, # Claude-SearchBot = search indexing, Claude-User = user-initiated fetch. # Claude-Web and anthropic-ai are deprecated tokens (fleet consolidated into # ClaudeBot, July 2024) — groups kept as belt-and-suspenders; no legitimate # traffic is expected under those strings. Anthropic honors Crawl-delay. User-agent: ClaudeBot Allow: / Crawl-delay: 10 User-agent: Claude-SearchBot Allow: / Crawl-delay: 10 User-agent: Claude-User Allow: / Crawl-delay: 10 User-agent: Claude-Web Allow: / Crawl-delay: 10 User-agent: anthropic-ai Allow: / Crawl-delay: 10 # --- Google AI (Gemini) --- User-agent: Google-Extended Allow: / Crawl-delay: 5 # --- Meta AI (Facebook/Instagram) --- User-agent: FacebookBot Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: meta-externalagent Allow: / # --- Apple Intelligence --- User-agent: Applebot-Extended Allow: / # --- Perplexity AI --- User-agent: PerplexityBot Allow: / Crawl-delay: 10 # Perplexity-User: user-initiated fetch, referral traffic → allowed # (docs.perplexity.ai/docs/resources/perplexity-crawlers; note Perplexity # documents that user-initiated agents generally ignore robots.txt — the # Allow group is consent documentation, not an enforcement mechanism) User-agent: Perplexity-User Allow: / Crawl-delay: 10 # --- Amazon Alexa --- User-agent: Amazonbot Allow: / # --- Cohere AI --- User-agent: cohere-ai Allow: / Crawl-delay: 10 # --- You.com --- User-agent: YouBot Allow: / # --- Common Crawl (CAUTIOUS) --- # Used by many AI companies - Allow with restrictions User-agent: CCBot Allow: / Disallow: /privacy.html Disallow: /terms.html Disallow: /*/privacy.html Disallow: /*/terms.html Crawl-delay: 20 # ============================================================================ # AI CRAWLERS - BLOCKED (No Strategic Value) # ============================================================================ # --- ByteDance/TikTok (Undisclosed AI usage) --- User-agent: Bytespider Disallow: / # --- Huawei (Undisclosed AI usage) --- User-agent: PetalBot Disallow: / # --- Commercial Scraping (No value to platform) --- User-agent: DataForSeoBot Disallow: / # ============================================================================ # SEO TOOL CRAWLERS - ALLOWED # ============================================================================ # Strategy: Full competitive intelligence access for market positioning # --- Ahrefs --- User-agent: AhrefsBot Allow: / # --- Semrush --- User-agent: SemrushBot Allow: / # --- Majestic --- User-agent: MJ12bot Allow: / # --- Moz --- User-agent: DotBot Allow: / # --- BLEXBot --- User-agent: BLEXBot Allow: / # --- MegaIndex --- User-agent: MegaIndex Allow: / # --- Linkpad --- User-agent: LinkpadBot Allow: / # --- AspiegelBot --- User-agent: AspiegelBot Allow: / # ============================================================================ # SOCIAL MEDIA CRAWLERS - ALLOWED # ============================================================================ # Essential for link previews and social sharing # --- Facebook --- User-agent: facebookexternalhit Allow: / # --- Twitter/X --- User-agent: Twitterbot Allow: / # --- LinkedIn --- User-agent: LinkedInBot Allow: / # --- Discord --- User-agent: Discordbot Allow: / # --- Telegram --- User-agent: TelegramBot Allow: / # --- WhatsApp --- User-agent: WhatsApp Allow: / # ============================================================================ # BAD BOTS - AGGRESSIVE BLOCKING # ============================================================================ # --- Site Copiers / Offline Browsers --- User-agent: HTTrack Disallow: / User-agent: WebCopier Disallow: / User-agent: WebStripper Disallow: / User-agent: WebZIP Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: Offline Explorer Disallow: / # --- Command-line Tools (Recursive) --- User-agent: wget Disallow: / User-agent: curl Disallow: / # --- Site Downloaders --- User-agent: SiteSnagger Disallow: / User-agent: Fetch Disallow: / User-agent: PageGrabber Disallow: / User-agent: WebReaper Disallow: / # --- Link Checkers (Excessive) --- User-agent: Xenu Disallow: / User-agent: LinkChecker Crawl-delay: 60 # --- Email Harvesters --- User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf Disallow: / User-agent: ExtractorPro Disallow: / # --- Spam Bots --- User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: NPBot Disallow: / User-agent: k2spider Disallow: / User-agent: grub-client Disallow: / User-agent: libwww Disallow: / User-agent: larbin Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / # --- Ad Crawlers --- User-agent: Mediapartners-Google Disallow: / # --- Additional Malicious Bots --- User-agent: Sosospider Disallow: / User-agent: WebLeacher Disallow: / User-agent: WebSauger Disallow: / User-agent: Website Quester Disallow: / User-agent: WebStripper Disallow: / User-agent: WebWhacker Disallow: / User-agent: Express WebPictures Disallow: / User-agent: BackDoorBot Disallow: / User-agent: Bandit Disallow: / User-agent: BatchFTP Disallow: / User-agent: BlackWidow Disallow: / User-agent: Bot mailto:craftbot@yahoo.com Disallow: / User-agent: BullsEye Disallow: / User-agent: CheeseBot Disallow: / User-agent: CherryPicker Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: cosmos Disallow: / User-agent: Crescent Disallow: / User-agent: Custo Disallow: / User-agent: DISCo Disallow: / User-agent: DIIbot Disallow: / User-agent: DittoSpyder Disallow: / User-agent: EirGrabber Disallow: / User-agent: EroCrawler Disallow: / User-agent: FlashGet Disallow: / User-agent: GetRight Disallow: / User-agent: GetWeb! Disallow: / User-agent: Go!Zilla Disallow: / User-agent: Go-Ahead-Got-It Disallow: / User-agent: GrabNet Disallow: / User-agent: Grafula Disallow: / User-agent: HMView Disallow: / User-agent: Image Stripper Disallow: / User-agent: Image Sucker Disallow: / User-agent: Indy Library Disallow: / User-agent: InterGET Disallow: / User-agent: Internet Ninja Disallow: / User-agent: JetCar Disallow: / User-agent: JOC Web Spider Disallow: / User-agent: Mass Downloader Disallow: / User-agent: Memo Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: MIDown tool Disallow: / User-agent: Mister PiX Disallow: / User-agent: Navroad Disallow: / User-agent: NearSite Disallow: / User-agent: NetAnts Disallow: / User-agent: NetSpider Disallow: / User-agent: Net Vampire Disallow: / User-agent: NetZIP Disallow: / User-agent: Octopus Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Offline Navigator Disallow: / User-agent: PageGrabber Disallow: / User-agent: Papa Foto Disallow: / User-agent: pavuk Disallow: / User-agent: pcBrowser Disallow: / User-agent: RealDownload Disallow: / User-agent: ReGet Disallow: / User-agent: SiteSnagger Disallow: / User-agent: SmartDownload Disallow: / User-agent: SuperBot Disallow: / User-agent: SuperHTTP Disallow: / User-agent: Surfbot Disallow: / User-agent: tAkeOut Disallow: / User-agent: Teleport Pro Disallow: / User-agent: VoidEYE Disallow: / User-agent: Web Image Collector Disallow: / User-agent: Web Sucker Disallow: / User-agent: WebAuto Disallow: / User-agent: WebBandit Disallow: / User-agent: WebCopier Disallow: / User-agent: WebFetch Disallow: / User-agent: WebGo IS Disallow: / User-agent: WebReaper Disallow: / User-agent: WebSauger Disallow: / User-agent: Website eXtractor Disallow: / User-agent: Webster Pro Disallow: / User-agent: WebStripper Disallow: / User-agent: WebWhacker Disallow: / User-agent: WebZIP Disallow: / User-agent: Widow Disallow: / User-agent: WWWOFFLE Disallow: / User-agent: Xaldon WebSpider Disallow: / User-agent: Zeus Disallow: / # ============================================================================ # END OF ROBOTS.TXT # ============================================================================ # # Contact: support@jynx.app # # ============================================================================