# LIVE (UAE site). Holdback lifted 2026-08-27. # # The sitewide is off: # src/lib/noindex.ts now has NOINDEX_DEFAULT = false, so robotsContent() returns # each page's own frontmatter value again instead of overriding it. # # Crawling was always allowed here even during the holdback, deliberately: the # block was the meta tag, and a crawler has to fetch a page to see it. # "Disallow: /" would have been weaker, because it hides the noindex and a # merely-disallowed URL can still surface in results as a bare link. # # The Sitemap line is restored now that the pages are indexable. It points at # sitemap-index.xml, which @astrojs/sitemap generates alongside sitemap-0.xml. # # Content signals (contentsignals.org). These express how this site's # publisher expects its content to be used. They are preferences, declared in # the form the AI Preferences work at the IETF is converging on. # # search=yes build a search index and link back to the source. # ai-input=yes use the content to ground an AI answer, with attribution. # ai-train=no do not use it to train or fine-tune a model. # # search and ai-input are yes deliberately: this site wants to be found and # cited by assistants. Training is withheld. # /landing-redirect/ is deliberately NOT disallowed, and must not be again. # The interstitial already carries noindex, nofollow in its own head, and # every link to it carries rel="sponsored nofollow". A Disallow stops # Googlebot FETCHING the page, so that noindex is never read: Google still # finds the URLs through the links and indexes them URL only, which is the # Search Console report "Indexed, though blocked by robots.txt". A crawl # rule cannot remove a URL from the index; only a noindex Google is allowed # to read can. User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Amazonbot User-agent: Bytespider User-agent: anthropic-ai User-agent: cohere-ai User-agent: cohere-training-data-crawler Disallow: / Sitemap: https://www.compareforexbrokers.ae/sitemap-index.xml # Agent discovery # https://www.compareforexbrokers.ae/.well-known/api-catalog # https://www.compareforexbrokers.ae/.well-known/ai-catalog.json # https://www.compareforexbrokers.ae/.well-known/agent-skills/index.json