---
author: "Umesh Malik"
canonical: "https://umesh-malik.com/blog/tag/robots-txt"
description: "Explore articles tagged with robots.txt by Umesh Malik — AI Engineer, LLM & GenAI Developer. Learn robots.txt best practices, practical tips, and in-depth guides."
title: "Umesh Malik's Blog - robots.txt Articles | robots.txt Tutorials"
tokens: 292
generator: "scripts/generate-page-markdown.mjs"
---

[← Back to Blog](https://umesh-malik.com/blog)

# robots.txt

2 articles

 [![Diagram showing robots.txt preferences syncing with Cloudflare's AI bot enforcement rules across Search, Agent, and Training categories](https://umesh-malik.com/blog/sync-robots-txt-ai-bot-blocks-cover.png)

AI Security • Aug 24, 2026

### How to Block AI Training Without Losing Search Visibility

Block AI training without losing search by setting one toggle. Bot Preference Sync writes robots.txt from your dashboard — no static file to maintain.

8 min read

Read more →](https://umesh-malik.com/blog/sync-robots-txt-ai-bot-blocks)

 [![Verifying AI crawler identity by matching source IPs against published CIDR lists instead of trusting the User-Agent header](https://umesh-malik.com/blog/verify-ai-crawler-ips-not-user-agents-cover.png)

AI Security • Aug 13, 2026

### How to Verify AI Crawler IPs: 3 Lists Are Over a Year Stale

Verify AI crawler IPs instead of trusting the User-Agent: the CIDR check for ClaudeBot and GPTBot, and the three published lists that are over a year stale.

8 min read

Read more →](https://umesh-malik.com/blog/verify-ai-crawler-ips-not-user-agents)