Programming
Cron Generator
Build crontab schedules visually instead of recalling the field syntax.
Programming
Robots.txt Generator
Build a robots.txt that tells crawlers what to index and what to skip.
Tool Panel
Processed locallyAbout Robots.txt Generator
Configures allow and disallow rules per user-agent group, with an optional sitemap line and a full preview before copying.
- 1
It writes exactly the directives you list: `User-agent` / `Allow` / `Disallow` and an optional `Sitemap` — no guessing at rules for you.
- 2
Allow lines come first in the output, but **order does not affect the meaning**: the most specific path wins, and Allow wins ties. That is why `Disallow: /` plus `Allow: /public/` is how you block everything except one directory.
- 3
One `User-agent` block per run; for different rules per crawler, generate several blocks and concatenate them (each starting with its own `User-agent` line).
- 1
Set the user agent
`*` (the default) covers every crawler; name one (like `Googlebot`) to target it specifically.
- 2
One Allow / Disallow per line
Paths start at `/`; blank lines are ignored.
- 3
Add the Sitemap and copy
Use an absolute URL (`https://…/sitemap.xml`), then save the text as robots.txt in the site root.
The most specific path wins, and Allow wins ties — **regardless of the order you wrote them in**. That is the most common misconception here, and one reason people see blocked paths being crawled anyway. The other reason: robots.txt is a **convention**, not access control. A crawler that ignores it is not stopped by it, so real restrictions belong behind authentication.
RFC 9309 and the mainstream crawlers (Google, Bing) support them: `*` matches any characters and `$` anchors the end (as in `/*.pdf$`). Smaller crawlers may ignore them, so never treat wildcards as a security boundary. Also note `Sitemap` is not part of the robots specification at all — it is a widely honoured convention, and search engines do read it.
Related tools
View allProgramming
Gitignore Generator
Produce a .gitignore suited to your stack instead of copying one from memory.
Programming
Programming
JWT Generator
Mint signed JSON Web Tokens for testing authentication flows.
Programming
Programming
UUID Generator
Generate random v4 UUIDs in bulk for fixtures and test data.
Programming
Programming
NanoID Generator
Generate compact, URL-safe random identifiers.
Programming
Programming
ULID Generator
Generate sortable unique IDs that preserve creation order.
Programming
Programming
Regex Tester
Test and debug regular expressions with live match highlighting.
Programming
Programming
Hash Generator
Compute MD5, SHA-1, SHA-256 and SHA-512 digests of any text.
Programming