What You Saw: robots.txt File Entries

The Disallow Entries:

User-agent: *
Disallow: /admin
Disallow: /wpadmin
Disallow: /backup
Disallow: /config
Disallow: /.git
Disallow: /api/v1/
Disallow: /test

These are GOLD for pentesters! They tell you exactly where the juicy stuff might be!

What robots.txt Does

Purpose: Tells search engines what NOT to index Reality: Shows attackers where to look!

Red Flag Directories in robots.txt:

# ADMIN PANELS - High value targets!
Disallow: /admin
Disallow: /administrator  
Disallow: /wpadmin        # WordPress admin
Disallow: /wp-admin       # Also WordPress
Disallow: /cpanel
Disallow: /manager        # Tomcat manager

# SENSITIVE DATA
Disallow: /backup         # Backup files!
Disallow: /backups
Disallow: /db            # Database dumps?
Disallow: /sql
Disallow: /config        # Configuration files
Disallow: /.env          # Environment variables!

# VERSION CONTROL - CRITICAL!
Disallow: /.git          # Git repository exposed!
Disallow: /.svn          # Subversion
Disallow: /.hg           # Mercurial

# API ENDPOINTS
Disallow: /api/
Disallow: /api/v1/
Disallow: /api/v2/
Disallow: /graphql

# DEVELOPMENT/TEST
Disallow: /dev
Disallow: /test
Disallow: /staging
Disallow: /demo
Disallow: /tmp