// Generate robots.txt files with custom rules per user-agent. Control crawler access with Allow/Disallow paths.