Check a URL before fetching it: is it alive, final URL after redirects, HTTP status, co…
Check a URL before fetching it: is it alive, final URL after redirects, HTTP status, content type, HTML title, and whether robots.txt allows your user-agent. Link checke…