Here to say once again that Screaming Frog SEO is an amazing nuclear-powered chainsaw of a tool for web work.
Its out of box config is geared towards SEO optimization (naturally, given the name) — making sure all your pages have decent metadata, stuff like that. But learning how it does what it does and configuring it to your needs turns it into something much, much more.
Most CMSs leave some default CSS classes or IDs in their markup to indicate template or content types used when rendering a page — Screaming Frog lets you run custom regexes during its crawl to extract those, generating a pre-categorized list of what content type each page is.
Tie it to a Google API account, and you can pull in PageSpeed Insights data — what’s the time to first interactive, say, on every URL on our site?
Link it to your GA account and see if that correlates to bounce rate…
Running a large-scale content audit and trying to get an idea of just how ugly the markup in old pages will be? Use an XPath to count the number of times “red flag” tags appear inside the main page body, per page.
Want to wire it all up with the rest of your systems? Yeah, it’s got a command line interface.