You’ve fallen victim to propaganda, my friend.

It’s only mandatory if they survive the fall…

There’s no fullproof way. Even if you somehow block every crawling automation, there’s still puppeteering where the bot behaves just like a normal user.

As someone who works for a paywallled website, that’s hardly a deterrent. If the site is important enough, they will pay for accounts and crawl until the server melts

The problem is not the functionality, it’s the business model.