You see the irony right? I genuinely can’t fathom your intent when telling this story, but it is an absolutely stellar example.

Yes, I did mean for it to be an example.

And yes, I do think that correctly framing a question is crucial whether you’re dealing with a person or an LLM. But I was elaborating on whether a person’s process of answering a question is fundamentally similar to an LLM’s process. And this is one way that it’s noticeably different. A person will size up who is asking, what they’re asking, and how they’re asking it… and consider whether they should actually answer the exact question that was asked or suggest a better question instead.

You can certainly work around it, as the asker, but it does require deliberate disambiguation. I think programmers are used to doing that, so it may feel like not that big of a deal, but if you start paying attention to how often people are tossing around half-formed questions or statements and just expecting the recipient to fill in the gaps… It’s basically 100% of the time.

We’re fundamentally social creatures first, and intelligent creatures second. (Or third, or not at all, depending.) We think better as groups. If you give 10 individuals a set of difficult questions, they’ll bomb almost all of them. If you give the questions to a group of 10, they’ll get almost all of them right. (There’s several You Are Not So Smart episodes on this, but the main one is 111.)

Asking a question to an LLM is just completely different from asking a person. We’re not optimized for correctly filling out scantron sheets as individuals, we’re optimized for brainstorming ideas and pruning them as a group.