A friend asked me what I thought about this article from Vern Poythress: Artificially Intelligent. I wrote back:
Thank you for sharing the article. I really respect Dr. Poythress and I like a lot of the article. I don’t think he is exactly right in the two sections titled “The Ugly” and “Value of the Products”. In these, he claims that the model AI has of the world was imparted by human programmers. On the contrary, with regard to large language models and the newer reasoning models, human programmers are simply imparting a way to learn, a way to model reality. The model itself emerges from the data of the world (including data in which reasoning about the world is exemplified) in a way similar to how humans build a picture of the world through sensory and intellectual input. I agree that this is a simulation of human intelligence, but that doesn’t entail that it isn’t also a new kind of intelligence that isn’t human but rather something else entirely. If the opaque machinery inside of Searle’s room (Poythress mentions the Chinese Room thought experiment from the philosopher Searle) always produces the output that an intelligent speaker of Chinese would produce, then there is a functional parity between the room and a human speaker of Chinese. It is the “no true scotsman” fallacy to define intelligence in a way that computers can never meet the definition, even analogously.