Robin Sloan, “Where is it like to be a language model?”:
Do not mistake this assessment for the deflation that goes: Oh, these things are just next-token predictors. I mean … there’s no “just” about it. Like waving a hand & saying nuclear fission is just uranium atoms releasing some neutrons: technically true … but … BOOM!
And:
I suggest that if you want an honest answer from a language model — the thing itself — you should pose your question such that it can be answered with a single token; answered, that is, by a single flashing forward pass that really exists, however briefly, somewhere on Earth.
Wow.