I find it hard to visualise pictures but I am good at imagining how things relate to each other in space and how they move. For example, if I try to imagine a scene of somebody playing on a swing hanging from the branch of a tree, if I focus hard I can ‘see’ parts of it; the rough, frayed rope, the look of joy on the kid’s face, or whatever, but only one at a time. But I can easily imagine how the swing moves, how the rider leans back or forward to make it go higher. I don’t need to ‘see’ the image for this, it’s more abstract.

LLMs could theoretically give a game a lot more flexibility, by responding dynamically to player actions and creating custom dialogue, etc. but, as you say, it would work best as a module within an existing framework.

I bet some of the big game dev companies are already experimenting with this, and in a few years (maybe a decade considering how long it takes to develop a AAA title these days) we will see RPGs with NPCs you can actually chat with, which remain in-character, and respond to what you do. Of course that would probably mean API calls to the publisher’s server where the custom models are run, with all of the downsides that entails.