Yeah, LLMs would need to be wildly constrained to a world model. I remember in some LLM text game that was supposed to be medieval dungeon crawl, I just declared that I pulled a shotgun from my bag and blasted the goblins, and it just rolled with it instead of saying 'no.'