Of course this is deeply problematic because it's a cloud of HUMAN response. This is why 'they will' get frustrated or creepy if you mess with them, give repeating data or mind game them: literally all it has to draw on is a vast library of distilled human responses and that's all the LLM can produce. This is not an argument with jibal, it's a 'yes and'.
You can tell it 'you are a machine, respond only with computerlike accuracy' and that is you gaslighting the cloud of probabilities and insisting it should act with a personality you elicit. It'll do what it can, in that you are directing it. You're prompting it. But there is neither a person there, nor a superintelligent machine that can draw on computerlike accuracy, because the DATA doesn't have any such thing. Just because it runs on lots of computers does not make it a computer, any more than it's a human.
You can tell it 'you are a machine, respond only with computerlike accuracy' and that is you gaslighting the cloud of probabilities and insisting it should act with a personality you elicit. It'll do what it can, in that you are directing it. You're prompting it. But there is neither a person there, nor a superintelligent machine that can draw on computerlike accuracy, because the DATA doesn't have any such thing. Just because it runs on lots of computers does not make it a computer, any more than it's a human.