This happened to me too. I eliminated it in text responses, but eventually figured out that there's a system prompt in voice mode that says to do this (more or less) regardless of any instructions you give to the contrary. Attempting to override it will just make it increasingly awkward and obvious.