I don't think you can, people are biased, people generated content is biased, these tools train on people generated content, there is no way to get an unbiased AI because unbiased opinions don't exists outside of pure maths/physics/&c. You'll never get an unbiased opinion about politics, or music, or culture
I'm not sure you can "fix it". If you slice your data and look at enough possible bias types I think it's impossible for highly dimensional data to be even in all possible comparison pairs.
I don't think we can "fix" bias but I think we can and should strive to reduce it.
Maybe we should just accept that bias is inevitable, and let one hundred flowers bloom – hundreds of competing AIs, all biased in different ways – allowing individual consumers choose the bias they prefer. This is just a temporary situation caused by this being a very new technology, with high entry costs, and still only one real player in the market. That will all change – one day, we'll each have our own personal ChatGPT-equivalent, personally trained to reflect our own personal biases.
It is like media – all media is biased, but there are so many different media outlets (especially online), biased in so many different ways, almost certainly you can find some whose bias appeals to you.
> I'm not sure how you fix it
I don't think you can, people are biased, people generated content is biased, these tools train on people generated content, there is no way to get an unbiased AI because unbiased opinions don't exists outside of pure maths/physics/&c. You'll never get an unbiased opinion about politics, or music, or culture