> people could probably buy just assets they like and expect an equivalent return to everyone else

That's pretty much true. Although risk and volatility does differ from asset to asset. So as a lone investor you can decide how much you are willing to risk to get better returns.