Consider New England, where in its early days most inhabitants consisted of farming families. Where there were farms and cow pastures, there is now mostly forest.

There are a few farms remaining in New England, yes, but in general farmers consider the soil there not worth farming compared to places like Iowa and Southern Illinois (where all the land really is utilized for farming) even though in the past most the (sizeable) population of New England made a living farming.

We know that most of the forest in New England is suitable for farming because there are still stone walls running through it: these wall were made of stones encountered by farmers while plowing (when the forest was farmland).