I considered something similar recently.
My CPA told me I would need to wait 2 years before breaking ground (something about the primary/homestead rule only applying for that amount of time), and I would need to be capable of proving that my original intent wasn't to use the land for my primary residence. His suggestion was to look into subdividing. That way, if the IRS ever asked, I could show them the plans and simply say that it fell through.
Sounded like a lot of work. That, plus the impending increase on capital gains tax, was enough for me to decide to cash out instead.