I usually wear a buff as a toque when working in the yard, during the summer - really helps keep the sweat out of my eyes.

When it's cooler, I wear a wool driving cap. I put it over my toque, and it was much less clammy. (Like most wool hats, my driving cap has a synthetic liner of some kind - seems similar to the liner of a sport coat.)

Usually, I take the liner out of my hats/put in a cut down silk bandana/might buy a silk liner, for when it's cold.

It seems like a wicking fabric liner/maybe linen would be a good choice for a hat when it's too cold for straw, but a bit warm for silk.