If the button is civil war era or prior, the fiber is certainly natural. IOW, cotton or wool or some such. In fact, it does look like wool to me.
And that raises a question: HOW does natural fiber survive for 150 yrs in eastern US soil conditions? It seems highly improbable.
Organic material can be naturally preserved for centuries and even millenniae in extremely dry climates such as desert, extremely cold climates (which also happen to be extremely dry DUE to the cold) such as the antarctic and siberia, or in anaerobic (which tends to be naturally antibiotic) environments such as peat bogs. The eastern US tends towards being moderately wet, temperate and biotically aggressive, so organic material breaks down very quickly. It's true that keratins (wool is keratin, cotton is not) last the longest of organic material, but I wouldn't expect even that to last more than a couple of years in the ground.
The fiber in your photo also appears rather "clean" and almost fluffy, as opposed to the flat and matted appearance one might expect from decomposing natural fiber.
So my wild guess is that it's a modern artificial fiber. Perhaps a metal button from a woman's sweater that was lost only a few years back.
