Very nice. I've found a few state seal buttons. (Found another yesterday, in fact.) The backmark is important if you want to date it. Some state seal buttons were made during and after the civil war. Worn by militias, veteran groups, military school/academy cadets, etc...
That definitely does not rule out civil war era. (There were many other makers of state seal buttons that didn't even start until the 1870s or so.) You've got a great location apparently!