I don't wear gloves either. Florida dirt, is really just sand, with a little decayed organic matter mixed in. I'm not in any rush, the sandy soil and a pinpointer, really no reason to be blindly groping around, where I might get poked, scratch, or cut. It's a hobby, I just take my time, enjoy the day. I've got heavy calluses on my hands and fingers from work. Seldom get a cut or splinter at work, that draws blood, and those wood pallets can be in bad shape.
I'll admit, I'm not Mr. Safety, use the bare minimum of safety gear. Only recently got into the habit of using a seat belt while driving. Not so much to keep me safe, in an accident, but because they change the law a few years ago. It's a primary offense, and a $116 ticket (last time I got popped). Never thought seat belts make you a safer driver, or that they are even 50% effective all the time. Better to stay focused on the road, and your driving, and avoid accidents in the first place, minimize the impact if possible.
I should get a picture of one of my neighbors, mowing his lawn sometime. Now he is Mr. Safety, to the max. Doesn't have a big yard, probably spends more time dressing for the task, than it actually takes to mow. From the combat boots, to the safety goggles, and dust mask. It gets hot in Florida, I usually wear sunglasses, shorts, and flip-flops. This guy is covered completely, long pants, long sleeves, heavy material too, and gloves. Guess he's lucky not to be mowing my yard, the heat would kill him, before he got half done. I usually take a cool down break, between the front and back.
Think safety gear really depends on the individual, and the task. Think most of the time, the provide a false sense of security, which causes people to be more careless, take shortcuts, get in a rush. Gloves will reduce a lot of injuries, but won't stop everything, all the time. Just be aware there is a lot of trash and nasty things in the ground, and work cautiously.