Personally I would just use an adapter. But if you want to cut and solder, why not just install a 3.5 mm jack wired in parallel with the existing jack? Then you could use either size, and you wouldn't have to deal with the 1/4 inch hole. I guess the switched conductor would be wired in series so that the speaker would cut off if there was a plug in either jack. But it shouldn't be hard to do.