Single click should work.
Even if you turn off the badges in grid view, they should still appear on the image in the filmstrip unless you've turned them off there as well. I can see clicking on them by accident in the film strip but unless you've got your grid thumbnail size set to be very small it should be pretty difficult to click the badge by accident in grid view.
I often keep the filmstrip pretty small so that the screen gets as much space as possible for the image. I found myself clicking on the badges in the filmstrip rather than the photo, so the badges went...
I think what I'd like to see is a Right click on the filmstrip give me a "Map" menu list entry that I can either go to the Map module or replace the filmstrip contents with everything in the database within 100' or 0.5 miles or 1 mile or... regardless of time so that if I'm editing one image, it is easy for me to bring up all of the other images that I've taken close by to see what other settings and perspectives I've used.