I have taken shots of the same subject, mostly landscapes, wth the camera on a tripod with and without the filters. I blow them up in Photoshop using a 21" LCD and I can't see any difference in sharpness. So in my own informal tests, I decided to leave the filters on all of the time.
Have you checked for contrast degradation, which is one other issues one would expect to pop up?