The optical path to the PDAF meter is not changed by removing the OLPF filter, but the optical path to the sensor is, so AF calibrated on the basis that on OLPF is present will be slightly inaccurate if that OLPF is not present.
how so ? silicone surface is not getting closer to the mount if you remove AA filter... it is not like you are trying to shoot IR images...