Skull caps and bandanas will change the helmet fit a bit. I often wear bandanas to soak up sweat and usually need to fiddle with the position of the knot and my ponytail knot to get the helmet seated and adjusted correctly. It's a bit easier with the POC because the harness and dial sit a bit lower on my skull.
On days when it's not too hot or humid I might go without a bandana and the helmets usually fit a bit better.