I use 170mm cranks on my road bikes and 175mm cranks on my gravel bikes. There is a bigger Q difference, my gravel bike has a very wide crank on it. I can't notice the change between bikes in this regard. My right knee is sensitive to seat height, too low and it swells up to grapefruit size. But I can't imagine 2.5mm making that much difference.
The only thing I would note is that setback wouldn't change with crank length. If you move it back 2.5mm, then your foot is 5mm further forward on the downstroke.