I have two bikes with dyno lights. The first one I used the on-off switch religiously, until I was surprised to get over the ridge leaving town and found out I'd left it on. I never noticed the difference. Now I just leave it on. The second has a light sensor/switch, and I let the light do the switching.
How much difference in speed? Pretty close to zero, certainly negligible in my riding.