I wear layers. Merino wool base layer. Insulating layer and then outer shell for wind and rain or snow.
You'll keep yourself warm as long as you are pedalling so you might find yourself unzipping to let the heat out. But if you are stopped for a long time, you'll start freezing.
I don't like fleece. Too much static. I like wool. Go to a thrift store and look for wool cardigans or sweaters.