I've started a couple dozen rides between 4-4:30am. It's a great hour around here since it gives enough time to take the shortest routes out of town on state highways and trunk roads that are unrideable even at 7-8am. So I don't see any issue, your logic is sound and the earlier start time sounds like it'll work great for what you want to do.