The only junk miles are when you force yourself to ride when you should be recovering. Easy rides are NOT junk miles. You don't need to push it at a certain level to have gains, every mile you logged is like money in the bank, unless you're doing it when your body truly needs resting.
Check out this video, it makes a great point on how many of us (I use to be one of them) cyclists, runners push it way too hard, during all of our rides. I still do speed work and good hard rides, but I also learned that I need to take it easy on much of my rides.