Like most hobbies, there are many different reasons to bike - that's why there are so many different forums on BikeForums!
Most of my riding hours are not training hours, I don't do racing at all and don't have a power meter on any of my bikes.. But (in normal years...) I'll often try to do some structured training aimed at some long ride (like the 2 day Seattle to Portland 209 mile ride I did in 2019) or a century ride, etc. Then, junk miles are either too short to count as my long ride or too low of power (indoor trainer on Zwift) or heart rate zone.
Those junk rides are still fun and still burn pretty close to the same calories/mile (not per minute) as the "non-junk" miles. On those rides where I did the structured training (pretty much following the Time Crunched Cyclist plans) I really did see improvement - which made the long rides more fun.