I tend to wear lycra when I am wearing bike specific shorts. If going riding for the sake of riding that is what I will have on. Otherwise, for a short ride to run and errand or something. I might just wear what I have on. That is typically Carhartt shorts or pants which tend to be pretty long wearing without any added panels. They are okay to ride to the local store or whatever. I haven't noticed that I ruin them by riding, but that is probably at least partly because I don't do a lot of miles wearing them. Probably about 99% of my riding is with lycra shorts or bibs.
On tour I might ride short distances after I am out of the bike shorts. In that case I am likely to be in running shorts.