Can the answer just be "it depends"**********
I came got super into cycling as an adult in NYC. After my initial shock of learning to ride in the city I started to listen to music via ear buds commuting to work through Brooklyn and Manhattan. I never felt super in danger but there were a few times I wasn't paying attention and it's also illegal there to have 2 ear buds in while riding. So I switched to a JBL clip on speaker. i loved it. Even then, sometimes I feel like listening to music, sometimes not.
Point is - for me it depends on what kind of riding I'm doing. I live in CT now. If I'm out on a quite gravel trail I probably don't want to listen to anything. But I might bring my speaker just in case. I usually like to just listen to the natural sounds of nothing but sometimes I just like that extra motivation. If I do listen to music out in nature, on a shared trail etc, I try to keep the volume reasonable as to not blast out anyone I might come across.
In the city, if I'm mashing around running errands, commuting to work or whatever on loud, busy city streets, I'm more likely to use my clip on speaker to blast some tunes and jam out. No one in NYC or Hartford is going to complain my bike speaker is too loud while I'm pulling up next to them on a bike. Just like anything, try to measure your safety with the safety and sanity of the folks around you and have a good time while not being a complete *******.