If you are adding loads to your bars that are anywhere close to the loads put on by a rider near the bike's rider weight limit, you will have more trouble just riding than with durability.
Yours is kind of like the oft-asked question 'does it damage a bike to hang it upside down' by the wheel - if the wheel cannot support the weight of the unladen bike, I wouldn't dare try to load my carcass and luggage around on it.