If you either pull or drive out the bearings, do it by the outer race. As you noted, pulling on the inner race will not be good for the bearings (read: will ruin). Only do this if you have replacements ready to install. Also, seat the new bearings by pressing on the outer race.