An easy way to see this is to put a block of wood on the non-cable attach side between the brake pad and the rim. You are making one side immovable. Pull the brake lever. The cable will move the arm towards the rim and the other arm will not move. Reverse it and the housing will move the arm and the cable side will just sit there.
John