Ah, I may have assembled it incorrectly. I just read a post about how there is a large and a small bearing race, and the size distinction is not obvious. I thought they all were the same. Tear down time again.
Just got it done, and yep, that was it. I had one set of races reversed. It's perfect now.