Yeah mine get funky also! If you look at the black plastic bite piece from inside the cap you'll see formed "wedges" that engage with the plastic bottle cap. You can push in the wedges and pop off the bite piece.
I did this to two bottles, one's been fine the other has started leaking.