I've gotten some feedback and some people have recommended Lestoil.
Rapha emailed me back and just suggested using warmer water which I'm a little wary of doing.
Crankycrank I have read some people suggest Lavasoap but that has pumice.
AnkleWork Interesting idea that it's entrapped particles. I'll give "dissolving" it back in a neutral oil a try.
More suggestions always welcome.
Thanks for all the help so far.