The sidewall rubber only covers the threads, which give the tire its strength. As long as the threads are intact, I think the tire is safe.
Rubber is quite weak compared to the structural components of the tire, the beads and threads. Rubber only supplies traction and in some cases cosmetic finish.