Have you pulled the Log file from the session and interrogated it to see if you have ANT+ /BLE disconnects or excessive network traffic or Wi-Fi dropouts. I think the Wi-Fi can disconnect and reconnect in a few seconds and not have all the riders disappear. I agree with one of the other posters that I this could be a network or connection problem.
I think Zwift insider has a log file analyzer tool if your uncomfortable navigating the log file.