Check out beckcycling.com. Based out of like the peachtree city area, but we have a good bit of cat3 and 4s that have fun and race hard. I'm over in Smyrna though and ride to douglasville often. Swing down and do a couple group rides or meet up with us at some races and see if you'd fit in!
Round Here Racing also seems to have a good bit of cat 3s and 4s, don't know too much about them though.
If you don't know about either of those, then just keep doing the group rides and see you enjoy hanging out with and guys that seem strong.