Gravity always wins...
Since wasn't familiar with the Lezyne Road Caddy, I took a quick peek thru google.
Most saddle bags have a strong taper from back to front, and often attach/secure to the SP.
The Road Caddy seems quite 'square' and is only secured by the one strap around the rails. Between the force of gravity and the narrowing of rails down to where they do their work, the bag will inevitably slide down (and somewhat forward). Bag is prolly wider than the saddle skirt bend, so will rub on legs and scuff up the shorts lycra... no good.
SO, how to resist the downward slide... nothing easy, light and not involved nor ugly comes to mind...
You could 'shape' some light material to fit between the rails and also resist the strap from moving forward/down, over the blocking material. ONly thing light I can immediately think if is Styrofoam - Ugly and surely will not last very long...
I would get a saddlebag with a proper taper to fit under the saddle, without extra width and will hold what you need to carry.
Road Caddy seems well intended but not very well designed nor tested out...
I know... a lot of words, without any real solutions.... sometimes it goes that way... LOL!
Ride On
Yuri