I just upgraded to Ventura 13.5.2 and had similar latency issues as described above in the thread. I am on a 2019 Intel Mac Pro. The small buffer range helped some, however it did not solve some hiccups in some of my more complex projects.
After a ton of testing, I figured out that Brainworx plugins were causing some of the issues. Updating all of my bx plugins seems to have solved my issues with Ventura and Logic. Plugin Alliance did a PA major update to a large group of their plugs recently and it seems it is required for Logic to work smoothly on Ventura macOS. Hope this helps some others.