Hmm... yes, you're right. It should have been exactly as you said.
The funny thing is my current project has a few plugins and fabfilter limiter on master channel. When I put 1024 samples I saw 50 ms latency in settings and it's really noticeable when I was playing. Then I turned on LLM (by the way, none of my plugins are disabled) and it felt like exactly the same 50 ms (give or take)... hehe. With buffer size 32-64-128 I didn't notice that behaviour. I didn't change anything - sample rate and so on.
P.S. this question is for only "academic purpose", I am just curious why this happened.