OK new update.
Further looking to the project, the latency inducing plugin is on the track stack of vocal track. So I guess that is considered as an aux track. If I put the plugin on the actual audio track, there is no latency. Sorry for the confusion I was not in front of the project.
But, I just checked the same setup with Cubase (All vocals routed to a group track which is an aux track in Cubase and plugin is inserted there does not cause any live midi playing latency issues like Logic. I also tried it on Ableton, no problem there either.
So I guess Logic is different how it handles the latency? Yet I do not understand why it would create latency since my instrument track is going directly to the master track without any plugins or routings.
Thank You.