So you are running Jellyfin on Pi4, and playing h265 content. The main culprit will be Transcoding, if your playback device does not support h265.
Pis don't have a powerful GPU, so even 1080p transcoding is too much.
Playing on VLC over the network, could be speed issue, or could be Pi not able...