Thanks for writing in
Unfortunately I'm not up to date on the latest VR packages. ViRMEn is the one I've heard most about. I'm not sure what its specs are, but with a good multi-core processor I'd imagine the MATLAB processes for the two instances will be on separate cores, so there should not be much interference in theory.
The rotary encoder is where you may run into issues. If you're initializing it in the the ViRMEn, you could initialize the RotaryEncoderModule class and use it to set thresholds, but the data wouldn't automatically propagate to ViRMEn. You can set the userCallbackFcn property of a RotaryEncoderModule object to call other MATLAB functions each time new data arrives from the encoder, so that may be one path to integration - though you'd probably have more latency than if you used ViRMEn's internal interface.
If you're not restricted to MATLAB, BonVision is a good option.
TCP/IP on localhost shouldn't be much of an issue. Bpod_Gen2 includes TCPCom, a class that wraps PsychToolbox's Pnet interface and should do most of what you need. I haven't updated it to use MATLAB's newer tcpclient and tcpserver interfaces, but it's on my ToDo list. If you don't have PsychToolbox you may be able to identify the relevant functions and plug them in to TCPCom, or build your own code using these built-in classes.
Please let me know if you run into any issues setting this up, I'd be glad to assist!
Josh