Bpod Freezes after Set Amount of Time

[Adam]

Hello,

I am encountering a bug where Bpod freezes in the middle of a protocol: valves do not open/close, the trial does not advance, and the Bpod Console displays a single state with no transitions. This happens on only one of two Bpod machines that I have, despite the fact that they are running the exact same protocol. (They are, however, connected to different hardware, as I describe below.) I have to Ctrl+C in order to end the program, and when I do, the following error message is printed:

Code:

Operation terminated by user during ArCOMObject_Bpod/write (line 295)

In SendStateMachine (line 333)
                       BpodSystem.SerialPort.write(ByteString(i), 'uint8'); % send byte-wise, to avoid
                       HW timer overrun

In DistributionalRL_6Odours (line 236)
   SendStateMachine(sma); % With TrialManager, you can send the next trial's state machine while the
   current trial is ongoing

In run (line 91)
evalin('caller', strcat(script, ';'));

In NewLaunchManager>LaunchProtocol (line 748)
run(ProtocolPath);

Interrupt while evaluating DestroyedObject Callback.

It appears that Matlab is failing to write information to the serial port, but I have no idea why that might be. This Bpod receives TTLs from a microscope in order to perform synchronization. It also sends a voltage out to trigger the microscope. Interestingly, the problem appears worse when the microscope is running (and Bpod is receiving TTLs). In this case, Bpod almost always freezes after exactly 31 trials. (If I run a slightly different protocol, it becomes ~30 trials, but a similar amount of time and TTLs.) However, it is worth noting that every so often (perhaps 10% of attempted runs?), the program does run successfully until completion, even when the microscope is also running. When the microscope is not running, Bpod generally runs longer without errors, and tends to have a higher success rate, but nonetheless still sometimes fails in the same manner, after around 80 trials. The other Bpod, which runs without error, is not connected to a microscope and never receives TTLs, but is otherwise configured in exactly the same way.

Perhaps related to this issue, I sometimes get a different error on the same Bpod State Machine:

Code:

Error using ArCOMObject_Bpod/read (line 359)
Error: The USB serial port did not return the requested number of bytes.

Error in TrialManagerObject/startTrial (line 89)
           SMA_Confirmed = BpodSystem.SerialPort.read(1, 'uint8');

Error in DistributionalRL_6Odours (line 189)
   TrialManager.startTrial(); % Start next trial's state machine (call with no argument since it was
   already sent)

Error in run (line 91)
evalin('caller', strcat(script, ';'));

Error in NewLaunchManager>LaunchProtocol (line 748)
run(ProtocolPath);

359                 error('Error: The USB serial port did not return the requested number of bytes.')

Here, the problem is again with serial communication between Matlab and Bpod, but at the read step. I suspect these are related somehow, but I don't know in what way.

I have taken a variety of steps to debug (some of which are suggested in this thread), but none of them have been successful. They have been:

1. Re-installing the Bpod software
2. Re-uploading the firmware (v22). Verified that upload was successful through the USB native port.
3. Changing out the USB cable from Bpod to computer
4. Powering Bpod with a separate 12V DC adapter, as opposed to just the USB
5. Disabling unused inputs in the Bpod Console

I would greatly appreciate any help you could provide!

Specs:

Bpod State Machine version 0.8
Latest Gen2 Bpod software
Bpod firmware version 22
Matlab 2019b 
Inspiron 3847 desktop with an Intel i7 processor and 16 GB of RAM
64-bit Windows 10 Pro OS
PsychToolbox is installed
Bpod is communicating with an 8-channel valve driver module and a port interface + valve
Only port 1 is selected in Bpod Console > Settings > Ports, corresponding to the lick sensor input
Other USB input is disabled in Bpod Console > USB module configuration menu
Only other USB devices connected to the computer are standard keyboard, mouse, and speakers

[Josh]

Hi Adam,

Some labs have notified us of a similar issue when using TrialManager, when the state machine receives bursts of events to relay to the PC.
We have a provisional fix. If you don't mind, please try the software on the 'develop' branch of the repository, here.
When you update, please remember to remove all existing Bpod_Gen2 folders from the MATLAB path before adding the folder from the develop branch.

If this fixes the issue, please let us know. It should be part of our next major release (on startup, your software will notify you when it is ready). Otherwise, it sounds like it will take some more in-depth debugging. Please contact support@sanworks.io and we'll set up a remote desktop session.

Thanks,
-Josh

[Adam]

Thank you so much for the quick response! I pulled the 'develop' branch of the repository and things worked flawlessly today, including for one imaging session, so we are very optimistic about this fix. We will test things more extensively over the coming weeks and let you know if this resolves our issues for good.

[Josh]

Thanks Adam!

[Adam]

So I have good news and bad news.

The good news is that the new software appears to partially resolve the problem: I get freezes less frequently, and power cycling the Bpod is often sufficient to get things to work normally again.

The bad news is that the fix is not 100%. I sometimes get both of the errors I included in my first post, and this sometimes happens even immediately after power cycling the Bpod.

I realize that the apparent stochasticity of this error makes it far more difficult to debug, and I apologize for not being able to give you a better description of the circumstances under which Bpod fails. The best I can do is to assure you that I have tried investigating this in as systematic a manner as possible, and I have come up empty. I will get in touch with support@sanworks.io, but I just wanted to keep this issue open in case other users encounter similar problems.

[Bowey]

Hi Adam,

Some labs have notified us of a similar issue when using TrialManager, when the state machine receives bursts of events to relay to the PC.
We have a provisional fix. If you don't mind, please try the software on the 'develop' branch of the repository, here.
When you update, please remember to remove all existing Bpod_Gen2 folders from the MATLAB path before adding the folder from the develop branch.

If this fixes the issue, please let us know. It should be part of our next major release (on startup, your software will notify you when it is ready). Otherwise, it sounds like it will take some more in-depth debugging. Please contact support@sanworks.io and we'll set up a remote desktop session.

Thanks,
-Josh

Hi Josh,

I'm not sure my issue is the same one. Basically, I connected touch sensors via Bpod shield to the Modules port. During running my protocol, whenever there's a burst of continuous touch events, the state machine will freeze at that state for a long period, sometimes it even messes up the timer of that state or state transitions.

Is there a way to fix this?

Best,
Bowey

[Josh]

Hi Bowey,

I replied in this thread. Let's continue there.

Thanks,
Josh

[Adam]

Thanks to Josh, we figured out that the problem was a user-written plotting function that runs during the trial. Sometimes that function took longer to run than the length of a trial (particularly when many events were received, hence the issues when the microscope was running). When this happens, it crashes the code in a rather opaque way. Moral of the story: especially when using the TrialManager object, make sure that any user functions run reasonably quickly and finish before your trial is over!

[Josh]

Hi Adam,

Thanks for the follow up message!
In the latest version of the Bpod_Gen2 repository, the system will print a warning to the console if it detects that user code takes too long to run.

-Josh