How to obtain video frame time stamps

nmtimme

Hello

I am recording mouse behavior with a Flir Chameleon 3 camera that I control via Matlab from within my bpod protocol. I've figured out how to start the camera recording at the start of the session via a softcode and the following snippet of code:

vidObjFLIR = videoinput('gentl', FLIRCameraID);
    vidObjFLIR.FramesPerTrigger = inf; % Continuous acquisition
    vidObjFLIR.LoggingMode = 'disk&memory'; % Log data to disk
    vidObjFLIR.FramesAcquiredFcnCount = 1;
    vidObjFLIR.FramesAcquiredFcn = {'framesAcquiredTimeStamp'};
    DiskLogger = VideoWriter(strcat(videoDir,filesep,'video'), 'Motion JPEG AVI');
    DiskLogger.Quality = 50;
    vidObjFLIR.DiskLogger = DiskLogger;
    start(vidObjFLIR)

Furthermore, I have figured out how to record time stamps from the frame grabs in matlab via the frame acquire callback:

function framesAcquiredTimeStamp(obj, event)
   [~, ts] = getdata(obj);
   obj.UserData = [obj.UserData;ts];
end

Is there a good way to link these time stamps to the global clock used by bpod so I can relate these frame grab time stamps to other bpod events?

I've found these resources helpful regarding frame acquisition callback functions:
https://www.mathworks.com/matlabcentral/answers/750774-how-can-i-get-accurate-timestamps-for-frames-acquired-while-writing-to-disk
https://www.mathworks.com/help/imaq/creating-and-executing-callback-functions.html
https://www.mathworks.com/help/imaq/creating-and-executing-callback-functions.html#f6-130859

I have also figured out how to record frame grab TTLs from the camera using the bpod, but the camera produces TTLs prior to starting the recording using the code above. I haven't been able to figure out a good way to align the TTLs with the frames in the recorded video.

I'd appreciate any help with this issue. I've looked around online and I haven't found a solution. However, if I missed something, I would appreciate a point in the right direction. Thank you!

_Nick

Josh

Hello Nick,

The Chameleon3 has a GPIO interface. With FLIR Flycap software, you can configure the camera to output a TTL whenever a frame is grabbed. You can patch the TTL into one of the state machine's digital input channels for a timestamp matching each frame acquired. IBL's rig setup docs explain this configuration in detail.

Josh

nmtimme

Hi @Josh ,

Thanks for that information! Unfortunately, I think there are some issues with the IBL approach that I'm hoping can be overcome. I am able to receive TTLs from the Chameleon3 camera via the GPIO interface and record them in the bpod. However, the camera produces the TTLs as long as it is powered. So, the TTLs start showing up in Matlab prior to starting the recording and I have no way of knowing which TTL corresponds to the first frame. (It can probably be narrowed down to a few frames, but I'm not sure how long it takes from Matlab telling the camera to start recording to it actually recording. Probably a few frames are dropped.)

There is another option with the Chameleon3, which is to include timestamps as pixel values in the images in a small ribbon of pixels in the upper left corner. I've messed around with this, but the problem is that if you perform any kind of compression, it corrupts the pixel values and the timestamps.

I'm hoping there is another option using the framesAcquiredFcn technique I described above along with the clock that the bpod uses to record events.

I'd appreciate any other advice you might be able to provide. Thanks!

_Nick

Josh

Hi Nick,

Instead of encoding the timestamps in a ribbon of pixels, I've seen some labs put a deep red or infrared LED (driven by a port interface board) in the camera's field of view. When you set up your state machine description for a trial, set the LED to be on in every state. This way, Bpod's trial-start and trial-end timestamps will match video frames where the LED turns on and off respectively. If you're using TrialManager, you don't have dead-time between trials, so you'd need to use a pair of LEDs that alternate with each trial.

A second possibility: I know that the camera can be TTL-triggered. If you can configure it to start and stop on sequential rising TTL edges, you can use any digital output from the state machine to start acquisition when the first trial starts, and to end it when the last trial ends.

Please let me know if these work for you!

Josh

nmtimme

Hi @Josh ,

Yeah, we've used the IR LED method in the past too, but I'd like to get away from it. You need to make sure it is in the frame (which is hard if you add more cameras) and it can change the illumination of the animal. Also, it takes some amount of time for the LED to turn on, which I've always worried might throw things off. (Though, I haven't verified that this is a problem.) We've got all the cameras hooked up to the computer, so we should be able to do it electronically! We might have to back to it if I can't find a good solution.

Regarding TTL triggering, that is a possibility. However, as far as I've been able to find in the camera documentation, I think I'd need to trigger every frame. Would there be an easy way to send out well timed TTLs from the bpod 30 times a second?

Based on this suggestion (https://teledynevisionsolutions.zendesk.com/hc/en-us/community/posts/29794044048269-How-to-align-Chameleon-3-TTLs-with-Matlab-recording), I was able to find a way to start image capture with the matlab call to start image acquisition by putting the camera into trigger mode using FlyCapture2. This method seems a bit janky. (I'm basically making the camera wait for a voltage trigger, then having matlab change its mode when image acquisition starts). Furthermore, when I tried it out, extra TTLs were arriving from the camera that weren't recorded in the video. (For instance, 6 extra TTLs over a ₁₅₀₀ frame video.) So, there is some slop in frames being acquired (and producing TTLs) and actually being recorded by matlab at the start, end or both of the video. Also, the frame rate can get messed up in ways I can't really explain (TTLs arriving at 90 Hz, but images acquired at 30 Hz, and only sometimes). So, though this looked promising a few hours ago, I don't think this method will work well.

I was able to make some progress using the FrameAcquiredFcn method. I found here (https://www.mathworks.com/help/imaq/retrieving-timing-information.html) about how to record the absolute time of the frame grabs in Matlab. Unfortunately, this has led to a new error, which I posted here (https://www.mathworks.com/matlabcentral/answers/2150864-sporadic-error-with-framesacquiredfcn-and-getdata). As I mention in that post, there's a janky work around for this problem, but I'm hoping someone has a good answer.

Assuming I can get that to work and I am able to obtain absolute time recordings for each frame, I can make this method work if I can get an absolute time for the bpod clock. Is there a way to find the absolute time for when the bpod clock starts running, or is there some other way to relate the event times recorded by the bpod to the absolute time?

As always, thanks for all your help!

_Nick

Josh

Hi Nick,

During a trial you can use a looping global timer linked to a TTL output to generate a 30Hz square waveform, as shown in this example. You'd set SendGlobalTimerEvents to 1, to have matching Bpod timestamps in your data. Unfortunately this method will not work in the dead-time between trials, and even with BpodTrialManager sampling would be interrupted by trial-end.

If you can trigger the start of camera acquisition from MATLAB, one option would be to use a USB soft code handler to have the state machine trigger MATLAB to start the camera in the first trial of your session. The camera would then output TTLs with each frame, and you'd be able to align because you'd capture the first one.

I hope this helps,
Josh

JulianHinz

The most straight forward way is to not have the camera running free, but to send out a 30 - 60Hz trigger signal with a 50% duty cycle and trigger every frame individually. Since the lag of digital high to aquisition is in the us range it should be good for all applications in question.