Hi Sandra,
The rotary encoder firmware must be uploaded to the rotary encoder module, not the state machine.
Make sure you select the rotary encoder module's COM port in the Arduino application when uploading the firmware.
In older versions of Arduino / Teensyduino, there was a bug that would sometimes load the firmware to the incorrect device, even if the correct COM port was selected. If you suspect that this is happening, unplug the other Bpod devices from the PC and load to the target device.
To fix the state machine you have, upload state machine firmware to the state machine.
I hope this helps!