No the usb fault leds shouldn’t be on, my bet is there is something wrong with the sd card. It seems to me like you’re having a similar issue to this post RFSoC 4x2 Does not show any IP address in the Display - #10 by kulla002, the author of that post had a screenshot with the usb fault leds on as well, reflashing the sd card solved their issue.
The fact that you managed to get a “test failed” on the OLED the first time you booted the board with the shipped sd card image means the board is certainly functional enough to boot. What method are you using to flash your sd card?
Thanks
Shawn