Haven’t tried this but according to the Trenz schematics they’ve kindly wired up a couple PMODs to the ZU3EG HD I/O pins that can go to the PL. This could allow say a PMOD camera to push image data into the PL 
Now just so you know: PYNQ for Ultra96 would have to be rebuilt as currently some of those pins are setup to the 96Boards “standard” to provide UARTs etc on the LS connector. But with Xilinx you can setup pins to do different things. You would use the Vivado block designer, remove the current PL UART, change the pin functionalities and then export to tcl, rebuild enough to get the hdf to then ingest that into the existing BSP for v1 or v2 and rebuild PYNQ. Some work but gives you a way to not have to run camera through the PS and you don’t have to mess with MIPI.
This is the adapter:
https://wiki.trenz-electronic.de/display/PD/TEP0006+TRM