# The Composable Video Pipeline **URL:** \ **Category:** Announcements\ **Created:** [June 16, 2021, 6:52pm UTC](https://discuss.pynq.io/t/the-composable-video-pipeline/2758 "2021-06-16T18:52:16Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![marioruiz](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/marioruiz/32/2177_2.png) [@marioruiz](https://discuss.pynq.io/u/marioruiz)\ **Post date:** [June 16, 2021, 6:52pm UTC](https://discuss.pynq.io/t/the-composable-video-pipeline/2758/1 "2021-06-16T18:52:16Z")
Hello PYNQ aficionados, **Edit 10 March 2022** : comments closed to request support open a new post in the [support category](https://discuss.pynq.io/c/support/30). I am pleased to announce that the PYNQ team has just released the composable (video) pipeline, this is an open source project and you can find it in GitHub [PYNQ\_Composable\_Pipeline](https://github.com/Xilinx/PYNQ_Composable_Pipeline) Note that through the blog, I will use the term “composable overlay” and “composable pipeline” interchangeably. ## What is a composable overlay? A composable overlay provides run time configuration as well as runtime hardware composability. More in composability in a moment. Our composable overlay is based on three pillars: 1. An AXI4-Stream Switch that provides the runtime hardware composability, defining how data flow from one IP core to another. 2. [DFX](https://www.xilinx.com/products/design-tools/vivado/implementation/dynamic-function-exchange.html) regions that brings new functionality to the design at runtime. 3. An API, built on top of `pynq`, that exposes the functionality to control the composable pipeline in a pythonic way. The AXI4-Stream Switch is cornerstone in achieving composability, its runtime configuration allows us to modify our design without redesigning the overlay, hence being more productive. This is the key of the composable pipeline. DFX regions are optional in a composable pipeline, however, having them brings an extra dimension of flexibility making the applicability of a composable overlay broader. For instance, not all application need the same functions or some functions are rarely used. ## The composable video pipeline To demonstrate the benefits of the composable overlay, we are providing a composable video pipeline that you can use out-of-the-box. An overview of the composable video pipeline is shown in the image below. ![cv-4pr](https://us1.discourse-cdn.com/flex019/uploads/pynq1/original/2X/7/74e5da02b580b621bc2981374c09cc7720aa3b05.png) This version implements several standard vision functions. The most common functions are implemented in the static region, these account for 6 functions. The composable overlay also provides 12 dynamic functions implemented across 4 DFX regions, note that pr\_0 and pr\_1 provide pairs of functions. | Static | | --- | | colorthresholding | | filter2d | | gray2rgb | | lut | | rgb2gray | | rgb2hsv | | Dynamic | DFX Region | | --- | --- | | absdiff | pr\_join | | add | pr\_join | | bitwise\_add | pr\_join | | cornerHarris | pr\_1 | | dilate | pr\_0 & pr\_1 | | duplicate | pr\_fork | | erode | pr\_0 & pr\_1 | | fast | pr\_0 | | fifo | pr\_0 & pr\_1 | | filter2d | pr\_0 | | rgb2xyz | pr\_fork | | subtract | pr\_join | The python API provides a nice visualization of the available functions in the `.c_dict` attribute ![c_dict](https://us1.discourse-cdn.com/flex019/uploads/pynq1/original/2X/1/121517fd093064e102294ff7b23241783dfc6864.png) Filtering by loaded `.c_dict.loaded` and unloaded `.c_dict.loaded` functions is also supported. The `.loadIP()` method allows you to load a function or list of functions. For instance, `cpipe.loadIP([cpipe.pr_1.dilate_accel, cpipe.pr_0.filter2d])` ## How to get started? If you have one of the supported boards (PYNQ-Z2 or PYNQ-ZU), burn a new SD card with its corresponding 2.6.0 image. Once you power on your board and the system boots, open Jupyter Lab (`http:///lab`) and then open a terminal ![jupyter-terminal](https://us1.discourse-cdn.com/flex019/uploads/pynq1/original/2X/9/946f7cba0db44c35757f29739746e29200e4e8db.png) Copy and paste the following commands in the terminal ```sh git clone https://github.com/Xilinx/PYNQ_Composable_Pipeline pip install PYNQ_Composable_Pipeline/ pynq-get-notebooks composable-pipeline -p $PYNQ_JUPYTER_NOTEBOOKS ``` You can see how `pynq` is updated to the latest stable version (2.6.2), and the composable video pipeline gets installed in the folder `composable-pipeline`. This process takes approximately 10 minutes ![jupyter-notebooks](https://us1.discourse-cdn.com/flex019/uploads/pynq1/original/2X/d/de7a7263d932b959c7860fee94945bcf557e6c03.png) ## Applications and Custom Pipeline The `composable-pipeline` folder contains two notebooks at the top level, one that describes how to setup the board and another that describes how some of the vision function operates (examples using OpenCV). ![composable-pipeline-folders](https://us1.discourse-cdn.com/flex019/uploads/pynq1/original/2X/a/a776afd7ebc30a27c8e0c1943a1de2cec429076c.png) ### Applications The `application` folder contains five pre-built applications, 3 of them can be controlled with [`ipywidgets`](https://ipywidgets.readthedocs.io/en/latest/) and the remaining two with the switches and buttons present in the board. The image below shows one of the applications, the API at the application level is straightforward but powerful enough to provide run time configuration. ![application](https://us1.discourse-cdn.com/flex019/uploads/pynq1/original/2X/1/1afc890f19cd78a0dce9af4d49b725e51ffc7ad7.png) ### Custom pipeline The `custom_pipeline` folder provides several notebooks that gradually introduce the API until you get familiar with it and build an application. There is a bonus notebook with advanced features! For instance, to compose a pipeline you will write ```python video_pipeline = [video_in_in, rgb2gray, filter2d, colorthr, gray2rgb, video_in_out] cpipe.compose(video_pipeline) ``` You can visualize what has been composed by calling the `.graph` attribute: ![pipeline](https://us1.discourse-cdn.com/flex019/uploads/pynq1/original/2X/e/ea4daf7109a4939248bae3ba6dd75d364dfe165b.png) ## Final remarks The composable overlay unleashes the benefits of the programmable logic while providing an unprecedented flexibility at the user level. The provided composable video pipeline has been carefully optimized to support three applications: - Difference of Gaussians - Corner Detect - Color Detect However, it is up to your imagination what you can compose! I love to hear your feedback and features request for future releases.This is an open source project, contributions are welcome. Hope you enjoy composing! 🎶 ---
**Author:** ![Msk\_Nak](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/msk_nak/32/2201_2.png) [@Msk\_Nak](https://discuss.pynq.io/u/Msk_Nak)\ **Post date:** [July 5, 2021, 6:16am UTC](https://discuss.pynq.io/t/the-composable-video-pipeline/2758/2 "2021-07-05T06:16:41Z")
I think the device is the same for both pynq-z1 and z2, but does it work with z1? ---
**Author:** ![marioruiz](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/marioruiz/32/2177_2.png) [@marioruiz](https://discuss.pynq.io/u/marioruiz)\ **Post date:** [July 5, 2021, 7:33am UTC](https://discuss.pynq.io/t/the-composable-video-pipeline/2758/3 "2021-07-05T07:33:33Z")
@Msk_Nak, I have not tested it. But, I believe that it should work. Once you clone the repo, edit this line with PYNQ-Z1 and proceed with the installation > ---
**Author:** ![Msk\_Nak](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/msk_nak/32/2201_2.png) [@Msk\_Nak](https://discuss.pynq.io/u/Msk_Nak)\ **Post date:** [July 5, 2021, 9:17am UTC](https://discuss.pynq.io/t/the-composable-video-pipeline/2758/4 "2021-07-05T09:17:55Z")
Thank you for your information.I applied that fix. Pynq-Z2-\> Pynq-Z1. However, I got an error when checking board\_folder. So, next, I copied board / Pynq-Z2 to create board / Pynq-Z1. After that, I continued the install work and it was successful. I have not confirmed the operation of the device yet, so I will confirm it in the future. ---
**Author:** ![Msk\_Nak](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/msk_nak/32/2201_2.png) [@Msk\_Nak](https://discuss.pynq.io/u/Msk_Nak)\ **Post date:** [July 5, 2021, 9:22am UTC](https://discuss.pynq.io/t/the-composable-video-pipeline/2758/5 "2021-07-05T09:22:48Z")
Thanks for your reply. OK ,I try it. 2021年7月5日(月) 16:43 Mario Ruiz via PYNQ \<[pynq1@discoursemail.com](mailto:pynq1@discoursemail.com)\>: ---
**Author:** ![Msk\_Nak](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/msk_nak/32/2201_2.png) [@Msk\_Nak](https://discuss.pynq.io/u/Msk_Nak)\ **Post date:** [July 6, 2021, 6:34am UTC](https://discuss.pynq.io/t/the-composable-video-pipeline/2758/6 "2021-07-06T06:34:15Z")
After completing the installation of PYNQ\_Composable\_Pipeline, there was an error running the notebook. > /usr/local/lib/python3.6/dist-packages/composable\_pipeline/composable.py in **init** (self, bitfile\_name, dtbo, download, ignore\_version, device) > 299 [‘C\_NUM\_MI\_SLOTS’]) > 300 > → 301 predefpaths = \_default\_paths[self.device.name] > 302 > 303 if len(switch) != 0: > KeyError: ‘Pynq-Z1’ I tried changing the environment variable on the board to “Pynq-Z2”, but it didn’t work. I thought it was because I didn’t have the “Pynq-Z1” key in “\_default\_paths” in “composable.py”. So, I uninstalled “composable-pipeline” once and copied “Pynq-Z2” in “\_default\_paths” of “composable.py” as “Pynq-Z1”. > \_default\_paths = { > ‘Pynq-Z1’: { > 0: {‘ci’ : 0, ‘pi’ : 0}, > 1: {‘ci’ : 1, ‘pi’ : 1} > }, > ‘Pynq-Z2’: { > 0: {‘ci’ : 0, ‘pi’ : 0}, > 1: {‘ci’ : 1, ‘pi’ : 1} > }, After fixing it, I installed it again. ・sudo python3 -m pip install PYNQ\_Composable\_Pipeline/ I ran “notebook” again and it was successful. It doesn’t work well if the HDMI source is 1080p, but there is no problem if it is 720p. Thank you for your advice. ---
**Author:** ![marioruiz](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/marioruiz/32/2177_2.png) [@marioruiz](https://discuss.pynq.io/u/marioruiz)\ **Post date:** [July 6, 2021, 8:14am UTC](https://discuss.pynq.io/t/the-composable-video-pipeline/2758/7 "2021-07-06T08:14:55Z")
@Msk_Nak thank you for confirming that the design works on the PYNQ-Z1. I’ll incorporate the changes into the repo, so it is easier to install it for PYNQ-Z1. The recommended resolution for the Zynq 7000 devices is 720p, support for 1080p is in the limit of the capabilities of these devices. Mario ---
**Author:** ![Msk\_Nak](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/msk_nak/32/2201_2.png) [@Msk\_Nak](https://discuss.pynq.io/u/Msk_Nak)\ **Post date:** [July 7, 2021, 2:17am UTC](https://discuss.pynq.io/t/the-composable-video-pipeline/2758/8 "2021-07-07T02:17:52Z")
Thank you for supporting Pynq-Z1. > The recommended resolution for the Zynq 7000 devices is 720p, support for 1080p is in the limit of the capabilities of these devices. I understood that. Thank you very much. ---
**Author:** ![Space\_Zealot](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/space_zealot/32/2353_2.png) [@Space\_Zealot](https://discuss.pynq.io/u/Space_Zealot)\ **Post date:** [September 8, 2021, 12:48pm UTC](https://discuss.pynq.io/t/the-composable-video-pipeline/2758/9 "2021-09-08T12:48:31Z")
Did my thesis work provide any inspiration for this? [https://scholarsarchive.byu.edu/etd/8620/](https://scholarsarchive.byu.edu/etd/8620/) It does seem to differ with better DFX, HLS library, and PYNQ python support. ---
**Author:** ![marioruiz](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/marioruiz/32/2177_2.png) [@marioruiz](https://discuss.pynq.io/u/marioruiz)\ **Post date:** [September 8, 2021, 1:10pm UTC](https://discuss.pynq.io/t/the-composable-video-pipeline/2758/10 "2021-09-08T13:10:04Z")
Hi @Space_Zealot, The idea of composable overlay is very similar to your thesis, but I was not aware of your work until now. Thank you for letting us know. 😀 Mario ---
**Author:** ![Space\_Zealot](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/space_zealot/32/2353_2.png) [@Space\_Zealot](https://discuss.pynq.io/u/Space_Zealot)\ **Post date:** [September 8, 2021, 3:33pm UTC](https://discuss.pynq.io/t/the-composable-video-pipeline/2758/11 "2021-09-08T15:33:46Z")
Well if you have any questions on the approach I tried, feel free to ask 😄 HW SRC: [GitHub - byuccl/BYU\_PYNQ\_PR\_Video\_Pipeline\_Hardware: BYU Pynq PR Video Pipeline Hardware](https://github.com/byuccl/BYU_PYNQ_PR_Video_Pipeline_Hardware) SW SRC: [GitHub - byuccl/BYU\_PYNQ\_PR\_Video\_Pipeline: The Demo that was presented at FCCM.](https://github.com/byuccl/BYU_PYNQ_PR_Video_Pipeline) ---
**Author:** ![Space\_Zealot](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/space_zealot/32/2353_2.png) [@Space\_Zealot](https://discuss.pynq.io/u/Space_Zealot)\ **Post date:** [September 8, 2021, 3:37pm UTC](https://discuss.pynq.io/t/the-composable-video-pipeline/2758/12 "2021-09-08T15:37:37Z")
Some videos of it: Demo: [PYNQ HLS Video Demo - YouTube](https://youtu.be/uyBZFCYEg3w) In-Depth: [PYNQ Video HLS and Dynamic PR (Retry) - YouTube](https://youtu.be/X3rv-vkDV1Q) ---
**Author:** ![cathalmccabe](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/cathalmccabe/32/3348_2.png) [@cathalmccabe](https://discuss.pynq.io/u/cathalmccabe)\ **Post date:** [December 6, 2021, 9:56am UTC](https://discuss.pynq.io/t/the-composable-video-pipeline/2758/13 "2021-12-06T09:56:43Z")
3 posts were split to a new topic: [Composable Overlay v2.7](https://discuss.pynq.io/t/composable-overlay-v2-7/3389) ---
**Author:** ![cathalmccabe](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/cathalmccabe/32/3348_2.png) [@cathalmccabe](https://discuss.pynq.io/u/cathalmccabe)\ **Post date:** [December 7, 2021, 10:49am UTC](https://discuss.pynq.io/t/the-composable-video-pipeline/2758/14 "2021-12-07T10:49:28Z")
A post was split to a new topic: [Composable Overlay question](https://discuss.pynq.io/t/composable-overlay-question/3400) ---
**Author:** ![marioruiz](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/marioruiz/32/2177_2.png) [@marioruiz](https://discuss.pynq.io/u/marioruiz)\ **Post date:** [March 10, 2022, 10:51am UTC](https://discuss.pynq.io/t/the-composable-video-pipeline/2758/15 "2022-03-10T10:51:52Z")
3 posts were split to a new topic: [Composable Overlay not working on PYNQ-ZU](https://discuss.pynq.io/t/composable-overlay-not-working-on-pynq-zu/3821) ---
**Author:** ![marioruiz](https://sea1.discourse-cdn.com/flex019/user_avatar/discuss.pynq.io/marioruiz/32/2177_2.png) [@marioruiz](https://discuss.pynq.io/u/marioruiz)\ **Post date:** [March 10, 2022, 10:57am UTC](https://discuss.pynq.io/t/the-composable-video-pipeline/2758/16 "2022-03-10T10:57:57Z")