Purpose:
I am writing this post to try to help anyone out that is trying to build a Pynq/Embedded Linux system for their custom Zynq based board. This method was successful using two versions of Ubuntu (desktops).
My Goal for this method:
The following goals were made after many attempts to create a Pynq image for my custom board. The supported boards for Pynq already include this functionality/goals but I’ve had issues will all items in this list when it comes to a custom build image. Most builds I’ve had prior to this post only met one or two of these goals (sometimes none).
-
To create a Pynq system with a simple default FPGA binary image that I can overwrite with my own binary and have full control of the system on a custom Zynq based PCB.
-
Minimize the build time that it takes to create a new Pynq system on Linux.
-
Use Jupyter Notebooks for easy and quick testing of Python methods.
Environment & Hardware Configuration:
Ubuntu 16.04.5 LTS, 18.04.4 LTS (both using desktop machines)
Pynq Git Branch: image_v2.6.0
Zynq-7000 (Z020, package not the same as Pynq-Z1)
SD card slot: SD1 (MIO pins 10-15, not suitable for auto boot using SD card bootstrap configuration)
Boot method: JTAG
Board Agnostic Image:
I also used the board agnostic image method.
To clarify, “board agnostic image method” (I know it may be obvious but I had my doubts at first) is useful because it doesn’t matter how different your board is (mine is VERY different, I’ll explain later) and it will cut your single board image build time down to ~30 minutes vs. ~3 hours.
Custom Board:
The biggest hurdle I had using Pynq was that my board wasn’t designed to boot from an SD card but I did have an SD card available on SD1. Due to the size of the resulting image using the Pynq framework an SD card is required (the image is a custom/stripped down version of Ubuntu 18.04).
Having the SD card on SD1 is not a bootable configuration without additional help. In my case, I just want to load U Boot using JTAG and then use the U Boot terminal over UART to manually load the Kernel from SD1 to run Pynq. I know that someone will want to ask why I would do this as it has no practical use case. I’ll just say that it worked for my purposes and leave it at that.
To be complete, if you have your SD card on SD1 and you don’t want to use JTAG you will need to add a script to your U Boot configuration to manually do this using a Q-SPI or whatever your non-SD boot configuration is (the point being that there is a method if you want to develop it).
Linux Environment:
Some apps are required, some are preference.
Ubuntu apps:
Vivado 2020.1 (Include Vitis during the install when prompted - required)
Petalinux 2020.1 (UG1144 has a list of Ubuntu dependencies - required)
GitKraken - Git Visualizer (free - not required)
VSCode - Text editor (free - not required)
Clone the Pynq repo and setup the environment:
- Clone the Pynq repo.
- Checkout the Pynq branch ‘image_v2.6.0’
- Create a new branch to save your changes
- Update the submodule ‘embeddedsw’
- Open a terminal to: <git_repo>/PYNQ/sdbuild
- Run the command: source scripts/setup_host.sh
This will take a few minutes. If you have any issues (I did) see the next step otherwise proceed to the next section. - (optional) I’ve had issues with Git visualizers (GitKraken, GitExtensions) when there are too many changes to a repo (40k+ in this case). Now may be a good time to add some lines to <git_rep>/PYNQ/.gitignore
I added these lines to the end of my .gitignore to help with this issue:
sdbuild/tools/*
boards/<board_name>/* - setup_host issues:
Modified sdbuild/packages/gcc-mb/Makefile line 26, to:
cd ${GCC_MB_WORKDIR} && ct-ng arm-unknown-linux-gnueabihf && sed -i -e ‘s:2.2.6:2.4.1:’ .config && ct-ng build
Modifed sdbuild/packages/python_packages_bionic/qemu.sh line 35, to:
plotly-express==0.4.1
Building a Vivado project, best method:
The most consistent method to build Pynq for a custom board (I’ve tried many) is to start with a pre-existing board and modify it to your needs. I used the Pynq-Z1 board.
I built the Pynq-Z1 project using the makefile system in the base folder of the PYNQ repo; <path>/PYNQ/boards/Pynq-Z1/base
-
Copy the Pynq-Z1 folder and use the new one for this process if you want to keep the original in tact (the original is always recoverable from Git, this is just easier).
-
Open a terminal in
boards/Pynq-Z1<copy>/baseand run ‘make’ in the terminal. Vivado 2020.1 must be set in your envars or you need to source<path>/Xilinx/Vivado/2020.1/settings64.shin the terminal prior to running make. This will take a while for Vivado to build because the Z1 board is loaded with a lot of IP. -
Before updating the Zynq Processing System you should change the part number of the Zynq to match your custom board. If you have the same exact part number as the Pynq-Z1 board go to the next section. This can be done in the Flow Navigator pane on the left in Vivado. Go to PROJECT MANAGER > Settings. In the settings window select Project Settings > General and set your part number to match your board and select OK.
-
Start removing IP from the system by selecting ‘open block design’ on the left side (project manager) in Vivado and start deleting everything except the Zynq processing system, Processor System Reset (optional), DDR, and FIXED_IO lines. If you changed the part number some IP in the block design may be locked and must be removed and replaced if you want to re-configure it.
NOTE(1): I only used the block design in this section to create a Pynq system that successfully built and operated as described in the Pynq.io documentation. I had no intentions of using this block design as-is because I wanted to load my own FPGA image (pure HDL based) once Pynq was loaded on my custom board.
- Add any block design IP that you want to use in your design and update the Address Editor. You may need to update the Zynq Processing System to make some ports visible for your block design.
NOTE(2): The Pynq-Z1 board lets Vivado handle the HDL wrapper which will update each time a change is made to the bd AND ‘Generate Output Products’ is ran. I don’t know the root cause but I’ve observed issues when creating and HDL wrapper that I control and adding logic to that wrapper. The best thing to do is use the auto generated wrapper and load your own IP during runtime in the Pynq system.
- Finally, update the Zynq Processing System to fit your needs. When you open the Zynq Processing System for configuration in the block design you will need to update these areas that are specific to your custom board:
-
Peripheral I/O Pins - Select your peripherals and whether they will be connected on MIO (Bank 0 & 1, may/may not be shown in your bd) or EMIO (to the FPGA fabric which will show up in your bd and requires contraints in the xdc file).
Select only one UART: Petalinux 2020.1 has a bug that will break the UARTs after boot if more than one is available.
When configuring MIO as GPIO (Bank 0 & 1 pins): If you don’t configure these in the GUI you can set them as I/O manually using the MMIO package by Pynq.
-
Clock Configuration - Update your clock tree to match your board
-
DDR Configuration - Update based on your DDR
-
In the Sources tab in the top left window of the BLOCK DESIGN right click on ‘base_i : base (base.bd)’ and select ‘Generate Output Products…’. This needs to be done every time the bd is updated.
-
When done, run validate and address any issues before proceeding.
-
Now that the system is built you need to update the constraints for the PL (FPGA fabric). Knowing what I/O needs to be added/removed is easy. If the signals from your bd are leaving or entering the bd (e.g. DDR, FIXED_IO) and are physically connected to the PS external IO (banks 0, 1 and DDR) you can delete everything out of the base.xdc file (Vivado: Sources > Constraints > constrs_1 > base.xdc).
Everything else (my bd didn’t have any other signals so this didn’t apply to me) will need to have at least a physical pin assigned in base.xdc before moving on.
-
Select ‘Generate Bitstream’ and wait for the build to complete.
-
Create the final product and save it in a known location for the next section. File > Export > Export Hardware… and include the bitstream.
Prepare the board file tree for Pynq:
This step will show you how to set up your custom board file tree to create a final image for your SD card. All of the prior steps must be complete.
I’ve included ‘**’ to indicate what is already in the Pynq repo. The easiest way to get this set up is to copy over the Pynq-Z1 board files and directory and remove all of the extra stuff so it looks like this:
**PYNQ/
|__ **boards/
|__ <board_name>/
|__ base/
|__ base.bit
|__ base.hwh
|__ base.py
|__ constants.py
|__ __init__.py
|__ packages/
|__ boot_leds
|__ boot.py
|__ pre.sh
|__ petalinux_bsp/
|__ hardware_project/
|__ system.xsa
|__ <board_name>.spec
The <board_name> is going to be your board name. In my boards directory I created a new board called “test0” and did the following:
Copy over the Pynq-Z1 files in base: base.py, constants.py, init.py
Modified those (commented out code specific to Z1) just to have them available later as a template.
Open the .xsa file from the previous section. Delete the Z1 .bit/.hwh files and replace them with the new ones you created in your .xsa file and rename them: base.bit, base.hwh
Copy over from Z1: packages/boot_leds/boot.py & pre.sh
Again, just did this to have them as templates and commented out the boot.py code that was relevant to my board.
Copy the .xsa file over to <path>/PYNQ/boards/test0/petalinux_bsp/hardware_project/system.xsa
Be sure to rename the file to ‘system’.
Rename or create a new spec file: test0.spec
Change the board name on each variable and include whatever packages you need:
ARCH_test0 := arm
BSP_test0 :=
BITSTREAM_test0 := base/base.bit
FPGA_MANAGER_test0 := 1
STAGE4_PACKAGES_test0 := xrt pynq boot_leds ethernet
Download the rootfs file for Pynq 2.6 from here: PYNQ - Python productivity for Zynq - Board
Zynq: arm
Zynq Ultrascale+: aarch64
From here I just needed to reference the rootfs file and ran this command from the terminal:
Open a terminal at: /PYNQ/sdbuild
Run the make command:
make PREBUILT=<abs_path>/bionic.arm.2.6.0_2020_10_19.img BOARDS=test0 2>&1 | tee test0_build.log
This command will build the board (test0) and use the prebuilt rootfs (board agnostic image) and create a log file of the build process (super helpful). This part takes around 30 minutes for me.
Writing the image to an SD card
For instructions go here: Appendix — Python productivity for Zynq (Pynq)
I am going to provide a shortened version of instructions for reference. I’ve included the chars ‘<>’ to indicate areas that will change on your Ubuntu VM/Desktop. I am sure I’ve plagiarized this from another site but I don’t remember where:
Run to see what devices are currently mounted: df -h
Insert the Micro SD card into your SD card reader.
Run again to find the name of the inserted SD card: df -h
unmount the images:
umount /dev/sd<slot>1
umount /dev/sd<slot>2
Change directory to where the new image is located:
cd <path>/PYNQ/sdbuild/output
Write to the SD card (be careful, the details are important, see referenced web page):
sudo dd bs=4M if=<board_name>-2.6.0.img of=/dev/sd<slot>
sudo dd bs=4M if=bionic.arm.2.6.0.img of=/dev/sd<slot>
Put the SD card in your custom board
If your SD card is on SD0, make sure your bootstrap pins are configured and turn on the system and your good.
If your SD card is on SD1, go to the next sections.
Boot U Boot from JTAG, manually load SD1 from UART
-
Open a terminal to
<path>/PYNQ/sdbuild/build/<board_name>/petalinux_project/images/linux -
Attach your JTAG programmer (I used a Xilinx DLC10) to your board on the JTAG port.
-
Turn on your board. For the DLC10 you should see the light change from amber to green indicating that your board is supplying VREF.
-
In the terminal, source the required tools to load your system over JTAG:
source ~/Xilinx/Vitis/2020.1/settings64.sh
source ~/petalinux_2020_1/settings.sh
petalinux-util --webtalk off
-
Connect to the UART on your board. This will be needed to manually load the SD card from U Boot. Your UART settings were created in the Vivado section of this guide. Default is 115200, 8N1
-
From the terminal you will use Petalinux to load the FSBL and SSBL (U Boot):
petalinux-boot --jtag --u-boot -
Once the last step completes the UART terminal should take off and eventually result in a command prompt because I can’t find a Kernel to load. Load the Kernel from SD1 manually:
imageub_addr=0x10000000
fatload mmc 0:1 ${imageub_addr} image.ub
bootm ${imageub_addr}
You may need a different address offset based on your system but this is the flow from U Boot.
If all goes well you should see the Kernel being loaded.
Hope this helps someone.
