Using PyPI to deliver FPGA Overlays

[rock]

Keywords: PyPI, Overlay, Bitstream, PYNQ

Introduction

In many FPGA applications consisting of both a hardware design and software drivers, there is a basic question: Can that application be delivered in standardized software packaging ecosystems (if at all)?

In this article, we focus on Python packaging. For Python packages that are purely software, there is PyPI (Python Package Index) as a packaging ecosystem. With Python source code organized in a GitHub repository, users can choose to construct binary distribution (*.whl file) or source distribution (*.tar.gz file) to make them installable from anywhere in the world using pip install ...

However back to FPGA applications and from the PYNQ team’s perspective, we often want to deliver more than just Python or C/C++ software. For example:

1. What if we want to deliver Jupyter notebooks along with the Python package?
2. What if we want to deliver hardware designs, for example, Programmable Logic (PL) bitstreams, along with the Python package?

The first question might be easy to answer. Many developers just put the notebooks inside their Python package, and rely on users to pull them out into correct locations. The pynq Python package also has helper functions to do this.

The second question is a bit tricky. On one hand, developers do not want to maintain large binary files (i.e. a bitstream can be as large as 10MB) directly on a GitHub repository. On the other hand, hardware bitstreams have to go hand-in-hand with the Python drivers so users can deploy those full applications, perhaps deploying on multiple boards.

In this article, we will answer the above questions. We will give a short review on how we use PyPI to package hardware bitstreams and software programs together, and deliver both of them to users. If you are familiar with PYNQ, you may have already noticed that, the pynq python package has been made available on PyPI; users can simply do pip install pynq on their Zynq/Zynq Ultrascale boards, Alveo platforms, or Amazon Web Service (AWS) F1 instances. Keep in mind that we are delivering not only the Python drivers, but also FPGA bitstreams (overlays) and Jupyter notebooks to your specific board or device.

Python Packaging

Python has a well-known packaging ecosystem PyPI. By the time of writing this article, there are above 222,000 projects alive on PyPI, as shown below. We still remember that in the year of 2016, there were only around 100,000 projects. And this number is still growing fast!

PYNQ and PYNQ-related projects are usually packaged using the source distribution (instead of binary distribution). The reason is that we want to deliver overlays or notebooks that are board-specific, while the binary distributions are only tied to specific processor architectures (e.g. x86 or aarch64). An interesting thing is that, if a binary distribution cannot be found for a desired architecture, pip install always falls back to look for the source distribution. For this reason, many projects on PyPI not only prepare a lot of binary distributions targeting various architectures, but also provide a source distribution as a backup.

To better illustrate our methodology, I will take the PYNQ-HelloWorld as an example. If you open this GitHub repository, you will see files are organized in a certain way.

GitHub Repository

Let’s now examine the folders and files that are relevant to our packaging methodology.

boards

This is the folder consisting of all the hardware projects. If users go inside, and choose a specific device or architecture, users can run make process to rebuild the hardware projects. You may wonder why we have no hardware files (*.bit, *.hwh, *.xclbin) in this repository - hold your horses, we will see where those files come from in a minute.

pynq_helloworld

This is supposed to be the main python package. Since pynq_helloworld package does not require any new Python driver to run, this folder only has trivial __init__.py.

You will also notice that we have a notebook folder, which contains the notebooks for edge and PCIE devices, respectively.

You can also see a few *.link files - if you open them, they usually look like:

{
    "Pynq-Z1": {
        "url": "https://www.xilinx.com/bin/public/openDownload?filename=pynqhelloworld.resizer.pynqz1.bit",
        "md5sum": "6cdc717b1989dd61fb8f242548f94e2a"
    },
	...
}

The *.link concept was added to pynq starting from v2.5.1 when we added Alveo bitstreams (which are big!). In general, we refrain from putting large binary files directly on GitHub repository in order to reduce the repository size and ease maintenance. The *.link files are resolved using those URLs when you copy overlay files using pynq get-notebooks. For more information on *.link files, feel free to take a look at the documentation on link file processing and downloading overlays with setuptools. We will cover how setuptools deliver the real files in detail later.

MANIFEST.in

This file lists all the folder and files required for source distribution packaging. This is a standard technique widely used for Python packaging. The idea is to include more files into the source distribution package rather than just the default Python package. For PYNQ-HelloWorld, it looks like:

include pyproject.toml
include *.md
include LICENSE
recursive-include pynq_helloworld *

As you can see, besides the Python package pynq_helloworld (which contains Python files), we have also included some other files.

pyproject.toml

This file specifies minimum build system requirements for Python projects. For PYNQ-HelloWorld, this is trivial since we do not require any fancy package to build the distribution. There’s a minimum pip version (10.0) for this to work.

[build-system]
requires = ["setuptools", "wheel", "pynq"]

setup.py

This file has to be tailored for each project, since each project may want to organize files and folders in its own way. You can look at the methods defined in that file, including:

1. find_version(): it will make the version number recorded in __init__.py as the single source of version number.
2. extend_package(): this will make sure no only Python, but also C and other files can also be included as the package data, so they don’t get lost when packaging.
3. get_platform(): this is a method that returns platform type since our package is now supporting both edge and PCIE architectures.

The main body of setup() call, is pretty regular. You can find more information in Python documents. However, I would like to point out a few things that PYNQ does exclusively:

	entry_points={
        "pynq.notebooks": [
            "pynq-helloworld = {}.notebooks.{}".format(
                module_name, get_platform())
        ]
    },
    cmdclass={"build_py": build_py}

First, the entry_points section allows us to create hooks to pynq get-notebooks call. The string
pynq-helloworld = {}.notebooks.{}".format(module_name, get_platform()
specifies the folder which gets created (pynq-helloworld), as well as the path where the notebooks are copied from (<module_name>/notebooks/<platform>). In our example, module_name is set to pynq_helloworld and platform is resolved dynamically to either edge or pcie. Again, this is totally a customized solution; and you can find more examples in other PYNQ-related project such as Alveo-PYNQ, 5point-PYNQ, and more.

Second, the cmdclass section suggests we are using build.py from pynq.utils, instead of the default one; the reason is that we want some customized methods to resolve *.link files when delivering overlay files.

Preparing the Source Distribution

Now you should be able to run the following in the command line:

python3 setup.py sdist

This will package your project based on your settings. After this command, you can see a *.tar.gz file generated in dist folder. Before you upload to PyPI, you may want to validate the distribution directly on the target by running:

python3 -m pip install dist/*.tar.gz

When iterating this process, you can uninstall using:

python3 -m pip uninstall -y <module_name>

Uploading to PyPI

Uploading your package to PyPI is easy. You can follow Python packaging tutorials. You need an account (free!) set up on PyPI first, though. Once your account is ready, the key steps are:

python3 -m pip install --upgrade twine
python3 -m twine upload dist/*

If everything is successful, you can see something like this:

And your project can be viewed on PyPI page as well:

Since PyPI has a special naming convention, it will automatically rename the package to pynq-helloworld based on the module name pynq_helloworld.

Downloading from PyPI

Now you should be able to use the following command to install your package!

python3 -m pip install pynq-helloworld

Don’t forget you can also get the notebooks now. In command line, go to your Jupyter notebook home folder and run:

pynq get-notebooks pynq-helloworld -p .

By default, the above command will dump the notebooks into a new folder pynq-helloworld under the current directory. The -p option specifies the path where the notebooks are to be delivered. Now you can log onto Jupyter and try the notebooks!

Conclusion

In this article, we have seen how to package a Python project consisting of FPGA overlays, Python source code, and Jupyter notebooks. Hopefully if you have an FPGA overlay you’d like to share with the world, this article is helpful. If so, send us a note on how to pip install your work.

Reference

1. PyPI
2. Packaging Python Projects
3. PYNQ HelloWorld
4. PYNQ Command Line Interface
5. pynq.utils Module