PYNQ: PYTHON PRODUCTIVITY

"sudo reboot" powers off the system 1 one minute after reboot

Hi,

I managed to build a PYNQlinux image for an (unsupported) iWave G35 board, based on a Xilinx’s ZU19 Zynq MPSoC

I followed the instructions on PYNQ website and the board’s documentation and example design, which are all based on Petalinux and Xilinx tools v2020.1.
I started from a prebuilt PYNQ 2.6 rootfs and a pre-packaged pynq python package to reduce build time.

Now everything seems to be working fine, except that if I issue a “sudo reboot” command, the board does indeed reboot completely, but somehow powers itself off after about 1-1.5 minutes without any further notice: the power supply LED goes out, and so does the Ethernet LED. Also, the reset switch does absolutely nothing.

Notice that in order to reproduce the above situation, a combination of things must be put in place:

  • Some bitstream must be loaded on the FPGA, regardless of whether the programming is done by u-boot or the FPGA manager; if no bitstream is loaded, the board reboots fine
  • A graceful “reboot” from PYNQ’s prebuilt file system (Ubuntu 18.04’s systemd) must be issued: if you issue “reboot -f”, for instance, the board reboots just fine without any side effect. Ditto for the rootfs provided by a vanilla petalinux.

Notice how stopping the reboot at the u-boot prompt does not help: the board will just power off after the same time.

So I’ve essentially restricted it to something that systemd must be doing while shutting down its services/units, somehow triggering a sort of “delayed poweroff”.

Does anyone have any idea what this might be due to?
Any hint would be very much appreciated!

Thank you!

The only thing I can think of is Systemd’s restart watchdog isn’t being stopped when the board resets - you could set that to 0 and see if that resolves the issue.

Power functions are usually board specific so I’m guessing there is some functionality in the platform firmware that is causing the board to get powered off when the watchdog hits. This is all complete guesswork though.

Peter

Hi @PeterOgden,

thank you very much for the insight, I think we’re on the right track!
So here’s what I’ve tried:

  • sudo poweroff does indeed halt the system as expected (I cannot restart it with the reset button) BUT it does not power off the board and the ethernet PHY, which is the side effect I’m observing
  • I have the following four watchdog devices:
    crw------- 1 root root  10, 130 Mar 22 15:24 /dev/watchdog
    crw------- 1 root root 247,   0 Mar 22 15:24 /dev/watchdog0
    crw------- 1 root root 247,   1 Mar 22 15:24 /dev/watchdog1
    crw------- 1 root root 247,   2 Mar 22 15:24 /dev/watchdog2

If I forcibly remove all of them before issuing the shutdown, the problem does not appear.
I guess I could now bisect to identify which one(s) of the 4 is responsible for this behavior and dig further.

I measured the time it takes for the board to hang after the reboot is issues and it’s approximately 90s, which looks like one of systemd’s default job timeouts… though I’m not sure it’s related to the ShutdownWatchdogSec mentioned in your link…

Any other hints?
Thanks!!!

Systemd by default pokes /dev/watchdog which will be a hard link to one of the others. The easiest workaround may be to just disable the three watchdogs in the device-tree unless you have a need for them for system recovery.

I didn’t think we set any of the watchdogs by default so I’m intrigued as to where it’s coming from. Once the system is running you could have a look in /proc/1/fds and see if systemd has opened the device before shutdown occurs.

Peter

Hi,

I played with ShutdownWatchdogSec in /etc/systemd/system.conf and collected the following empirical evidence:

  • 0 makes the problem disapper
  • 10s, 90s, 120s, 130s makes it freeze (powerdown) instead of rebooting
  • 150s, 180s, 240s (or just commenting the line which should default to 10minutes) makes it freeze exactly 90s after reboot

As for the /dev/watchdog devices, I have 3 of them (as opposed to 4) on a different ZCU104 eval board running a stock PYNQ 2.4.

There are three timers in the silicon but I’ve only seen two turned on in Linux - the third is generally used by the CSU firmware. Those plus the 1 “default” which links to a numbered on gives you the three or four you are seeing.

I don’t know why the the third one is showing up in your case, I’m guessing there is something in the BSP which is enabling the CSU WDT or it’s a completely separate thing on the board. Unfortunately I don’t have access to either the schematic or the BSP to dig deeper into it. Turning off the board also requires some co-ordination with an external power chip so you could dig deeper there - I’d be inclined to just disable it in SystemD and call it a day.

Peter

Hi @PeterOgden,

I can of course disable ShutdownWatchdogSec in /etc/systemd/system.conf, but I’d rather disable the whole driver altogether (as per iWave’s suggestion).
While I realize this is as simple as setting CONFIG_DA9062_WATCHDOG=n in the kernel config, I’m bit at loss as to where this setting would fit in the whole build process (PYNQ’s sdbuild).
Is there any introductory document describing how sdbuild interacts with petalinux, and in turn how petalinux interacts with yocto, and so on?

Thank you!

The main documentation is the readthedocs and the blog post which are probably at different levels to what you want

PYNQ basically reuses the provided BSP wholesale one enabling FPGA manager and tweaking the device-tree. The best bet to disable the timer is to create a new Petalinux project from the BSP, make changes as necessary and then repacakge the BSP to pass to PYNQ. Alternatively you can put any changed files in a petalinux_bsp directory in your board folder and the sdbuild process will patch the BSP once it’s unpacked - see the ZCU104 for an example.

Peter