Connection refused error while programming zcu-104

Support
1

Hi All,

I have a zcu-104 board with pynq image on it, and I normally program it with a script that goes through the following steps (copied from Jupiter examples):

import sys
sys.path.append(“/home/xilinx/pynq”)
from random import randint
from pynq import MMIO
from pynq import PL
from pynq import Overlay
from pynq import Interrupt
import os

bspath = ‘bitfiles/’
PL = Overlay(bspath + ‘bistream.bit’ )

Everything worked ok till few days ago when I last programmed the board.
Today at PL = Overlay(bspath + ‘bistream.bit’ ) I’m facing this:

Traceback (most recent call last):
File “”, line 1, in
File “/usr/local/lib/python3.6/dist-packages/pynq/overlay.py”, line 315, in init
self.download()
File “/usr/local/lib/python3.6/dist-packages/pynq/overlay.py”, line 379, in download
super().download()
File “/usr/local/lib/python3.6/dist-packages/pynq/pl.py”, line 1938, in download
PL.shutdown()
File “/usr/local/lib/python3.6/dist-packages/pynq/pl.py”, line 1458, in shutdown
ip = cls.ip_dict
File “/usr/local/lib/python3.6/dist-packages/pynq/pl.py”, line 1297, in ip_dict
cls.client_request()
File “/usr/local/lib/python3.6/dist-packages/pynq/pl.py”, line 1416, in client_request
cls._remote = Client(address, family=‘AF_UNIX’, authkey=key)
File “/usr/lib/python3.6/multiprocessing/connection.py”, line 487, in Client
c = SocketClient(address)
File “/usr/lib/python3.6/multiprocessing/connection.py”, line 614, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
Error in sys.excepthook:
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/apport_python_hook.py”, line 63, in apport_excepthook
File “/usr/lib/python3/dist-packages/apport/init.py”, line 5, in
from apport.report import Report
File “”, line 971, in _find_and_load
File “”, line 955, in _find_and_load_unlocked
File “”, line 665, in _load_unlocked
File “”, line 674, in exec_module
File “”, line 779, in get_code
File “”, line 487, in _compile_bytecode
ValueError: bad marshal data (unknown type code)

Original exception was:
Traceback (most recent call last):
File “”, line 1, in
File “/usr/local/lib/python3.6/dist-packages/pynq/overlay.py”, line 315, in init
self.download()
File “/usr/local/lib/python3.6/dist-packages/pynq/overlay.py”, line 379, in download
super().download()
File “/usr/local/lib/python3.6/dist-packages/pynq/pl.py”, line 1938, in download
PL.shutdown()
File “/usr/local/lib/python3.6/dist-packages/pynq/pl.py”, line 1458, in shutdown
ip = cls.ip_dict
File “/usr/local/lib/python3.6/dist-packages/pynq/pl.py”, line 1297, in ip_dict
cls.client_request()
File “/usr/local/lib/python3.6/dist-packages/pynq/pl.py”, line 1416, in client_request
cls._remote = Client(address, family=‘AF_UNIX’, authkey=key)
File “/usr/lib/python3.6/multiprocessing/connection.py”, line 487, in Client
c = SocketClient(address)
File “/usr/lib/python3.6/multiprocessing/connection.py”, line 614, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused

This happens always: by running the script, by doing step by step manually, by connection to the board via ssh, by using the terminal.
Apparently nothing has been changed on the board till the last time it was programmed successfully. Any idea of what is happening?

Thanks in advance.

1 Like
2

It looks like the PL server is failing to start. Can you run sudo systemctl status pl_server and sudo journalctl -u pl_server.service and see if there is any information there as to why?

Peter

3

Yes you are right. Here what I found:

systemctl status pl_server â pl_server.service - PYNQ PL Server
Loaded: loaded (/lib/systemd/system/pl_server.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2021-05-10 12:35:23 CEST; 1h 10min left
Process: 3261 ExecStart=/usr/local/bin/pl_server.sh (code=exited, status=0/SUCCESS)
Main PID: 3262 (code=exited, status=1/FAILURE)
May 10 12:35:23 pynq pl_server.sh[3261]: line = next(iter(version_lines), ‘’)
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 2627, in _get_metadata
May 10 12:35:23 pynq pl_server.sh[3261]: for line in self.get_metadata_lines(name):
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 1408, in get_metadata_lines
May 10 12:35:23 pynq pl_server.sh[3261]: return yield_lines(self.get_metadata(name))
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 1405, in get_metadata
May 10 12:35:23 pynq pl_server.sh[3261]: return value.decode(‘utf-8’) if six.PY3 else value
May 10 12:35:23 pynq pl_server.sh[3261]: UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xd6 in position 9: invalid continuation byte
May 10 12:35:23 pynq systemd[1]: pl_server.service: Main process exited, code=exited, status=1/FAILURE
May 10 12:35:23 pynq systemd[1]: pl_server.service: Failed with result ‘exit-code’.

and

sudo journalctl -u pl_server.service
– Reboot –
May 10 12:35:20 pynq systemd[1]: Starting PYNQ PL Server…
May 10 12:35:21 pynq systemd[1]: Started PYNQ PL Server.
May 10 12:35:23 pynq pl_server.sh[3261]: Traceback (most recent call last):
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/local/bin/start_pl_server.py”, line 6, in
May 10 12:35:23 pynq pl_server.sh[3261]: from pkg_resources import load_entry_point
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 3088, in
May 10 12:35:23 pynq pl_server.sh[3261]: @_call_aside
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 3072, in _call_aside
May 10 12:35:23 pynq pl_server.sh[3261]: f(*args, **kwargs)
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 3101, in _initialize_master_working_set
May 10 12:35:23 pynq pl_server.sh[3261]: working_set = WorkingSet._build_master()
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 565, in _build_master
May 10 12:35:23 pynq pl_server.sh[3261]: ws = cls()
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 558, in init
May 10 12:35:23 pynq pl_server.sh[3261]: self.add_entry(entry)
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 614, in add_entry
May 10 12:35:23 pynq pl_server.sh[3261]: for dist in find_distributions(entry, True):
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 1964, in find_on_path
May 10 12:35:23 pynq pl_server.sh[3261]: for dist in factory(fullpath):
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 2034, in distributions_from_metadata
May 10 12:35:23 pynq pl_server.sh[3261]: root, entry, metadata, precedence=DEVELOP_DIST,
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 2473, in from_location
May 10 12:35:23 pynq pl_server.sh[3261]: py_version=py_version, platform=platform, **kw
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 2835, in _reload_version
May 10 12:35:23 pynq pl_server.sh[3261]: md_version = _version_from_file(self._get_metadata(self.PKG_INFO))
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 2437, in _version_from_file
May 10 12:35:23 pynq pl_server.sh[3261]: line = next(iter(version_lines), ‘’)
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 2627, in _get_metadata
May 10 12:35:23 pynq pl_server.sh[3261]: for line in self.get_metadata_lines(name):
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 1408, in get_metadata_lines
May 10 12:35:23 pynq pl_server.sh[3261]: return yield_lines(self.get_metadata(name))
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 1405, in get_metadata
May 10 12:35:23 pynq pl_server.sh[3261]: return value.decode(‘utf-8’) if six.PY3 else value
May 10 12:35:23 pynq pl_server.sh[3261]: UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xd6 in position 9: invalid continuation byte
May 10 12:35:23 pynq pl_server.sh[3261]: Error in sys.excepthook:
May 10 12:35:23 pynq pl_server.sh[3261]: Traceback (most recent call last):
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/apport_python_hook.py”, line 63, in apport_excepthook
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/apport/init.py”, line 5, in
May 10 12:35:23 pynq pl_server.sh[3261]: from apport.report import Report
May 10 12:35:23 pynq pl_server.sh[3261]: File “”, line 971, in _find_and_load
May 10 12:35:23 pynq pl_server.sh[3261]: File “”, line 955, in _find_and_load_unlocked
May 10 12:35:23 pynq pl_server.sh[3261]: File “”, line 665, in _load_unlocked
May 10 12:35:23 pynq pl_server.sh[3261]: File “”, line 674, in exec_module
May 10 12:35:23 pynq pl_server.sh[3261]: File “”, line 779, in get_code
May 10 12:35:23 pynq pl_server.sh[3261]: File “”, line 487, in _compile_bytecode
May 10 12:35:23 pynq pl_server.sh[3261]: ValueError: bad marshal data (invalid reference)
May 10 12:35:23 pynq pl_server.sh[3261]: Original exception was:
May 10 12:35:23 pynq pl_server.sh[3261]: Traceback (most recent call last):
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/local/bin/start_pl_server.py”, line 6, in
May 10 12:35:23 pynq pl_server.sh[3261]: from pkg_resources import load_entry_point
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 3088, in
May 10 12:35:23 pynq pl_server.sh[3261]: @_call_aside
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 3072, in _call_aside
May 10 12:35:23 pynq pl_server.sh[3261]: f(*args, **kwargs)
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 3101, in _initialize_master_working_set
May 10 12:35:23 pynq pl_server.sh[3261]: working_set = WorkingSet._build_master()
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 565, in _build_master
May 10 12:35:23 pynq pl_server.sh[3261]: ws = cls()
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 558, in init
May 10 12:35:23 pynq pl_server.sh[3261]: self.add_entry(entry)
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 614, in add_entry
May 10 12:35:23 pynq pl_server.sh[3261]: for dist in find_distributions(entry, True):
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 1964, in find_on_path
May 10 12:35:23 pynq pl_server.sh[3261]: for dist in factory(fullpath):
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 2034, in distributions_from_metadata
May 10 12:35:23 pynq pl_server.sh[3261]: root, entry, metadata, precedence=DEVELOP_DIST,
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 2473, in from_location
May 10 12:35:23 pynq pl_server.sh[3261]: py_version=py_version, platform=platform, **kw
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 2835, in _reload_version
May 10 12:35:23 pynq pl_server.sh[3261]: md_version = _version_from_file(self._get_metadata(self.PKG_INFO))
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 2437, in _version_from_file
May 10 12:35:23 pynq pl_server.sh[3261]: line = next(iter(version_lines), ‘’)
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 2627, in _get_metadata
May 10 12:35:23 pynq pl_server.sh[3261]: for line in self.get_metadata_lines(name):
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 1408, in get_metadata_lines
May 10 12:35:23 pynq pl_server.sh[3261]: return yield_lines(self.get_metadata(name))
May 10 12:35:23 pynq pl_server.sh[3261]: File “/usr/lib/python3/dist-packages/pkg_resources/init.py”, line 1405, in get_metadata
May 10 12:35:23 pynq pl_server.sh[3261]: return value.decode(‘utf-8’) if six.PY3 else value
May 10 12:35:23 pynq pl_server.sh[3261]: UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xd6 in position 9: invalid continuation byte
May 10 12:35:23 pynq systemd[1]: pl_server.service: Main process exited, code=exited, status=1/FAILURE
May 10 12:35:23 pynq systemd[1]: pl_server.service: Failed with result ‘exit-code’.

Considering that it always worked can be a simple case of corrupted filesystem?
Thanks
Stefano

1 Like
4

Could be that a package that’s installed has a weird character in it’s metadata - python - Can't run pip: UnicodeDecodeError - Stack Overflow is very similar to the error you’re seeing.

Peter

5

At the end of the day the problem was effectively 99.99% due to a corrupted filesystem on the sdcard that, I can suggest also to others, I learned is the first thing to be checked when some strange behaviours starts to happen on this board. In my case lot of errors was listed by fsck and the same tool was not able to fix them at all. A fresh sd solved everything.