Here's what I see in the logs this evening:

Error: ✗ Failed to send request: Get "https://grafana.com/api/plugins/repo/flant-statusmap-panel": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

(This repeats over 3000 times.)

A quick DDG search doesn't see anything obvious, and I can hit https://grafana.com/api/plugins/repo/flant-statusmap-panel from my browser okay. Not sure what's up, but it's causing the Grafana container to keep restarting and never fully launch the web UI.

The speedtest container seems to just go unhealthy from time to time, with nothing in the logs indicating an issue. It's just like the flask app locks up and stops returning HTTP responses altogether.

This leads to one of my favorite graphs having blank periods until I log into the Pi and manually restart the container (composer restart speedtest):

I asked about this problem upstream here: https://github.com/MiguelNdeCarvalho/speedtest-exporter/issues/48#issuecomment-816360616

But I'm thinking of adding a simple cron job / script that checks if the container is unhealthy (maybe every 5 or 10 minutes), and if so, restarts it.

speedtest-cli allows you to manually choose a server instead of auto picking the best server. This can be using using the -s flag with a server_id example speedtest -s 35945 this allows us to tet the speeds without letting speedtest choose the best server, which might not pick the best server always and won't provide the most accurate data.

also maybe adding a way to test multiple servers could be a good idea since my ISP only promises speeds up to their node i can monitor speeds on different nodes / countries to check their peering and routing. I Currently try to do this with a cronjob and a python script which stores data to a sqlite DB. Grafana and Prometheus solution might be a much better choice in this case.

Through my internet-pi project, I'd like to add in Starlink monitoring using https://github.com/danopstech/starlink_exporter (though I've run into one issue here: https://github.com/danopstech/starlink_exporter/issues/1

Description

on startup the speedtest container crashes (and restarts continuously) with the following error:

speedtest_1   | Current thread 0xb6fa0390 (most recent call first):
speedtest_1   | <no Python frame>
speedtest_1   | Fatal Python error: init_interp_main: can't initialize time
speedtest_1   | Python runtime state: core initialized
speedtest_1   | PermissionError: [Errno 1] Operation not permitted

Environment

The OS for the raspberry pi was the provided raspberry OS from factory (which already makes me think I should make a fresh, controlled, install by myself).

The environment this is running is:

+pi@raspberrypi:~ $ whoami
pi
+pi@raspberrypi:~ $ groups
pi adm dialout cdrom sudo audio video plugdev games users input netdev lpadmin docker gpio i2c spi
+pi@raspberrypi:~ $ uname -a
Linux raspberrypi 5.10.17-v7l+ #1421 SMP Thu May 27 14:00:13 BST 2021 armv7l GNU/Linux
+pi@raspberrypi:~ $ cat /proc/cpuinfo  | grep Model
Model		: Raspberry Pi 4 Model B Rev 1.4

Instalation

The project setup was done following your article, up to the point of docker-compose up -d (since I'm running this locally and don't need remote access).

Fixes attempted

• Changed speedtest container to different versions ( 2.8, 3.0 and 3.2.2)
• Run the container with elevated privileges (which led me to a perhaps rash PR #16 )

To Do

• [x] reinstall a fresh Raspberry OS and try from scratch. I saw similar issues solved by this solution

Left docker containers running for over 24 hours and all three gauges started to return N/A.

Fixed by removing the "max data points" in the query options for the panels.

I am by no means an expert in this, I just like to play around with cool projects. Is this the correct way to resolve?

I am having a hard time getting the speed test to work. This is my first time working with docker and grafana-as a result I am unsure how import the speed-test portion into the dashboard. When I navigate to /targets, I can see that it is down.

Any help? Thanks

Hi there,

I am not sure if I may be doing something wrong, but the speedtest-exporter container is healthy and up, but on the Grafana dashboards on :3030, I am only occasionally seeing speedtest data.

Any clues as to what may be happening?

Thanks. ilyesm

Hi,

Is it possible to mark a square in the statusmap red when downtime occurred and also keep it red in larger timespans? Like if one minute is red also make the day red?

I have a 2 GB Raspberry Pi running 32-bit PiOS that I otherwise use as an ADS-B receiver. I installed this repo on it, but the speedtest container keeps restarting. This is what I see added in the container log every minute: Current thread 0xb6f76390 (most recent call first): Fatal Python error: init_interp_main: can't initialize time Python runtime state: core initialized PermissionError: [Errno 1] Operation not permitted

What could be wrong?

Logs show [2021-04-15 14:16:46.303] [error] Configuration - Cannot retrieve configuration document (0) ,[2021-04-15 14:16:46.304] [error] ConfigurationError - Could not retrieve or read configuration (Configuration) ,[2021-04-15 14:16:46.304] [error] ConfigurationError - Could not retrieve or read configuration (Configuration) ,{ , "type": "log", , "timestamp": "2021-04-15T14:16:46Z", , "message": "Configuration - Could not retrieve or read configuration (ConfigurationError)", , "level": "error" ,} ,ERROR:Speedtest-Exporter:Exception on /metrics [GET] ,Traceback (most recent call last): , File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app , response = self.full_dispatch_request() , File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1952, in full_dispatch_request , rv = self.handle_user_exception(e) , File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1821, in handle_user_exception , reraise(exc_type, exc_value, tb) , File "/usr/local/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise , raise value , File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request , rv = self.dispatch_request() , File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request , return self.view_functionsrule.endpoint , File "/app/exporter.py", line 70, in updateResults , r_server, r_jitter, r_ping, r_download, r_upload, r_status = runTest() , File "/app/exporter.py", line 47, in runTest , output = subprocess.check_output(cmd) , File "/usr/local/lib/python3.9/subprocess.py", line 424, in check_output , return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, , File "/usr/local/lib/python3.9/subprocess.py", line 528, in run , raise CalledProcessError(retcode, process.args, ,subprocess.CalledProcessError: Command '['speedtest', '--format=json-pretty', '--progress=no', '--accept-license', '--accept-gdpr']' returned non-zero exit status 2. ,[2021-04-15 14:38:22.019] [error] Trying to get interface information on non-initialized socket. ,[2021-04-15 14:38:27.535] [error] Configuration - Couldn't resolve host name (HostNotFoundException) ,[2021-04-15 14:38:27.535] [error] Configuration - Cannot retrieve configuration document (0) ,[2021-04-15 14:38:27.536] [error] ConfigurationError - Could not retrieve or read configuration (Configuration) ,[2021-04-15 14:38:27.536] [error] ConfigurationError - Could not retrieve or read configuration (Configuration) ,{ , "type": "log", , "timestamp": "2021-04-15T14:38:27Z", , "message": "Configuration - Could not retrieve or read configuration (ConfigurationError)", , "level": "error" ,} ,ERROR:Speedtest-Exporter:Exception on /metrics [GET] ,Traceback (most recent call last): , File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app , response = self.full_dispatch_request() , File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1952, in full_dispatch_request , rv = self.handle_user_exception(e) , File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1821, in handle_user_exception , reraise(exc_type, exc_value, tb) , File "/usr/local/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise , raise value , File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request , rv = self.dispatch_request() , File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request , return self.view_functionsrule.endpoint , File "/app/exporter.py", line 70, in updateResults , r_server, r_jitter, r_ping, r_download, r_upload, r_status = runTest() , File "/app/exporter.py", line 47, in runTest , output = subprocess.check_output(cmd) , File "/usr/local/lib/python3.9/subprocess.py", line 424, in check_output , return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, , File "/usr/local/lib/python3.9/subprocess.py", line 528, in run , raise CalledProcessError(retcode, process.args, ,subprocess.CalledProcessError: Command '['speedtest', '--format=json-pretty', '--progress=no', '--accept-license', '--accept-gdpr']' returned non-zero exit status 2. ,[2021-04-15 16:28:01.854] [error] Trying to get interface information on non-initialized socket. ,[2021-04-15 16:28:07.373] [error] Configuration - Couldn't resolve host name (HostNotFoundException) ,[2021-04-15 16:28:07.373] [error] Configuration - Cannot retrieve configuration document (0) ,[2021-04-15 16:28:07.374] [error] ConfigurationError - Could not retrieve or read configuration (Configuration) ,[2021-04-15 16:28:07.374] [error] ConfigurationError - Could not retrieve or read configuration (Configuration) ,{ , "type": "log", , "timestamp": "2021-04-15T16:28:07Z", , "message": "Configuration - Could not retrieve or read configuration (ConfigurationError)", , "level": "error" ,} ,ERROR:Speedtest-Exporter:Exception on /metrics [GET] ,Traceback (most recent call last): , File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app , response = self.full_dispatch_request() , File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1952, in full_dispatch_request , rv = self.handle_user_exception(e) , File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1821, in handle_user_exception , reraise(exc_type, exc_value, tb) , File "/usr/local/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise , raise value , File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request , rv = self.dispatch_request() , File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request , return self.view_functionsrule.endpoint , File "/app/exporter.py", line 70, in updateResults , r_server, r_jitter, r_ping, r_download, r_upload, r_status = runTest() , File "/app/exporter.py", line 47, in runTest , output = subprocess.check_output(cmd) , File "/usr/local/lib/python3.9/subprocess.py", line 424, in check_output , return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, , File "/usr/local/lib/python3.9/subprocess.py", line 528, in run , raise CalledProcessError(retcode, process.args, ,subprocess.CalledProcessError: Command '['speedtest', '--format=json-pretty', '--progress=no', '--accept-license', '--accept-gdpr']' returned non-zero exit status 2. ,

I will soon be archiving this repository, after closing out remaining issues.

I have merged it into my internet-pi repository, though most of the configuration could still be used completely independently, if you copy out the internet-monitoring directory, then manually template out the relevant files (prometheus and munin config files) from the templates directory.

Otherwise, I recommend setting it up using the playbook included in the internet-pi repository, which sets everything up for you and gets it running on Raspberry Pi OS or any Debian-like OS.