Lost connection to server, no reconnect

[simtra]

Over the last few months I have been having an odd issue with my tracker. It just stops returning data to the flightaware server, however it never seems to try to reconnect.

Flightaware show I am offline however my local skyaware flight page shows flights.

I can see the active piaware process active. However systemctl restart piaware take a couple of minutes to return. After that it seems to correct the problem. The only reason I bring this up is the last line in the log. I figured I would just post an post as it seems to be happening every few weeks and the system used to be stable and I personally am not aware of any changes

Sep 21 18:45:56 piaware piaware[477]: 19035360 msgs recv'd from beast-splitter (8348 in last 5m); 19031363 msgs sent to FlightAware
Sep 21 18:45:56 piaware piaware[477]: 892915 msgs recv'd from dump978-fa (650 in last 5m); 892787 msgs sent to FlightAware
Sep 21 18:50:56 piaware piaware[477]: 19043823 msgs recv'd from beast-splitter (8463 in last 5m); 19039826 msgs sent to FlightAware
Sep 21 18:50:56 piaware piaware[477]: 893529 msgs recv'd from dump978-fa (614 in last 5m); 893401 msgs sent to FlightAware
Sep 21 18:55:39 piaware piaware[477]: mlat-client(1760): Receiver status: connected
Sep 21 18:55:39 piaware piaware[477]: mlat-client(1760): Server status:   clock unstable
Sep 21 18:55:39 piaware piaware[477]: mlat-client(1760): Receiver: 1728.3 msg/s received      750.7 msg/s processed (43%)
Sep 21 18:55:39 piaware piaware[477]: mlat-client(1760): Server:      0.2 kB/s from server    0.1kB/s TCP to server     5.3kB/s UDP to server
Sep 21 18:55:39 piaware piaware[477]: mlat-client(1760): Results:  73.2 positions/minute
Sep 21 18:55:39 piaware piaware[477]: mlat-client(1760): Aircraft: 18 of 28 Mode S, 157 of 282 ADS-B used
Sep 21 18:55:56 piaware piaware[477]: 19052103 msgs recv'd from beast-splitter (8280 in last 5m); 19048106 msgs sent to FlightAware
Sep 21 18:55:56 piaware piaware[477]: 894259 msgs recv'd from dump978-fa (730 in last 5m); 894131 msgs sent to FlightAware
Sep 21 18:56:50 piaware piaware[477]: Lost connection to adept server at 206.253.84.195/1200: server closed connection

[simtra]

Might have found the problem. This might be a large site problem. While tailing the /var/log/piaware.log I saw the following:

Sep 22 12:34:41 piaware piaware[19583]: traceback: no space left on device
Sep 22 12:34:41 piaware piaware[19583]:     while executing
Sep 22 12:34:41 piaware piaware[19583]: "close $f"
Sep 22 12:34:41 piaware piaware[19583]:     (procedure "write_status_file" line 9)
Sep 22 12:34:41 piaware piaware[19583]:     invoked from within
Sep 22 12:34:41 piaware piaware[19583]: "write_status_file"
Sep 22 12:34:46 piaware piaware[19583]: failed to write status file: no space left on device
Sep 22 12:34:46 piaware piaware[19583]: traceback: no space left on device
Sep 22 12:34:46 piaware piaware[19583]:     while executing
Sep 22 12:34:46 piaware piaware[19583]: "close $f"
Sep 22 12:34:46 piaware piaware[19583]:     (procedure "write_status_file" line 9)
Sep 22 12:34:46 piaware piaware[19583]:     invoked from within
Sep 22 12:34:46 piaware piaware[19583]: "write_status_file"

The timelapse addin is storing its data in /run and since it is “only” 185M it is filling up /run