Interesting, I had a similar issue yesterday as well with one of my sites. Just in the fwiw and if it helps department.
During the course of the morning, I looked at my stats page to see how the the day was progressing, to find that it was reported that my 1090 feeder (Site: 183579) hadn’t reported in three hours. My 978 feeder (Site: 184474) was reporting fine during the same timeframe, and shares the same internet connectivity. (My first hunch)
I did look at piaware.log (shown below) and syslog, but there isn’t any entries during this time (11:19-14:14 EST).
Graphs1090 showed I was still seeing AC messages:
and my bandwidth charts showed continuous activity:
but clearily something wasn’t working properly during this time:
Unfortunately, I didn’t check the health of dump1090 in the moment, or do any further troubleshooting, as I was in the middle of my work day… I just rebooted the box at 14:14.
Oct 28 10:30:39 ThinAware piaware[522]: mlat-client(1363233): Receiver status: connected
Oct 28 10:30:39 ThinAware piaware[522]: mlat-client(1363233): Server status: synchronized with 527 nearby receivers
Oct 28 10:30:39 ThinAware piaware[522]: mlat-client(1363233): Receiver: 2045.4 msg/s received 586.7 msg/s processed (29%)
Oct 28 10:30:39 ThinAware piaware[522]: mlat-client(1363233): Server: 0.1 kB/s from server 0.1kB/s TCP to server 4.4kB/s UDP to server
Oct 28 10:30:39 ThinAware piaware[522]: mlat-client(1363233): Results: 30.9 positions/minute
Oct 28 10:30:39 ThinAware piaware[522]: mlat-client(1363233): Aircraft: 37 of 49 Mode S, 113 of 276 ADS-B used
Oct 28 10:33:39 ThinAware piaware[522]: 5712906 msgs recv'd from dump1090-fa (6869 in last 5m); 5712704 msgs sent to FlightAware
Oct 28 10:37:55 ThinAware piaware[522]: mlat-client(1363233): Route MTU changed to 1500
Oct 28 10:38:07 ThinAware piaware[522]: mlat-client(1363233): Route MTU changed to 1476
Oct 28 10:38:39 ThinAware piaware[522]: 5719771 msgs recv'd from dump1090-fa (6865 in last 5m); 5719569 msgs sent to FlightAware
Oct 28 10:43:39 ThinAware piaware[522]: 5726771 msgs recv'd from dump1090-fa (7000 in last 5m); 5726569 msgs sent to FlightAware
Oct 28 10:45:39 ThinAware piaware[522]: mlat-client(1363233): Receiver status: connected
Oct 28 10:45:39 ThinAware piaware[522]: mlat-client(1363233): Server status: synchronized with 520 nearby receivers
Oct 28 10:45:39 ThinAware piaware[522]: mlat-client(1363233): Receiver: 2068.6 msg/s received 601.0 msg/s processed (29%)
Oct 28 10:45:39 ThinAware piaware[522]: mlat-client(1363233): Server: 0.1 kB/s from server 0.1kB/s TCP to server 4.7kB/s UDP to server
Oct 28 10:45:39 ThinAware piaware[522]: mlat-client(1363233): Results: 26.0 positions/minute
Oct 28 10:45:39 ThinAware piaware[522]: mlat-client(1363233): Aircraft: 25 of 43 Mode S, 120 of 262 ADS-B used
Oct 28 10:48:39 ThinAware piaware[522]: 5733883 msgs recv'd from dump1090-fa (7112 in last 5m); 5733681 msgs sent to FlightAware
Oct 28 10:53:39 ThinAware piaware[522]: 5741006 msgs recv'd from dump1090-fa (7123 in last 5m); 5740804 msgs sent to FlightAware
Oct 28 10:58:09 ThinAware piaware[522]: mlat-client(1363233): Route MTU changed to 1500
Oct 28 10:58:23 ThinAware piaware[522]: mlat-client(1363233): Route MTU changed to 1476
Oct 28 10:58:32 ThinAware piaware[522]: Lost connection to adept server at piaware.flightaware.com/1200: server closed connection
Oct 28 10:58:32 ThinAware piaware[522]: multilateration data no longer required, disabling mlat client
Oct 28 10:58:33 ThinAware piaware[522]: fa-mlat-client exited normally
Oct 28 10:58:33 ThinAware piaware[522]: reconnecting in 5 seconds...
Oct 28 10:58:33 ThinAware piaware[522]: mlat-client(1363233): Disconnecting from localhost:30005: Lost connection to multilateration server, no need for input data
Oct 28 10:58:33 ThinAware piaware[522]: mlat-client(1363233): Exiting on connection loss
Oct 28 10:58:38 ThinAware piaware[522]: Connecting to FlightAware adept server at piaware.flightaware.com/1200
Oct 28 10:58:39 ThinAware piaware[522]: 5748413 msgs recv'd from dump1090-fa (7407 in last 5m); 5748057 msgs sent to FlightAware
Oct 28 10:58:39 ThinAware piaware[522]: Connection with adept server at piaware.flightaware.com/1200 established
Oct 28 10:58:39 ThinAware piaware[522]: TLS handshake with adept server at piaware.flightaware.com/1200 completed
Oct 28 10:58:39 ThinAware piaware[522]: FlightAware server certificate validated
Oct 28 10:58:39 ThinAware piaware[522]: encrypted session established with FlightAware
Oct 28 10:58:40 ThinAware piaware[522]: logged in to FlightAware as user cnuwer
Oct 28 10:58:40 ThinAware piaware[522]: my feeder ID is xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Oct 28 10:58:40 ThinAware piaware[522]: site statistics URL: https://flightaware.com/adsb/stats/user/cnuwer#stats-183579
Oct 28 10:58:40 ThinAware piaware[522]: multilateration data requested
Oct 28 10:58:40 ThinAware piaware[522]: Starting multilateration client: /usr/lib/piaware/helpers/fa-mlat-client --input-connect localhost:30005 --input-type dump1090 --results beast,connect,localhost:30104 --results beast,listen,30105 --results ext_basestation,listen,30106 --udp-transport 2620:13d:c000:11::198:19649:3601478145
Oct 28 10:58:40 ThinAware piaware[522]: mlat-client(1829149): fa-mlat-client 0.2.12 starting up
Oct 28 10:58:40 ThinAware piaware[522]: mlat-client(1829149): Using UDP transport to 2620:13d:c000:11::198 port 19649
Oct 28 10:58:40 ThinAware piaware[522]: mlat-client(1829149): Listening for Beast-format results connection on port 30105
Oct 28 10:58:40 ThinAware piaware[522]: mlat-client(1829149): Listening for Extended Basestation-format results connection on port 30106
Oct 28 10:58:40 ThinAware piaware[522]: mlat-client(1829149): Route MTU changed to 1500
Oct 28 10:58:40 ThinAware piaware[522]: mlat-client(1829149): Input connected to localhost:30005
Oct 28 10:58:40 ThinAware piaware[522]: mlat-client(1829149): Input format changed to BEAST, 12MHz clock
Oct 28 10:58:41 ThinAware piaware[522]: mlat-client(1829149): Beast-format results connection with ::1:30104: connection established
Oct 28 10:58:56 ThinAware piaware[522]: mlat-client(1829149): Route MTU changed to 1476
Oct 28 11:03:39 ThinAware piaware[522]: 5756193 msgs recv'd from dump1090-fa (7780 in last 5m); 5755818 msgs sent to FlightAware
Oct 28 11:08:39 ThinAware piaware[522]: 5764031 msgs recv'd from dump1090-fa (7838 in last 5m); 5763656 msgs sent to FlightAware
Oct 28 11:13:39 ThinAware piaware[522]: 5771449 msgs recv'd from dump1090-fa (7418 in last 5m); 5771074 msgs sent to FlightAware
Oct 28 11:13:41 ThinAware piaware[522]: mlat-client(1829149): Receiver status: connected
Oct 28 11:13:41 ThinAware piaware[522]: mlat-client(1829149): Server status: synchronized with 528 nearby receivers
Oct 28 11:13:41 ThinAware piaware[522]: mlat-client(1829149): Receiver: 1975.0 msg/s received 598.0 msg/s processed (30%)
Oct 28 11:13:41 ThinAware piaware[522]: mlat-client(1829149): Server: 0.1 kB/s from server 0.1kB/s TCP to server 4.4kB/s UDP to server
Oct 28 11:13:41 ThinAware piaware[522]: mlat-client(1829149): Results: 43.6 positions/minute
Oct 28 11:13:41 ThinAware piaware[522]: mlat-client(1829149): Aircraft: 30 of 46 Mode S, 111 of 275 ADS-B used
Oct 28 11:18:39 ThinAware piaware[522]: 5778436 msgs recv'd from dump1090-fa (6987 in last 5m); 5778061 msgs sent to FlightAware
Oct 28 11:18:45 ThinAware piaware[522]: mlat-client(1829149): Route MTU changed to 1500
Oct 28 11:19:12 ThinAware piaware[522]: mlat-client(1829149): Route MTU changed to 1476
Oct 28 11:19:16 ThinAware piaware[522]: Lost connection to adept server at piaware.flightaware.com/1200: server closed connection
Oct 28 14:14:33 ThinAware piaware[513]: creating pidfile /run/piaware/piaware.pid
Oct 28 14:14:33 ThinAware piaware[513]: ****************************************************
Oct 28 14:14:33 ThinAware piaware[513]: piaware version 7.2 is running, process ID 513
Oct 28 14:14:33 ThinAware piaware[513]: your system info is: Linux ThinAware 5.10.0-19-amd64 #1 SMP Debian 5.10.149-2 (2022-10-21) x86_64 GNU/Linux
Oct 28 14:14:34 ThinAware piaware[513]: Connecting to FlightAware adept server at piaware.flightaware.com/1200
Oct 28 14:14:34 ThinAware piaware[513]: Connection to adept server at piaware.flightaware.com/1200 failed: couldn't open socket: Temporary failure in name resolution
Oct 28 14:14:34 ThinAware piaware[513]: reconnecting in 5 seconds...
Oct 28 14:14:36 ThinAware piaware[513]: ADS-B data program 'dump1090-fa' is listening on port 30005, so far so good
Oct 28 14:14:36 ThinAware piaware[513]: Starting faup1090: /usr/lib/piaware/helpers/faup1090 --net-bo-ipaddr localhost --net-bo-port 30005 --stdout --lat 39.16x --lon -78.15x
Oct 28 14:14:36 ThinAware piaware[513]: Started faup1090 (pid 664) to connect to dump1090-fa
Oct 28 14:14:36 ThinAware piaware[513]: UAT support disabled by local configuration setting: uat-receiver-type
Oct 28 14:14:39 ThinAware piaware[513]: Connecting to FlightAware adept server at piaware.flightaware.com/1200
Oct 28 14:14:40 ThinAware piaware[513]: Connection with adept server at piaware.flightaware.com/1200 established
Oct 28 14:14:40 ThinAware piaware[513]: TLS handshake with adept server at piaware.flightaware.com/1200 completed
Oct 28 14:14:40 ThinAware piaware[513]: FlightAware server certificate validated
Oct 28 14:14:40 ThinAware piaware[513]: encrypted session established with FlightAware
Oct 28 14:14:40 ThinAware piaware[513]: adept reported location: 39.16xxx, -78.15xxx, 746ft AMSL
Oct 28 14:14:40 ThinAware piaware[513]: logged in to FlightAware as user cnuwer
Oct 28 14:14:40 ThinAware piaware[513]: my feeder ID is xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Oct 28 14:14:40 ThinAware piaware[513]: site statistics URL: https://flightaware.com/adsb/stats/user/cnuwer#stats-183579
Oct 28 14:14:40 ThinAware piaware[513]: multilateration data requested
Oct 28 14:14:40 ThinAware piaware[513]: Starting multilateration client: /usr/lib/piaware/helpers/fa-mlat-client --input-connect localhost:30005 --input-type dump1090 --results beast,connect,localhost:30104 --results beast,listen,30105 --results ext_basestation,listen,30106 --udp-transport 2620:13d:c000:11::201:9012:1627360442
Oct 28 14:14:41 ThinAware piaware[513]: mlat-client(784): fa-mlat-client 0.2.12 starting up
Oct 28 14:14:41 ThinAware piaware[513]: mlat-client(784): Using UDP transport to 2620:13d:c000:11::201 port 9012
Oct 28 14:14:41 ThinAware piaware[513]: mlat-client(784): Listening for Beast-format results connection on port 30105
Oct 28 14:14:41 ThinAware piaware[513]: mlat-client(784): Listening for Extended Basestation-format results connection on port 30106
Oct 28 14:14:41 ThinAware piaware[513]: mlat-client(784): Route MTU changed to 1500
Oct 28 14:14:41 ThinAware piaware[513]: mlat-client(784): Input connected to localhost:30005
Oct 28 14:14:41 ThinAware piaware[513]: mlat-client(784): Input format changed to BEAST, 12MHz clock
Oct 28 14:14:42 ThinAware piaware[513]: mlat-client(784): Beast-format results connection with ::1:30104: connection established
Oct 28 14:14:45 ThinAware piaware[513]: piaware received a message from dump1090-fa!
Oct 28 14:14:45 ThinAware piaware[513]: piaware has successfully sent several msgs to FlightAware!
Oct 28 14:15:00 ThinAware piaware[513]: mlat-client(784): Route MTU changed to 1476
Oct 28 14:15:07 ThinAware piaware[513]: 997 msgs recv'd from dump1090-fa; 997 msgs sent to FlightAware
Oct 28 14:20:07 ThinAware piaware[513]: 7443 msgs recv'd from dump1090-fa (6446 in last 5m); 7443 msgs sent to FlightAware
Oct 28 14:25:07 ThinAware piaware[513]: 13910 msgs recv'd from dump1090-fa (6467 in last 5m); 13910 msgs sent to FlightAware
Oct 28 14:29:41 ThinAware piaware[513]: mlat-client(784): Receiver status: connected
Oct 28 14:29:41 ThinAware piaware[513]: mlat-client(784): Server status: synchronized with 518 nearby receivers
Oct 28 14:29:41 ThinAware piaware[513]: mlat-client(784): Receiver: 1935.1 msg/s received 567.0 msg/s processed (29%)
Oct 28 14:29:41 ThinAware piaware[513]: mlat-client(784): Server: 0.0 kB/s from server 0.1kB/s TCP to server 4.3kB/s UDP to server
Oct 28 14:29:41 ThinAware piaware[513]: mlat-client(784): Results: 7.0 positions/minute
Oct 28 14:29:41 ThinAware piaware[513]: mlat-client(784): Aircraft: 26 of 46 Mode S, 100 of 241 ADS-B used
Oct 28 14:30:07 ThinAware piaware[513]: 20403 msgs recv'd from dump1090-fa (6493 in last 5m); 20403 msgs sent to FlightAware
Oct 28 14:35:07 ThinAware piaware[513]: 26467 msgs recv'd from dump1090-fa (6064 in last 5m); 26467 msgs sent to FlightAware
Oct 28 14:40:07 ThinAware piaware[513]: 33014 msgs recv'd from dump1090-fa (6547 in last 5m); 33014 msgs sent to FlightAware