PiAware AGC stopped scanning and stuck on max gain, why?

All four (4) of of my receivers have adaptive-gain=yes and adaptive-burst=yes.

Three of these behave like Graph-1, and one behaves like Graph-2

The 3 receivers with Grapg-1 have good antenna and dongles are FA Prostic Plus and RB Flightstick having builtin LNA. The receiver chip therefore gets strong signal.

The fourth receiver with Graph-2 has a generic DVB-T (black) and its antenna is the stock whip antenna which came with DVB-T. Naturally the receiver chip gets much weaker signal than those of other 3 recevers. In my opinion gain stuck at a fixed valuue is resilt of week RF signal input to receiver chip.

The weak RF signal input to receiver chip may be caused by low dc voltage hence LNA of dongle is not working properly. A bad or dying power supply adaptor may be the cause. Other reason of low RF signal may be moisture ingress inside antenna, or loose cable connectors.

Graph-1: FA ProStic plus and FA antenna

dump1090-localhost-misc-24h-6

 

Graph-2: Generic DVB-T and stock whip antenna

dump1090-localhost-misc-24h-7

1 Like

Back to trying to debug this problemā€¦

I have three questions that hopefully someone (PiAware developer?!) can answer:

  1. Under what circumstances is this message produced, ā€œadaptive: reached upper gain limit, halting dynamic range scan hereā€?
  2. Why then does all further agc scanning stop (ā€œhaltā€), forever (or until I reboot)? This is particularly odd behavior, why not just keep scanning?
  3. Why would this happen suddenly, when agc has been chugging along happily doing its thing for months without hitting the issue?

Tia.

2 Likes

When adaptive gain is doing an upward scan, has reached maximum gain, and the dynamic range is still acceptable. (At this point it canā€™t increase gain further, because itā€™s reached the maximum).

The trigger for a decrease in gain is if measured dynamic range drops below a threshold (measured dynamic range + half a gain step < dynamic range target). If measured dynamic range doesnā€™t drop, you wonā€™t see any further adaptive gain activity, because thereā€™s nothing to do ā€“ thereā€™s no apparent need to reduce gain (dynamic range is acceptable) and gain is already at maximum so thereā€™s no need to probe for a higher gain. dump1090 continues to measure dynamic range continuously, but if it never makes a decision to change the gain, it wonā€™t log anything.

Hardware changes, RF environment changes, software bug ā€“ take your pick.

I want to see stats.json so I can see what the measured dynamic range is ā€“ thatā€™d let me identify or rule out a software bug. You might want to capture/graph dynamic range (itā€™s under adaptive.noise_dbfs in stats)

2 Likes

Great information, thank you!

I am learning so please bear with me, is ā€œdynamic rangeā€ basically the difference between the strongest and weakest signals received, in dB?

If so (or in any case), does this graphs1090 chart essentially show dynamic range (the difference between the peak and weakest signals)? And then, if so, note how when the issue arises (between July 3 and midday July 7), and the gain is stuck at max and agc is halted, the difference between the peak and weakest signals is only about -20dB, but after a reboot and agc is operating and gain is set to 49.6 (midday July 7 onwards), the difference between peak and weakest is about -30dB. Isnā€™t that better, so agc should have detected that and restarted scanning on its own before I had to reboot?

Apologies if I am in left field. Appreciate your time.

1 Like

The adaptive gain logic estimates dynamic range by looking at the 40th percentile of all samples that are not part of a successfully decoded message (sorted by magnitude), so itā€™s not directly looking at the strength of any demodulated message. Itā€™s roughly trying to estimate the noise floor (in a different way to what graphs1090 is plotting as ā€œnoiseā€). The target is 30dB, i.e. it will try to find the highest gain that keeps the dynamic range >30dB (i.e. noise floor below -30dBFS)

4 Likes

Understood, thanks.

So the big difference in the peak:weakest signal level shown in the graphs1090 chart (~20dB vs ~30dB), does not indicate that agc should have restarted automatically, right? i.e. that chart does not mean that gain = 49.6 is better than max gain?

Btw, wrt the possible reasons for the sudden (and stuck) gain change (ā€œHardware changes, RF environment changes, software bug ā€“ take your pick.ā€), there havenā€™t been any hardware or software changes, and any RF environment changes would surely still be true after a reboot, so presumably the gain wouldnā€™t go straight back down again on a reboot.

Adaptive gain tries to pick an appropriate gain automatically based on what it measures. Thereā€™s hysteresis in how it does this to avoid repeatedly changing the gain if there are two gain settings that are right on the threshold for changing gain; perhaps in your setup that hysteresis means that a sufficiently ā€œquietā€ period can cause it to increase gain (and it may take some time after a restart for a period like this to happen), but subsequently it never gets sufficiently ā€œloudā€ to bring the gain back down again.

Adaptive gain tries to provide a good automatic default, and Iā€™d argue itā€™s providing a reasonable setting for your site (both gain settings seem like plausibly good values) ā€¦ but if you have better knowledge of which gain setting works better, then you should disable adaptive gain and manually set that gain.

Hardware characteristics can change over time.

Sure, but see above re hysteresis.

3 Likes

That explanation seems very plausible, thanks.

Looking at other stats (range, a/c seen, message rate, etc), I canā€™t discern any significant difference between the two different gain settings, so as you say, either seems to work.

One other curiosity, when gain = max, CPU utilization is higher than when at 49.6.

The demodulator basically has two parts:

  1. scan every sample looking for something that might be a preamble pattern (a sequence of pulses at the start of every Mode S / ADS-B message)
  2. when we see a possible preamble pattern, try to decode the following samples and see if it is a valid message

Part (1) is an approximately constant CPU load, and (per sample) is cheap to run ā€“ it has to run on every sample, so it has to be cheap.
Part (2) is more expensive, which is why we only run it selectively.

When the gain is set higher, thereā€™s more noise received, and there are more false positives in step (1) where noise is misinterpreted as a possible preamble. That means that part (2) needs to run more often, increasing the overall CPU load.

3 Likes

Excellent!

@obj, I really do appreciate the time you have taken to provide such clear and detailed answers to my questions. What I have learned is very enlightening, interesting, and I have enjoyed the ā€œconversationā€. This software is very smart, and well written (I am a software engineer myself, so I have some idea)!

Thatā€™s all the questions I have, for now :slight_smile:

1 Like