There are two different gain settings in librtlsdr, which those options control
–enable-agc enables the digital AGC within the RTL2832U. In practice I find this is not very helpful (it’s just scaling the data within the digital stage, so there’s no additional information being extracted)
–gain -10 enables the analog AGC in the tuner chip, which has more potential for being useful.
–gain (something else) will disable the analog AGC and directly set the tuner gain to that value in dB (or whatever the tuner supports that is nearby)
With a R820T tuner the tuner gain setting actually controls both the LNA (RF) gain and the mixer gain. You can’t control them independently via librtlsdr.
The analog AGC works best if you have a signal that stays at a particular strength for some time, to give the feedback loop some time to adjust - which doesn’t really hold for 1090MHz signals, as the bursts are very short (around 100us at most) and you will be getting a large range of signal strengths.
Personally I run with --gain 48 - one notch down from max gain on a R820T. Max gain (the default) seems to add a disproportionate amount of noise. For strong/close signals or with an amplified antenna you might want a lower gain to avoid clipping in the ADC stage.