I don’t know – but sometimes it seems like me, LOL.
1 Like
I have been doing some testing over the last week or so. Here are some results, in case they are of interest.
The test set up comprises:
wideband LNA → 1090 Filter → splitter |→ Airspy Mini → Raspberry Pi4B
. |→ Airspy R2 → Raspberry Pi5B
The main components are shown in the photo.
I already had the LNA, R2 and Pi4. Ideally you would use the same receiver and Pi in each branch for comparisons, but I wanted to acquire parts I could use as spares for my main feeder which has a mini and a Pi5. The setup is attached to a FlightAware 66cm antenna with a 5 metre LL195 cable.
I’ve adjusted the settings to try to get the two branches as closely matched as possible when both are running airspy_adsb. The Pi5 is throttled to a CPU frequency of 1.8GHz to match the Pi4’s maximum – both are running at constant clock speed. With these settings the Airspy Mini/Pi4 branch produces two percent less positions and messages than the R2/Pi5 branch. The airspy_adsb parameters on both machines are identical:
-v -t 300 -f 1 -w 5 -P 7 -e 60 -R rms -l 47787:beast -c 127.0.0.1:30004:beast -g 19 -m 20
When Stream1090 is running, the airspy_rx parameters set the same gain and sample rate (20 MSPS raw = 10 MSPS IQ):
-a 10000000 -f 1090 -g 19 -r -
Here are the airspy graphs corresponding to the period in the comparison charts that follow. The Mini/Pi4 branch has slightly higher signal and noise, but the SNRs are quite close.
R2/Pi5
Mini/Pi4 (gap when Stream1090 was running)
I made a script that extracted the last1min data from readsb’s stats.json file every minute, and looked at the time series for the messages_valid, adsb_icao and max_distance fields. I captured two hours of data for each configuration.
The charts shows Message Rates for the R2/Pi5 branch running airspy_adsb throughout the test. The Mini/Pi4 branch starts running airspy_adsb, swaps to stream1090_10M (16 bit IQ) -u 24 after two hours, then swaps to stream1090_10M (16 bit IQ) -u 20 for two hours, and then back to airspy-adsb for another two hours. (I decided to use the IQ setting because I found that adding the filter gave a significant improvement in performance of the unfiltered R2/Pi4 setup that had been running previously.)
Allowance needs to be made for the Mini/Pi4 branch’s 2% disadvantage when running airspy_adsb. From this and earlier comparisons, I estimate that - for my setup and settings - Stream1090 is currently achieving approximately 90% of airspy_adsb’s performance in terms of messages and positions. Average Max Distance is at 98%
I’ve also tested using the -r switch. stream1090_10M -r -u24. The results were about 2% worse for Messages and 1% worse for positions. There was no change in Average Max Distance performance.
Are my results consistent with what you are seeing?
1 Like
Thank you very much for your feedback. Yes this looks about right and is to be expected. I know where parts of the problem are, and there are some things in testing. stream1090 discards some messages, including position messages. Let me try to explain this in a simple way.
- Stream1090 is very good in inventing ghost messages. The reason for this is that it constantly checks multiple bit sequences for valid Mode-S messages. The question of course is what a valid message is? One can argue with the CRC-sum etc., but this will not work out. Although chances are very low for inventing wrong messages, the amount of ghost messages is considerable when checking 24 mio bit sequences per second.
- To solve this problem, stream1090 has to learn what a valid plane (valid icao address) is. During this learning phase, it will drop valid messages. Or in other words: “Oh, a (possible) valid message from plane XY. It could be a real plane, but i don’t know yet. I will write that address XY down, but the message has to go to the bin.”.
- The proper solution to this problem is to not only write down the address, but also the messages that are currently discarded for this message. Once, the address has been verified, flush the buffer of these messages. However, this requires some clever coding with some more infrastructure which currently simply isn’t there.
1 Like
Emitting messages out of order will cause some messages to be non realtime.
Most current decoders don’t do it.
You’re probably aware of the whitelist setting for airspy adsb.
DF17 no fixed bits is 4, 1 fixed bit is 2.
DF11 (iid 0) no fixed bits is 2, 1 fixed bit is 1.
Default threshold is 5.
Works relatively well, possibly not a bad idea to further reduce DF11 value.
1 Like
@jimMerk2 @obj Thank you guys for the math. It took some time to understand. I got it now why my current implementation works, which basically is
I feel a bit stupid now. HOWEVER, this has some implications. While the current procedure works, it means also that the first Q-sample forms a valid pair with the second I-sample. As a result one can interleave these two sequences. I did some plotting in python and indeed this looks reasonable.
The colors indicate if you start at 0 or 1 (I-sample has an even index or odd index).
What does this mean for stream1090? Full speed ahead. Every raw sample will generate one IQ-pair. So raw 12Msps will be really then 12 mio IQ-pairs. Same for 20 Msps. Does this really work? Yes. Stream1090 is quite flexible and the passthrough samplers were already there anyways, so implemented a quick dirty test version. It works and it makes a difference.
1 Like
Ok, I see what you are saying about the interleaving. However, there’s no lowpass filtering. Apparently that’s not a problem.
Also, in that figure, the peaks seem to be about 34 samples apart. What sample rate is being used there? Thanks.
12Msps raw. Let me do the last groccery run for xmas and then i will quickly setup some proper visualization which i can also put into the repo.
So this took longer as expected and is not finished yet. However, for those who like fiddle around and to look at data, you can now do the following:
- Record some samples with airspy_rx or rtl_sdr.
- Run it through stream1090.
- Pick a random message from the output.
- Use the python script in the repo and get something like this (check the Readme)
If you look at the script, you will quickly see that most of it has been copiloted. I really don’t like writing python.
Edit: Here is a nice example of two overlapping messages:
At the end of the message are the 4 pulses of the preamble of the next message.
1 Like
One note for people who use rtl_sdr:
The above pictures are done with an 10Msps airspy dataset. This of course looks nice, because it goes well with the 2MHz signal (1MHz bits). The same holds for 6Msps. You can see in the second screenshot how nicely 10 samples of the 10Msps fit into one bit marker (1Mhz).
This does not work for 2.4Msps of rtl_sdr for obvious reasons. However, stream1090 does some non-trivial upsampling to 8MHz and you may ask sometimes: What the hell? But it is fine. 2.4Msps just does not properly align with the 1 MHz bits. This is not shown here yet.
Low pass filter is not trivial in this scenario. I would have implemented it if that would be the case. But there are questions to answer first. I am upsampling and then downsample into channels. So just low passing the input might not be the best idea.
Another question (kind of off the wall) – how reliable is the USB connection? That is, can the USB drop samples? If it can, wouldn’t that screw up the IQ relationship? How does stream1090 recover from that? Or maybe the fact that you’re doing I^2 + Q^2 makes it somewhat impervious to dropping samples?
First of all you don’t want to drop samples. This will break locally messages and is also a problem for the MLAT timestamp which relies on all samples arriving. The cheap rtl dongles are therefore limited to 2.4Msps. You can run them faster, but the ADC quality will degrade and samples will be lost.
Regarding the data itself. Yes with I² + Q² you are on the right track. When only interested in the magnitude, you dont care about the sign of I and Q. All you need is that they are phase shifted by 90°. So you will instantly recover once the sample stream is fine again.
Think of it in an easier way: Let’s say you get raw ADC data at a rate of 12 MHz and now consider them with a clock that runs at 3 MHz. You will have 4 samples evenly spaced out around the clock. Clearly two subsequent ones are shifted to each other by 90°. Take the two first ones, and the last two and you have two IQ-pairs (with wrong signs), resulting in magnitude values at a rate of 2 x 3 MHz = 6 MHz .
the python script has been overhauled. You can now specify multiple messages and also multiple sample sources.
There is a small update in the repo for stream1090. This concerns mostly people who do 10Msps → 24 Msps (slightly better upsampling function) and/or people who use the raw mode (still no low-pass filter, but DC removal has been added)
2 Likes
There is one thing that should be mentioned here. @JRG1956 did a great job with the measurements and this 10% has been confirmed from other sources as well. HOWEVER, this does not scale. Once you approach the saturation mark which i think is around 2.5k messages/s, stream1090 gets very close and depending on your setup may take over. This is a matter of statistics. The messages will be packed (overlap) so much that preamble detection will get to its limits.
1 Like
Regarding the message plotting utility: I was able to collect the sample file:
airspy_rx -g 16 -f 1090.0 -a 6000000 -n 1000000 -r - > samples.raw
produced a 4 Mbyte samples.raw file. However: cat samples.raw | ../build/stream1090_6M
produced an error. Should it be: cat samples.raw | ./build/stream1090_6M ?
If so, that produced:
pi@Bookworm:~/stream1090 $ cat samples.raw | ./build/stream1090_6M
[Stream1090] build 251209
[Stream1090] Input sampling speed: 6 MHz
[Stream1090] Output sampling speed: 6 MHz
[Stream1090] Input to output ratio: 1:1
[Stream1090] Number of streams: 6
[Stream1090] Size of input buffer: 768 samples
[Stream1090] Size of sample buffer: 768 samples
But no other output. Any help would be appreciated. Thanks.
Edit to add: Never mind, it helps to turn on the bias-t to collect data:
airspy_rx -b 1 -g 16 -f 1090.0 -a 6000000 -n 1000000 -r - > samples.raw
That captured some data:
@0000000ba1088dac335b585b122db553d1cf49e6;
@0000000eddc88dac335be10e0a00000000f2f218;
@00000011d7c28da11e8b234cb5f3c31d60de1f89;
@000000134c608dac335bea1d687d7f5c080f6e81;
@0000001e5a528da14fbd591da5a5e6176108c766;
@0000001e63f08da14fbd991088847808014f282e;
1 Like
If you want to fool around with the raw samples, here is some snippet which does the interleaving IQ stuff:
import numpy as np
import argparse
def load_iq_u16(filename):
raw = np.fromfile(filename, dtype=np.uint16)
if len(raw) % 2 != 0:
raise RuntimeError("Input file has an odd number of words — not valid interleaved IQ")
I_raw = raw[0::2] & 0x0FFF
Q_raw = raw[1::2] & 0x0FFF
# Convert to float32 in [-1, 1]
I = (I_raw.astype(np.float32) - 2048.0) / 2048.0
Q = (Q_raw.astype(np.float32) - 2048.0) / 2048.0
return I, Q
def compute_magnitude(I, Q):
return np.sqrt(I * I + Q * Q).astype(np.float32)
def compute_overlap_magnitude(I, Q):
N = len(I)
out = np.empty(2 * N - 1, dtype=np.float32)
# even indices: mag(i_n, q_n)
out[0::2] = np.sqrt(I * I + Q * Q)
# odd indices: mag(q_n, i_{n+1})
out[1::2] = np.sqrt(Q[:-1] * Q[:-1] + I[1:] * I[1:])
return out
def main():
ap = argparse.ArgumentParser(description="Convert 12-bit IQ (uint16) to magnitude stream")
ap.add_argument("--infile", required=True, help="Input raw IQ file (uint16 interleaved)")
ap.add_argument("--outfile", required=True, help="Output magnitude file")
ap.add_argument("--dtype", choices=["f32", "u16"], default="f32",
help="Output format: float32 or uint16 normalized")
args = ap.parse_args()
I, Q = load_iq_u16(args.infile)
mag = compute_overlap_magnitude(I, Q)
if args.dtype == "f32":
mag.astype(np.float32).tofile(args.outfile)
else:
# scale to 16-bit unsigned magnitude
mag_u16 = np.clip(mag * 65535.0, 0, 65535).astype(np.uint16)
mag_u16.tofile(args.outfile)
print(f"Wrote {len(mag)} magnitude samples to {args.outfile}")
if __name__ == "__main__":
main()
You can also do the full-speed-ahead mode for stream1090 with -u 24 for upsampling into 24 channels. The script is by now not relying anymore on any samples. For the messages the MLAT timestamp is the one that counts, no matter how you you got that message. The samples are being adjusted to the time domain and are independent of the messages.
After doing the usual fiddling around with installing pip, numpy, matplotlib, I got the script vis_util.py to run successfully. However this is on a headless RPi, ie no graphics, so no plots. Can you save the plot data to a file that can be sent to a desktop machine? Thanks.
1 Like
let me ask my copilot.
Edit: --save my_output.png should work now. However, this is not really a good idea. You should do these things on your non-headless computer, because of the the zoom.
Keep in mind that the vis_util.py is not related to any SDR stuff. You will need only the recordings.