Stream1090

Here is more for you. Python script that crawls through the message stream.

import sys
import rs1090

planes = {}
msg_per_plane = {}
numMsgs = {}

for line in sys.stdin:
	try:
		if (not line.startswith('@')):
			continue

		curr_time = int(line[1:13], 16)
		line = line[13:].strip(';\n')
		
		res = rs1090.decode(line)
		if (res == None):
			binary_message = format(int(line, 16), f'0{len(line)*4}b')
			print("ERROR " + binary_message[:5] + " " + line[2:8] + " " + binary_message[33:38])
			continue
		
		# remove this for stats only
		print(res)
		
		df = res['df']
		if ((df == '17')):
			p = res['icao24']
			if not (p in msg_per_plane):
				msg_per_plane[p] = 0
			msg_per_plane[p] += 1
		
		if not (df in numMsgs):
			numMsgs[df] = 0
		numMsgs[df] += 1
	except:
		print("error: " + line)

totalCount = 0
for df, cnt in numMsgs.items():
	totalCount += cnt
	print("DF " + df + " " + str(cnt))

for icao, cnt in msg_per_plane.items():
	print(icao + "\t" + str(cnt))

It uses rs1090 which you can install via pip. It simply counts the DF-17 per plane and makes stats about the other messages. rs1090 offers way more. You can adjust as you like. It also shows you how to convert a message into a binary string so you can have a closer look. Although rs1090 can do most of these things.

You can record sample datasets by piping them into a file. For testing i am using the raw output of rtl_sdr and not the messages. I can then say:

cat output_24_13.raw | ../stream1090/build/stream1090 | ../python/bin/python msg_stats.py

Thereby dumping the raw iq samples into stream1090 and its output is then run through the python script. Should work with airspy_rx and corresponding stream1090 settings the same way.

1 Like

So here is a longer reply regarding these bogus messages and planes. Stream1090 is a demodulator which i personally use to feed readsb which then serves as the decoder. I have to draw at some point a line in the sand and say: This is something that has to be taken care of by the decoder. I think that is understandable. I have implemented some logic that prevents forwarding crappy messages. However, this already requires me to implement some decoder logic.

Here is now the catch. Stream1090 is very aggressive in how it is demodulating things aka framing messages. For airspy hardware it could in theory with a bit of luck frame two messages from different planes at nearly the same time. And yes, i have such an example. The difference here is that messages are being framed in parallel whith a slight phase shift. Running an internal speed of 24 MHz means 24 shift registers in parallel are trying to make sense of 0s and 1s at a speed of 1MHz each. At this speed, the chance that a combination of 0s and 1s makes sense and the CRC sum is also correct gets quite high. Keep in mind that if that code would be only doing 99.99% correct, it would be an absolute diseaster. 0.01% failure is equal to 2400 msg/s of crap.

On top of this, there is CRC error correction. Compared to other software, the current state seems aggressive, but i have used versions in the past that were not public that applied a way more aggressive strategy. Will this create bogus messages? Yes. It will however also create a huge amount of usable messages.

Edit: my numbers were wrong. I was thinking about 2.4Mhz not 24MHz. So 0.01% = 2.4k crap/s

Thanks for the python script - I will have a play with it when I have time.

Re the bogus planes/messages - I think your current settings are fine.

If I’ve understood correctly, the bogus message rate is independent of the actual number of real messages being received because they are being generated out the background noise. As I have low real message rates, the proportion of bogus messages is higher, but not the absolute number being generated.

1 Like

A very easy way to check if an icao address is valid and in range is going for some site that uses tar1090. Something like this: https://globe.adsbexchange.com/?icao=40749b just replace the icao address.

1 Like

Hello everyone,

just a quick update what is going on currently. After a short break, I got my hands on an airspy which makes it a lot easier for me to implement things for the airspy side of things. Currently i am testing some new code that takes the raw output of airspy_rx as input instead of the default INT16_IQ.

4 Likes

Thankyou for this been waiting on rtlsdr for couple weeks now :slight_smile:

the raw mode (-r flag) update is in the repo together with some other changes. Let me now if you experience some zombie stuff, because i made some changes to the cache timeouts.
The raw mode should greatly reduce airspy_rx cpu usage. There are two shell scripts that outline the differences.

It is me again. Regarding the raw input. So i talked a bit with @wiedehopf who provided me with some extra tool for some benchmarking. Thank you for that. I came to a preliminary conclusion after some thinking without doing a week long of testing. For this it is important to know some technical details how things work.

  • airspy_rx can provide different output formats. Float, uint16, int16, raw etc at different speeds. In the end everything is an IQ-pair. That is two values always form a pair. The default is int16 IQ. So if you say: Sample at 6Msps, then you will get 6 Mio pairs per second. If you say: 12Msps raw pls, then airspy_rx will give you first an I, the next value will be the Q. So basically 6Msps IQ-pairs. Raw mode does not magically increase the sampling speed.
  • What is not directly obvious to a user of airspy_rx is that for some formats (including the default int16 IQ) some magic is applied. Not really magic, but a band pass filter is applied and some DC removal takes place.
  • However, raw is really a passthrough and therefore you will get some good reduction in CPU time. No magic is taking place and therefore no band-pass filter.

So what? The thing is the following: I have a setup with very little interference. The LNA is directly hooked up to the antenna which is a high gain narrow bandwidth directional. From there on i have 30cm (1ft) of nearly osciloscope grade coax to the airspy. It could be that for me this band-pass filter does not matter. It may however matter for you. So you have to do some testing, i will later push a tiny update for the non-raw version. You have to figure out:

  • Plenty of interference and you really need this band-pass filter. Then use for now the non-raw version.
  • Few interference and you are too lazy as me to install a cooler on your RaspberryPi 5 and want it sitting there at 1.7GHz, then use the raw mode.
2 Likes

@JRG1956 @mgrone
To continue the discussion on airspy from the other thread, take a look at the following paper: How RTL-SDR dongles work
I think the Airspy approach is closest to what is in Figure (b) shown below:


Notice that there is one A/D and that the quadrature mix is performed digitally (where the mixes become multiplies and the lowpass filters are FIR filters). This is a much better than an analog approach shown in Figure (a). The analog implementation has inherent imbalances in the IQ arms. This produces images at the negative frequency of the desired signal. Also the digital implementation has simplifications when the if freq = fs/4 where fs is the sample rate as I mentioned in an earlier post. Definitely want to go digital (DSP).

Thanks for making the effort to explain. I hadn’t read enough to know that the digital approach was an option.

Airspy is close to (b) except that the quadrature mixing is happening on the host side; what is transferred over USB is the raw ADC data.

I haven’t checked, but I assume that the “raw” format that the airspy host library can produce is just the raw ADC stream without the quadrature mixing.

For ADS-B, doing the quadrature mixing is useful because you basically want to do envelope detection on the received signal, and that is extremely easy to do once you have the input in quadrature form (just take the vector magnitude of each IQ value). But there are other ways you could do envelope detection from the raw ADC data.

The LPFs in the quadrature mixing stages are to avoid aliasing problems rather than to really clean up the signal at all. The IQ signal, in theory, has the same amount of information as the raw ADC data, just frequency-shifted.

2 Likes

Thank you. That’s the answer to the question I have been asking for the last three days.

1 Like

fwiw, here’s the host-side code that does the sample conversion for various different modes: airspyone_host/libairspy/src/airspy.c at c6721000f19601512e9ba6b0340e5d9ced22a900 · airspy/airspyone_host · GitHub

Although it’s the ADC samples that are sent over the USB bus, you could say for the special case of IF Freq = Fs/4 where Fs is the sample rate, that if you take two consecutive ADC samples, a pair of IQ samples are obtained That is the two streams are in quadrature. But you still need to rotate to 0 freq and do the low pass filtering though (in the RPi). So I don’t see the advantage to that approach. I am probably missing something here.

AIUI, the special case is always used, so if the sample rate is 20 MHz, the IF is 5 MHz, if the sample rate is 12 MHz the IF is 3 MHz.

Aren’t the lowpass filters used to filter out the IF freq (5 MHz for example for 20 MHz sample rate) ?

This is a bit awkward to explain without a bunch of diagrams, but here’s an attempt (and this is just one way of thinking about it, there’s a few different ways of explaining it that end up at the same result):

Let’s say you sample real values at 20MHz. You can interpret that in the frequency domain as if it had frequency components at 0..10MHz (the signal you want), and a mirror image at 0..-10MHz. i.e. this is what you’d get if you did a DFT on your input time-domain data to get frequency-domain values. The mirrored frequencies are because the input is real-valued only (that’s just how the DFT behaves with real-valued input, the negative frequencies are required to cancel out any imaginary values that would otherwise appear)

It’s also useful to remember here that while conventionally a DFT is going to produce values for frequencies in the range (-10..+10), you could equivalently treat it as if it was (+10..+30) or (-30..-10) or any other multiple-of-20MHz offset; numerically, plugging any of those aliased frequencies into a complex-exponential expansion would produce the same time-series data.

Now, you multiply your time-domain data by a complex-valued signal with frequency -Fs/4, which (and this is why you pick that magic frequency) is just this simple repeating sequence:

1 + 0j
0 - 1j
-1 + 0j
0 + 1j

Multiplying by -Fs/4 in the time domain is equivalent to convolving by -Fs/4 in the frequency domain, i.e. it will shift all the input frequencies by -Fs/4.

This multiplication is, I suspect, where the idea of “you can just interpret it as pairs of IQ samples” comes from – that’s not really what’s happening here though. The result of this multiplication is, conveniently, a stream of complex values where every second value is real-valued-only and the other values are imaginary-only, which lets you further optimize calculations later because there are lots of zeros that you can skip over.

When you shift that spectrum by -Fs/4 i.e. -5MHz by multiplying by a complex-valued oscillator at -Fs/4, you end up with this:

the signal you want at (0..+10) shifts to (-5..+5) (this is what you want to end up in the I/Q data)

the mirrored signal at (-5..0) shifts to (-10..-5); you don’t want this, it’s duplication of the signal

the mirrored signal at (-10..-5) has an alias at (+10..+15), which shifts to (+5..+10); you don’t want this either.

The LPF discards those mirrored signals at >5 and <-5MHz, since they’re not really useful when you’re working directly with complex IQ data, you only want the one copy of the signal and not the mirrored versions.

Finally, since you’ve just applied a LPF that threw away half the bandwidth, and there is now little-to-no energy <-5 or >+5MHz, you can also safely decimate-by-2 by just discarding every second sample. Or, equivalently, you can just calculate every second output of the LPF.

If the LPF happens to be a halfband FIR, then it turns out that the structure of the halfband FIR constants together with the alternating-real-and-imaginary values from the original -Fs/4 multiplication means that you can just apply a modified filter to the real-only values, and a simple delay to the imaginary-only values. This looks really weird if you just look at the final code, but it really is doing a full LPF-and-decimate-by-2, it’s just that there are a lot of zeros that can be ignored along the way.

If you didn’t apply the LPF and decimate-by-2, then the resulting baseband signal would be at twice the sampling rate you needed, and the extra bandwidth would be taken up with those mirror-images of the signal you want. Usually you don’t want that.

If you didn’t apply the LPF but did decimate-by-2 anyway, then reducing the sample rate would make aliases of the mirror-image copies overlap and interfere with the signal you want. This is approximately what “just interpret the real-valued samples as I/Q pairs” would do.

(I don’t know if that helped!)

4 Likes

Yes that helped. Thanks for the effort to explain it. I’m going to pick up the Lyons DSP book that you recommended in another thread a few years ago. The diagrams help a lot.
Thanks again.

Just to make sure I understand this:
The in-phase mixer is just the sequence: 1 0 -1 0
You have denoted the quad-phase sequence using the j operator: 0j -1j 0j 1j
But that’s just: 0 -1 0 1 and that sequence is just shifted 90 deg from in-phase.

So really the operation is:
take an ADC sample that’s the first I sample
take the next ADC sample and invert it that’s the first Q sample
take the next ADC sample and invert it that’s the second I sample
take the next ADC sample that’s the second Q sample

Hence “the ADC provides IQ pairs” – kind of, but not exactly.

1 Like

There’s the “not exactly”. They are not paired; they are separated in time.

If you have real-valued ADC input a(t) = a(0), a(1), a(2), a(3), … sampled at Fs; then after multiplying by -Fs/4 you have a complex-valued baseband signal b(t) also sampled at Fs:

b(0) = a(0) + 0j → I=a(0), Q=0
b(1) = 0 - a(1)j → I=0, Q=-a(1)
b(2) = -a(2) + 0j → I=-a(2), Q=0
b(3) = 0 + a(3)j → I=0, Q=a(3)
etc.

Ah, I forgot that: every other I value is a 0 and every other Q value is a 0.

Thanks.

If I is 0 and Q is 0, then IQ is 0. Who is this guy with zero IQ? :wink: