Stream1090

[mgrone]

That is some very interesting sample set. I don’t know what the heck is going on, but for sure not Comm-B:

cat ../../samples/cnuwer_10msps.raw | ../build/stream1090_10M -r -u 24 > /dev/null 
[Stream1090] build 260103
[Stream1090] Input sampling speed: 10 MHz
[Stream1090] Output sampling speed: 24 MHz
[Stream1090] Input to output ratio: 5:12
[Stream1090] Number of streams: 24
[Stream1090] Size of input buffer: 15360 samples 
[Stream1090] Size of sample buffer: 36864 samples 

-------------------------------------------------------------
|     Type |  #Msgs |  %Total |    Dups |   Fixed |   Msg/s | 
-------------------------------------------------------------
|    ADS-B |  17663 |   24.5% |   58.4% |   44.1% |     589 | 
|   Comm-B |     68 |    0.1% |   17.1% |         |   2.267 | 
|     ACAS |  23502 |   32.7% |   28.6% |         |   783.7 | 
|     Surv |   3385 |    4.7% |   33.6% |         |   112.9 | 
|    DF-11 |  27358 |     38% |   61.2% |   33.4% |   912.2 | 
-------------------------------------------------------------
|  112-bit |  18402 |   25.6% |   57.8% |     43% |   613.6 | 
|   56-bit |  53574 |   74.4% |   50.2% |   21.9% |    1786 | 
-------------------------------------------------------------
|    Total |  71976 |    100% |   52.4% |     28% |    2400 | 
-------------------------------------------------------------
(Max. msgs/s 2400)
Messages Total 71976

[caius]

I typically get just under 3000 msgs/s in winter so will try and grab a sample when its busy.

[cnuwer]

Hmm, that is interesting for sure. To be honest, with the past holidays and all I haven’t been looking at this with any granularity. Other than applying Github repo updates, I’ve just been looking at each side of the splitter to see how the Stream1090 box is doing compared to airspy_adsb box on graphs1090. (which is very, very close for me)

[mgrone]

I updated the repo with the latest coefficients. 10 Msps is now 33 taps. No changes to the 6 Msps for now.

[mgrone]

It would be nice to get a RAW sample from you. I dont care too much about the msg rate.

[cnuwer]

@cnuwer

I’ve just been looking at each side of the splitter to see how the Stream1090 box is doing compared to airspy_adsb box on graphs1090. (which is very, very close for me)

Completely in the “for what it’s worth bucket” I wanted to share some insight on my previous comment earlier - as they say a picture is worth a thousand words. I exported stats out of the RRD files on both my airspy and stream1090 boxes for the last 24 hours and threw them together in Excel. Sure the formatting could certainly be better but just a quick example view of what I see from my antenna - as fuel for comparision & thought. Really impressive to me how close stream1090 is when compared to airspy_adsb.

[mgrone]

Thank you very much for the feedback. That is encouraging. It is getting there. I will investigate this Comm-B issue tomorrow. This should be a software problem on my side.

[cnuwer]

Sure thing, no problem at all. Thanks for taking a look into it.

[mgrone]

I did some debugging the other day. The situation was tricky. Two messages sent nearly at the same time in your data set. Something like this:

Those are two valid Aer Lingus planes overlaying each other with a DF-11 messages. The case i was considering was some Ryanair pair. Took me some time to get to the bottom of it. I asked wiedehopf about the time when he got that sample from you. The plane (including its transponder code) was retired after that (some years ago).

Edit: not clear on the screenshot that these are two messages.

Edit2: Ignore the green sample line. Leftover of some optimization earlier.

[mgrone]

Just in case someone wants to know “How can you separate those two messages” It is the stream1090 idea. Upsample to heaven (24MHz heaven), run CRC-based framing on each of the 24 streams.

[JRG1956]

@mgrone

I updated the repo with the latest coefficients. 10 Msps is now 33 taps.

I just ran the previous and current version of stream1090_10M -r -u 24 on a 48 billion raw sample file. The message rate in the sample was around 200/sec.

There was a very slight improvement- +0.04% messages and +0.07% positions.

I would like to compare the performance using the IQ option - will the python snippet you posted earlier the thread convert the raw sample to one the can be used with stream1090 -u 24?

[mgrone]

Ok, fair point. I will release the filter stuff.

[mgrone]

It is in the repo. I will fix the stuff with the “checkpoint”

[mgrone]

You want to remove DC with the script. From there on you can use the opt filter script.

Edit: The remove_dc.py will take its time. It is python. It takes some time.

[JRG1956]

@mgrone

It takes some time

Thanks - I will give it try on the Pi5 but will need a bigger SD card. The sample file is 72GB and I assume the converted one will be 96GB

I did try to run the code on my MacBook Pro - it sat there for quite a long tome, then the Mac crashed.

[mgrone]

You are doing something really wrong.

-rw-r--r-- 1 martin martin 1436024832 Apr 10  2025 ../../samples/samples_5m_02.raw
-rw-r--r-- 1 martin martin 1436024832 Apr 10  2025 ../../samples/samples_5m_03.raw

< 1.5 GB So i am not sure what you are doing there

[JRG1956]

@mgrone

You are doing something really wrong

Perhaps silly rather than wrong.

I wanted a really long sample period. My file is 48 Giga-samples, and a sample is 12 bits.

So the file size is 48 x 12bits / 8 bits per byte = 72GB

The time period is 48 Giga samples = 48000 Mega samples / 20 MSPS = 2400 seconds = 40 minutes.

I use the following command to “playback” the sample through stream1090 into readsb and then look at the stats.json readsb produces.

cat samples48G.raw | /usr/bin/stream1090_10M -r -u 24 | socat -u - TCP4:localhost:30001

It takes about 16 minutes to process the 40 minute sample, so the aircraft tracks are sped up in tar1090.

[mgrone]

So with the investigation things took a bit longer. The problem is not solved, but it seems i am not alone with this. Here is a general question for people who know about these things:

I am based in Europe and used to have plenty of long Comm-B and Surveillance replies. Especially Comm-B offers additional data. If i now go to the U.S. on some tracking site. There is plenty of data missing. Here is some random plane.

This is not an adsb.lol thing:

If i now take an American plane flying in Europe, i get this:

This data (i think) has to been requested by secondary ground radar. Is this simply not done in the U.S.?

Edit: Or is this a coverage problem?

Edit 2: Apparently there is a coverage problem in Europe. Too many stations requesting too much data. Here is an interesting article from EUROCONTROL on this matter from a few years ago.

EUROCONTROL Data Snapshot #10

[obj]

Yes, SSRs in the US tend to interrogate for fewer types of Comm-B data

You can see the effect if you go to FlightAware ADS-B Coverage Map - FlightAware, switch to Data Coverage, and enable the “weather” option (the magenta layer below)

Weather-data coverage requires more Comm-B data types; you can see how this varies a lot by location, and there are only a handful of SSRs (with very clearly defined coverage areas) in North America that are interrogating for that data, with very little coverage in the US.

Brazil is similar.

[mgrone]

So quick status update.

The low pass filter is fine, but it is not as good as i was hoping for. There are several reasons. The two main reasons are:

1. Currently (in the repo version) the filter is applied to the magnitude stream and has been optimized using some python script. The location of the filter is a bad idea. Why did i put it then there in the first place? I have very few knowledge in DSP or in other words, because i am stupid.
2. The performance of the filter does not only depend on the input speed, but also on the upsampling rate as experiments show. It is very tedious to find good filters for all the sample combinations.

So what is next. I had quite some time on my hand during the weekend to read and write code. The plan is to retire this current low pass filter approach and raw mode (-r) will work as in the previous version (just raw, no filter).

However, i will introduce the -q option. This will put stream1090 into raw-iq mode. I and Q values will be properly handled. Separate DC-removal, sign adjustments and then they will be sent independently through two identical low pass filter. Afterwards, they will go through the usual pipeline in stream1090 by computing the magnitude of the pair.

However, i have some cleanup left to be done. Today i did some testing, but the day was not really good for testing (high pressure system, weekend, sun was out).