Stream1090

I get similar results to what you got. That is, no samples dropped for rtl_test -s 2560000, but samples dropped for 3200000. This is with either a FlightAware Pro Stick Plus or an RTL-SDR.com V3 dongle.

1 Like

I’ve tried it on a few I have here.
Apart from the initial 100 or so byte drop when starting they were all error free.
The ones I tried were:
RTL-SDR Blog V3,
Nooelec NESDR SMArt v4,
FlightAware Prostick plus (blue) and
FlightAware Prostick (orange).

1 Like

You are absolutely right, it would have been good to see. Unfortunately, I forgot to grab the data. I had my eye on the console for a while, off and on - it felt like a pop but I can’t really say as I was multitasking at work. I might try another run this week and I’ll grab the next run for sure.

2 Likes

@jimMerk2 @LawrenceHill Funny thing: That seems to be the sweet spot. If you try slightly less than like 2.56 MHz. For example sth like this:

rtl_test -s 2550000

I am already getting in trouble.

Exact sample rate is: 2550000.151992 Hz
[R82XX] PLL not locked!
Sampling at 2550000 S/s.
....
lost at least 68 bytes
lost at least 68 bytes
lost at least 68 bytes
lost at least 188 bytes
lost at least 68 bytes
lost at least 68 bytes
lost at least 68 bytes
lost at least 68 bytes
lost at least 68 bytes
lost at least 68 bytes
lost at least 68 bytes
lost at least 68 bytes
lost at least 136 bytes

Why this is interesting? So currently stream1090 has only one option for rtlsdr: take 2.4 MHz and upsample to 8 MHz. The math behind this is the following: I will need an upsampled stream that runs at a multiple of 2 MHz.

To get from 2.4 MHz to 8 MHz, one can compute the greatest common divisor of 24 and 80 which is 8. Dividing 24 and 80 by 8 yields: 3 input samples have to be upsampled to 10 output samples.

One can do that also with 2.56 MHz to 8 MHz. That is, the GCD of 256 and 800 which is 32. This then yields an input : output ratio of 8:25. Annoying, but maybe worth thinking about.

Keep in mind that differential evolution (which is used here) is more or less a guessing game. So once a run has finished (which consists of many iterations), the first iteration of the next run will be the best of all iterations of the previous run. This is on purpose so that from run to run you are drifting away from a good solution.

Maybe it has something to do with 2.55 MHz not being an exact sample rate for the RTL-SDR dongle whereas 2.56 MHz is an exact sample rate. The ADC sample rate is 28.8 MHz and that is decimated by an overall factor of 11.25 to get to a sample rate of 2.56 MHz (exact) whereas for 2.55 MHz the factor is 11.29411765 .

2 Likes

So here is an update:

over the last few months there have been plenty of updates. Everything was cool. Some ideas worked, others did not. However, this growth led to some problems.

Take the InputReader as an example: stream1090/include/InputReader.hpp at main · mgrone/stream1090 · GitHub

The purpose in the beginning was to read in IQ pairs from rtl_sdr and convert them to magnitude. Then airspy came around. Then low pass filtering. Everything very messy. So these parts have to leave the boat. Not in the long run, they have to leave now. The initial idea of the InputReader was to feed the upsampler which in turn then feeds the demodulator.

In practice this works in reverse order. The demodulator requires that amount of samples at that frequency, gets them from the upsampler, which in turn requires then a certain amount of IQ samples (their magnitude) from the layer below and so on.

Or in other words: If you say “i want 10 MHz to 24 MHz” then the input to output ratio will be 5 : 12.

  • The demodulator will split 24 MHz into 24 parallel phase shifted bit streams, but it needs a sample stream of 24 MHz for that.
  • The upsampler (Sampler) knows how to do that. Give it 6 magnitude samples (not 5) and it will create 12 new ones out of this. The 6th sample has to be considered again. However, this then means that the upsampler processes stuff in blocks (which even overlap).
  • It will then tell to the underlying layer: "Guys i need every iteration that amount of new samples, otherwise things get crazy.
  • So basically the upsampler dictates to everyone how much stuff they have to process in one batch.
  • This then propagates downwards: All the way to where you get actually the raw samples from.

Improving this part of the code of stream1090 has to be done before implementing a full native version for airspy/rtlsdr. Improving means to make this more maintainable. Of course you may ask know: Why are you not cutting this dependency between layers? Answer is simple: Because the compiler is smart and knows what to do. There is a difference between

for (int i = 0; i < 1234; i++) {}

and

for (int i = 0; i < i_am_going_to_ask_someone_at_runtime; i++) {}

In the former case the compiler knows what the loop is doing, in the latter case, well it depends.

Bottom line: the critical stuff for having native support for airspy and rtlsdr being ready, requires the current pipeline to undergo some serious reengineering.

However, C++ is very powerful. Stuff like this compiles into something really fast.

    using RawInputType = uint16_t;
    using Sampler = Sampler_10_0_to_24_0_Mhz;
    using LowPass = IQLowPass<Sampler::InputSampleRate, Sampler::OutputSampleRate>;

    auto iqPipeline = make_pipeline(DCRemoval(), FlipSigns(), LowPass());

    InputStdStreamReader<RawInputType, Sampler, decltype(iqPipeline)> inputReader(file, iqPipeline);
    
    SampleStream<Sampler> sampleStream;
    sampleStream.read(inputReader);

It tells basically: Read unsigned int 16 IQ pairs, convert them to float, run them through dc removal, flip their signs in an alternate fashion to get ready for the low pass. The compiler will put all these things together at compile time. It is very powerful and fast.

1 Like

thats plenty … pretty sure someone was using not enough bits somewhere…

Okay, this curiousity has been continously in my mind since we talked about it, so I ran a run of 1,000 tests against a 15-minute sample I had so I could visually view what the results looked like. Hope I’m thinking about all of this accurately. :slight_smile:

Looking at Score and Total for each run, I see the following results. The diamonds are my highest results as reported by the script.

The script is now at the 1,200 mark in the iteration count, and there hasn’t been any new best scores since stream1090 iteration 930 - 6 hrs 57 minutes into testing. I feel that this visualization has helped me a lot - fuel for thought perhaps.

2 Likes

The optimizer has its limits. You can try other tap sizes. But i cannot say much there.

Thanks for the comment, @mgrone Yes, I agree with you. Just to be clear - it wasn’t a knock on the optimizer at all, but actually, I saw it as a good thing that the results went from fairly wide spread to pretty finite.

You read my mind on additional taps, that is my next test and I’ll kick off for an overnight run here in a few. :slight_smile: I have not tried to create 32 taps with the optimizer but I have created a great set (with message improvement and minimal CPU usage increase for me) with CoPilot. (CoPilot does not like the idea of 64-taps with stream1090 at all.) :man_shrugging: I’ll reply tomorrow with what that test yields!

1 Like

Another update:

The weekend is coming up and i hope that i can push some stuff to play with. Most of this will be highly experimental. Expect problems. Current status:

  • native airspy support (done, a bit of testing)
  • native rtlsdr support (done, no testing)
  • ini config files for the native support (nearly done, a bit of testing)
  • compile your own taps into the binary (done, not much to test there, works)
  • writing documentation on all the above (0% done)

Regarding the ini-files: you will not have a parameter mess for this. Something likes this:

# Airspy configuration
[airspy]
serial = 0x1234567890ABCDEF 
frequency = 1090000000
linearity_gain = 20
bias_tee = false

It will expose much more and will follow airspy_rx in this regard. Serial is of course optional. rtlsdr has something similar.

Regarding the custom taps. The custom filter directory will have a file called OverrideFilterTaps.hpp. You can do things like this there:

STREAM1090_OVERRIDE_TAPS(
    Rate_6_0_Mhz,
    Rate_12_0_Mhz,
    0.1544143,
    0.262295,
    0.18557717,
    0.262295,
    0.1544143,
)

This will then override the default 6 → 12 MHz filter. The idea is that also people with no c++ knowledge can add their own taps and compile them into the binary. You then just rebuild the project with make. The compiler will check if something has been defined there, otherwise will fall back to the default taps.

So what are the bad news: I will remove some features. Some input formats will have to leave the boat.

One thing that i still haven’t decided is if i will split executables again. While one executable for all sounds cool, i do not see really the purpose right now. You either run it on an rtlsdr based dongle, or on an airspy. You have to understand that the complete DSP pipeline is being constructed by the compiler. So there will be presets. That means that the average rtlsdr user will have all the airspy DSP pipelines in the executable (not the native support of the device).

What is left to do?

  • finish the main.cpp
  • write the code for the default read from std in. This has to be done from scratch.
  • Readme (welp)

Edit: why this ini-file stuff for the devices? So the stuff is implemented that most of the key-value pairs are being understood as commands. The airspy can be running and you still send it “vga_gain=4”. This is a feature that me and also other people would like to have in the future. I hope that works out somehow.

2 Likes

Thanks for this - it is a really good way of showing the results.

When I did my optimisations, I usually stopped after about 20 iterations, and your results made me wonder if I should follow your example and do a very long run. The thing I find interesting is that the optimiser had achieved a score of 1,850,000 within the first ten iterations (as best as I can read the graph) and the best score is 1,856, 461. So the net improvement for all of those hours of processing was 0.35%

Given the variability between samples, I think most people could get by with a reasonably small number of iterations.

1 Like

By the way: these taps do not have to be symmetric. Just saying. The code will check and if they are not, treat them as such. However, you have to always make sure to create a low pass style filter. I managed to get your dataset all the way up to nearly 4k messages until i noticed that the taps represent a band filter and they delay stuff so much that some messages are being counted twice. stream1090 has some duplicate removal, but only within the same phase.

So the main.cpp that puts all the parts together is done (from scratch). It was a wild ride. The reading from stdin is also working again.

Both native devices had some testing. But there is here and there something missing. Testing this stuff does not only involve running the things, but also get out of your chair, pull the usb plug and see what is happening (btw on the todo list to fix that).

I also did some performance benchmarks. The big question now is “Whoa no more piping! Yea! How much do we get??” There is a short and a long answer, i will try to combine them:

  • If you are running 10 MHz → 10 MHz, no low pass filter you will gain a very little bit.
  • If you are running 6 MHz → 24 MHz: If i have time i may find a way to measure it :smiley:

Bottom line: it gives you control over the device, plenty of work, but you will not gain anything in performance. Will the option to read from stdin go away at some point? No. This offers easy feeding of data. It also does not constrain the demodulator to be on the same machine as the SDR machine. You can socat your way. SDR → rtl_sdr(airspy_rx) → socat → … → socat → stream1090 → socat … → socat → readsb (dump-fa)

So what is left to do:

  • Priority (Building): Everything has to work without the dev versions of libairspy or librtlsdr around.
  • ReadMe

This brings me to main point of this post: Currently you will get 3 executables when building stream1090.

Should it stay that way?

  1. Answer: No, one executable to rule them all
  2. Answer: stream1090_rtlsdr and stream1090_airspy,
  3. Answer: As it is. stream1090 = rtlsdr, stream1090_6M and stream1090_10M

I am thankful for some input on this matter.

One binary it is. All changes are in the repo now. If you want to check out native support you will need

sudo apt install libairspy-dev 

for airspy and similarily for rtlsdr based dongles

sudo apt install librtlsdr-dev

cmake will look for these automatically. The readme hasn’t been properly updated. However,
stream1090 -h should explain a bit:

Stream1090 build 260207
Native device support: Airspy RTL-SDR

Usage:
  stream1090 [options]

Options:
  -s <rate>            Input sample rate in MHz (required)
  -u <rate>            Output/upsample rate in MHz
  -d <file.ini>        Device configuration INI file for native devices
                       See config/airspy.ini or config/rtlsdr.ini
                       Note that native device support requires librtlsdr-dev
                       and/or libairspy-dev to be installed.
  -q                   Enables IQ FIR filter with built-in taps (default or custom)
  -f <taps file        Taps to load that are used for the IQ FIR filter
  -v                   Verbose output
  -h, --help           Show this help message

Supported sample rate combinations:
  2.4  →  8 (uint8 IQ)
  6  →  6 (uint16 IQ)
  6  →  12 (uint16 IQ)
  6  →  24 (uint16 IQ)
  10  →  10 (uint16 IQ)
  10  →  24 (uint16 IQ)

Examples:
  rtl_sdr -g 0 -f 1090000000 -s 2400000 - | ./build/stream1090 -s 2.4 -u 8
  ./build/stream1090 -s 2.4 -u 8 -d ./configs/rtlsdr.ini

  airspy_rx -t 4 -g 20 -f 1090.000 -a 12000000 -r - | ./build/stream1090 -s 6 -u 12 -q
  ./build/stream1090 -s 6 -u 12 -d ./configs/airspy.ini -q
2 Likes

Have got the new version up and feeding.

Before:

After (edited to make more legible):

Are you still intending to provide an option to compile the custom filter at some stage?

1 Like

Oh sorrry, i haven’t updated the readme yet. But it is already there in the custom_filters folder:

1 Like

@JRG1956
One note: with the -v option (verbose), you will see the taps being used in the built-in IQ pipeline (-q option)

1 Like

Thanks the verbose mode works well with the service log too. I have the compiled filter version up and feeding.

jrg@pi52:~/projects/stream1090 $ sudo systemctl start stream1090
jrg@pi52:~/projects/stream1090 $ sudo systemctl start piaware
jrg@pi52:~/projects/stream1090 $ sudo systemctl status stream1090
● stream1090.service - stream1090 service
     Loaded: loaded (/usr/lib/systemd/system/stream1090.service; enabled; preset: enabled)
     Active: active (running) since Sun 2026-02-08 12:34:46 AEDT; 30s ago
 Invocation: 5c037042f0f94332ae48758dff85b26b
   Main PID: 2643104 (stream1090.sh)
      Tasks: 6 (limit: 2320)
        CPU: 13.615s
     CGroup: /system.slice/stream1090.service
             ├─2643104 /bin/bash /home/jrg/projects/stream1090/stream1090.sh
             ├─2643107 /usr/bin/stream1090 -s 10 -u 24 -d /home/jrg/projects/stream1090/configs/airspy.ini -v -q
             └─2643108 socat -u - TCP4:localhost:30001

Feb 08 12:34:46 pi52 stream1090.sh[2643107]: [Stream1090] Size of input buffer: 15360 samples
Feb 08 12:34:46 pi52 stream1090.sh[2643107]: [Stream1090] Size of sample buffer: 36864 samples
Feb 08 12:34:46 pi52 stream1090.sh[2643107]: [DCRemoval] alpha: 0.005
Feb 08 12:34:46 pi52 stream1090.sh[2643107]: [FlipSigns] enabled
Feb 08 12:34:46 pi52 stream1090.sh[2643107]: [IQLowPass] tap count: 15 symmetric: 1
Feb 08 12:34:46 pi52 stream1090.sh[2643107]: [IQLowPass] taps: {0.000566716, 0.020335, 0.00571821, -0.0545961, 0.0605642, 0.167134, 0.162813, 0.27493, 0.162813, 0.167134, 0.0605642, -0.0545961, 0.00571821, 0.020335, 0.000566716}
Feb 08 12:34:46 pi52 stream1090.sh[2643107]: [Stream1090] Async Device Mode
Feb 08 12:34:46 pi52 stream1090.sh[2643107]: [Stream1090] Device created.
Feb 08 12:34:46 pi52 stream1090.sh[2643107]: [Stream1090] Device successfully configured.
Feb 08 12:34:46 pi52 stream1090.sh[2643107]: [Stream1090] Device is running.