All times are UTC - 6 hours [ DST ]




Post new topic Reply to topic  [ 182 posts ]  Go to page Previous  1 ... 4, 5, 6, 7, 8  Next
Author Message
 Post subject: Re: Gain Adjustment...
PostPosted: Tue Nov 22, 2016 8:02 am 
Offline
FlightAware Staff
obj - FlightAware user avatar

Joined: Tue Sep 30, 2014 7:14 pm
Posts: 3203
piaware-config.txt is in /boot


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Tue Nov 22, 2016 4:06 pm 
Offline
WarrnamboolCity - FlightAware user avatar

Joined: Wed Apr 06, 2016 6:23 pm
Posts: 29
Location: Warrnambool, Victoria, Australia
These are the only files I can see in /Boot
bin boot.cmd initrd.img-3.4.112-sun8i uInitrd
bin.old boot.scr script.bin vmlinuz-3.4.112-sun8i
boot.bmp config-3.4.112-sun8i System.map-3.4.112-sun8i zImage

I used this: find / -name 'piaware-config.txt' to no avail but dropping off the .txt revealed /usr/lib/piaware-config and
/usr/bin/piaware-config
One was jibberish and the other contained nothing.


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Tue Nov 22, 2016 4:35 pm 
Offline
FlightAware Member
abcd567 - FlightAware user avatar

Joined: Sat Feb 15, 2014 3:08 am
Posts: 1862
Location: Toronto CYYZ
@WarrnamboolCity
The file /boot/piaware-config.txt exists only if you have written a Piaware 3 img to your card. Since you have written Armbian, then installed separate packages of dump1090-fa and Piaware 3 (add-on), the piaware-config.txt will not be there. You have to edit another file to adjust gain:

Code:
#open file dump1090-fa in a text editor to change value of gain
pi@orangepipc:~$ sudo nano /etc/default/dump1090-fa

#after editing the gain value and saving above file, restart dump1090-fa for new gain value to take effect.
pi@orangepipc:~$ sudo systemctl restart dump1090-fa

#now check status of dump1090-fa.
#it will show, among other things, the gain value set.
pi@orangepipc:~$ sudo systemctl status dump1090-fa -l



.


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Tue Nov 22, 2016 5:16 pm 
Offline
WarrnamboolCity - FlightAware user avatar

Joined: Wed Apr 06, 2016 6:23 pm
Posts: 29
Location: Warrnambool, Victoria, Australia
@Abcd567
Thanks for that. Now modified and showing up as suggested.
Thanks,
Paul


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Tue Nov 22, 2016 6:02 pm 
Offline
FlightAware Staff
obj - FlightAware user avatar

Joined: Tue Sep 30, 2014 7:14 pm
Posts: 3203
Oops, missed that you weren't running the standard image, sorry about that.


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Tue Nov 22, 2016 8:02 pm 
Offline
FlightAware Member
abcd567 - FlightAware user avatar

Joined: Sat Feb 15, 2014 3:08 am
Posts: 1862
Location: Toronto CYYZ
obj wrote:
Oops, missed that you weren't running the standard image, sorry about that.

It is very natural to make this mistake because people normally say "Piaware" for both the image as well as data feeder. I have rarely seen anyone using full names "Piaware img (SD card)" and "Piaware (add-on).

It would have been much better if these two had distinct names. An example is Flightradar24 software. Their img is named "Pi24" and the data-feeder add-on is named "fr24feed".


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Tue Nov 22, 2016 8:29 pm 
Offline
FlightAware Staff
obj - FlightAware user avatar

Joined: Tue Sep 30, 2014 7:14 pm
Posts: 3203
20/20 hindsight..


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Wed Nov 30, 2016 12:53 am 
Offline
Boeings - FlightAware user avatar

Joined: Sun May 29, 2016 2:09 am
Posts: 25
Location: Sydney
Hi, what does the red grass in the performance graphs mean, legend = Messages > -3 dBFS/hr.

Does more red mean there are more strong messages, or more weak ones?

Thanks, B


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Wed Dec 14, 2016 6:27 pm 
Offline
wittrup58 - FlightAware user avatar

Joined: Sun Sep 06, 2015 5:28 am
Posts: 19
Forgive me for not reading the last 3 pages thoroughly, yet.
Inspired by this post and the work behind it, I stared modifying...

I have ended with a script that runs on both:
Code:
Python 2.7.9 (default, Sep 17 2016, 20:26:04) [GCC 4.9.2] on linux2

and
Code:
Python 3.5.2 |Anaconda 4.2.0 (64-bit)| (default, Jul  5 2016, 11:41:13) [MSC v.1900 64 bit (AMD64)] on win32

Latter one, obviously without the actual gain changing...

New features:
  • Formatting output, possible to select delimiter, default (tab)
  • Timeout, will not wait for socket receive if time is passed, needed in target poor environment
  • Handling packets directly, should be more memory efficient if run over a long time
  • Arguments parser, script now takes arguments for behaviour modification, help gives this:
Code:
pi@piaware:~$ pyh/optimize-gain-piaware3.py -h
usage: optimize-gain-piaware3.py [-h] [-t TESTTIME] [-n TESTSNUM] [-s SERVER]
                                 [-p PORT] [-g GAINS] [-c CONFIG] [-r GREPSTR]
                                 [-v] [-d] [--delim DELIM]

optional arguments:
  -h, --help     show this help message and exit
  -t TESTTIME    Seconds for each test
  -n TESTSNUM    Number of tests to perform
  -s SERVER      hostname for data feeder, default 'localhost'
  -p PORT        port, default 30003
  -g GAINS       list of strings with gains to test
  -c CONFIG      path to config file, default '/boot/piaware-config.txt'
  -r GREPSTR     gain settings string in config, default 'rtlsdr-gain'
  -v             verbose output
  -d             runs the script without actually changing the gain
  --delim DELIM  delimiter to seperate output


Script is available here: github/wittrup/piawarebash/pyh/optimize-gain-piaware3.py


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Thu Dec 15, 2016 11:49 am 
Offline
Charter Member
dstreufe - FlightAware user avatar

Joined: Sun Sep 18, 2005 11:00 pm
Posts: 195
wittrup58 wrote:
Forgive me for not reading the last 3 pages thoroughly, yet.
Inspired by this post and the work behind it, I stared modifying...

I have ended with a script that runs on both:
Code:
Python 2.7.9 (default, Sep 17 2016, 20:26:04) [GCC 4.9.2] on linux2

and
Code:
Python 3.5.2 |Anaconda 4.2.0 (64-bit)| (default, Jul  5 2016, 11:41:13) [MSC v.1900 64 bit (AMD64)] on win32

Latter one, obviously without the actual gain changing...

New features:
  • Formatting output, possible to select delimiter, default (tab)
  • Timeout, will not wait for socket receive if time is passed, needed in target poor environment
  • Handling packets directly, should be more memory efficient if run over a long time
  • Arguments parser, script now takes arguments for behaviour modification, help gives this:
Code:
pi@piaware:~$ pyh/optimize-gain-piaware3.py -h
usage: optimize-gain-piaware3.py [-h] [-t TESTTIME] [-n TESTSNUM] [-s SERVER]
                                 [-p PORT] [-g GAINS] [-c CONFIG] [-r GREPSTR]
                                 [-v] [-d] [--delim DELIM]

optional arguments:
  -h, --help     show this help message and exit
  -t TESTTIME    Seconds for each test
  -n TESTSNUM    Number of tests to perform
  -s SERVER      hostname for data feeder, default 'localhost'
  -p PORT        port, default 30003
  -g GAINS       list of strings with gains to test
  -c CONFIG      path to config file, default '/boot/piaware-config.txt'
  -r GREPSTR     gain settings string in config, default 'rtlsdr-gain'
  -v             verbose output
  -d             runs the script without actually changing the gain
  --delim DELIM  delimiter to seperate output


Script is available here: github/wittrup/piawarebash/pyh/optimize-gain-piaware3.py


Nice....

Is there anything else that needs tweaking on a typical dump1090 install? I keep hearing about adjusting the receiver's PPM, but for something like a Pro Stick/Pro Stick + with a TCXO, I'm not sure if it really makes a difference....?


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Sat Dec 17, 2016 1:21 am 
Offline
FlightAware Member
Nitr0 - FlightAware user avatar

Joined: Thu Apr 21, 2016 11:48 pm
Posts: 94
Well done wittrup58 :)
Very nice contribution - arg parser makes things very dynamic for those who use. As a cautionary, and after a quick glance through your nice script, this would be for those using dump1090-FA only (so far). Mutability would require a slightly different method to adjust the gain and restart. Any plans for an -m switch? :mrgreen:

I have found while messing around with something similar, based from the original script, that shorter duration testing/iterations, randomizing gain values and compiling multiple test results from different times of day tend to give much better results. Yes, it's much more work however. Can easily test this theory by submitting the same gain value multiple times...results will definitely differ within the same test run - all with the same gain, but it smoothens out significantly if multiple tests/results are compiled at various times. This is great for those who may not have separate rigs to compare head-to-head in real time.

RE: PPM - imho, so long as your dongle is TCXO clocked (rtl-sdr, FA dongles, nooElec), it's not needed, especially with ADSB in mind. If using a stock blue dongle, best to let her warm up for a good 20 mins or so before running your rtl_test to get a fix on what may be needed here - but overall, for ADSB, it shouldn't matter much even then if at all. The thing to worry about with 'stock' dongles is taking samples before it's crawled it's way up to normal operating temp. The clock drift is horrid on the way up. If want to see this in action, simply load up SDR# or whatever you prefer to view waterfall with - hone in on a good strong signal, then puff some air over the dongle (out of the case) - watch the waterfall move across your screen.


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Sun Dec 18, 2016 5:04 pm 
Offline
wittrup58 - FlightAware user avatar

Joined: Sun Sep 06, 2015 5:28 am
Posts: 19
Thanks!

Mutability support: I did some attempts last night with setting up linux virtualbox for implementing this, ran in to all sorts of issues like virtualbox dpkg not beeing found, my user not in sudoers, sudo requiring password, blah blah, the whole thing ended with me, yet again, regretting that I did not spend more time with linux when I was younger and learning was easy, it just don't feel natural. I hope some else will fork, fix mutability support and send a pull request :wink:

Last post I forgot to mention that I just finished the script and only had done some short time tests.
After that I found my peak hours of flight traffic and set up a cronjob:
Code:
#at 09:00 and 15:00
# forum forbids me to post, it should be "$ { HOME }" without spaces
0 9,15 * * * $ { HOME }/pyh/optimize-gain-piaware3.py >> optimize-gain.log 2>&1


After reviewing logs I found that sudo systemctl status dump1090-fa -l also dumps the output to stderr (and this messes up the order of the logfile), as you may have noted, I used Popen for "piaware-config rtlsdr-gain %s" % gain to capture and suppress the messages (unless explicitly asked for through the verbose argument). The same thing has now been added to sudo systemctl status dump1090-fa -l.
Then I made a script optimus-parser.py that sums up the result for each gain tested.
My result:
Code:
Gain   Msg   Pos   Acs
−10   14357   1412   35
36,4   7709   651   35
38,6   9176   753   35
40,2   9617   866   35
42,1   10652   947   35
44,5   12391   1151   35
48   15380   1435   35
49,6   17435   1654   35


When it comes to gain guessing I was about to add a silly line like:
Code:
gains = [gain if gain.lower() != 'r' else str(randrange(500)/10) for gain in gains] # untested

Before it struck me that it is absolutely not what we are looking for. What we are looking for is some sort of binary search tree. Something that starts of with testing 12.5, 25, 37.5, builds up a tree from that, decides to go either left or right, in my case, 37.5 on the right side, then continues by testing 31.25 and 43.75, in my case probably go right there as well, test 40.625 and 46.875, travels right, test 43.75 and 50, goes right, end with max. All this with testing the same amount of gains as the current script do with iterating over the gains list. This method could go on until it has determined everything down to the nearest firmware gain "step". It just require some more and beautiful logic. That is if, and that is a huge IF, it is so that this measuring will have only one peak on a graph. I was about to ask where to obtain a list of firmware gain steps...: rtlsdr_get_tuner_gains @ librtlsdr.c source

Since this whole thing is traffic dependent, it is somewhat difficult and you are absolutely right that shorter duration testing/iterations and multiple tests from different times of day gives better result. After doing a few time sudo systemctl restart dump1090-fa I came to the average time to be around 600ms, which seems too much for me. Also it gives this nasty Problem fetching data from dump1090. AJAX call failed (error: Not Found). Maybe dump1090 is no longer running? The displayed map data will be out of date. message on the web site. I tried some ways of measuring the time spent for rtl_sdr -f 1090000000 -s 2000000 -g 50 - to figure out if that is quicker. As I wonder if it is possible to start dump1090 with the --ifile parameter and just restart the rtl_sdr -f 1090000000 -s 2000000 -g 50 ifile.bin with different gains without breaking the binfile pipe?...

PPM: I suggest using https://github.com/steve-m/kalibrate-rtl if not using TCXOed dongle. Otherwise I find it to be outside the script scope (if that was suggested).


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Tue Dec 20, 2016 8:48 am 
Offline
myalias - FlightAware user avatar

Joined: Thu Mar 24, 2016 9:15 pm
Posts: 24
wittrup58 wrote:
After that I found my peak hours of flight traffic and set up a cronjob:
Code:
#at 09:00 and 15:00
# forum forbids me to post, it should be "$ { HOME }" without spaces
0 9,15 * * * $ { HOME }/pyh/optimize-gain-piaware3.py >> optimize-gain.log 2>&1



FWIW, $HOME works. Curlies aren't needed.

I hope to take a look at your autotuning script soon. I may also want to change the amplification setting at different times during the day, so perhaps I'll configure sampling for every half hour. Now I'm thinking about getting a second setup so I can test independently.


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Mon Jan 09, 2017 12:28 am 
Offline
Boeings - FlightAware user avatar

Joined: Sun May 29, 2016 2:09 am
Posts: 25
Location: Sydney
Hi, I have PA 3.1.0, and would like to run the gain optimization script.

Can someone please advise all the command lines required, and where to read the results?

Will my reports still go through to FA during the tests?

Thanks, B


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Mon Jan 09, 2017 5:28 am 
Offline
wittrup58 - FlightAware user avatar

Joined: Sun Sep 06, 2015 5:28 am
Posts: 19
I give up. Whenever I try to post all the command lines required I end up with "Forbidden You don't have permission to access /posting.php on this server."

Take a look at the command wget.
Download this file: https://raw.githubusercontent.com/wittr ... iaware3.py
And run it by typing: optimize-gain-piaware3.py
Wait for result.

Yes, your reports will still go through to FA during the tests. Except for the apx. 600 milliseconds every time dump1090-fa restarts.


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Mon Jan 09, 2017 8:31 am 
Offline
FlightAware Staff
obj - FlightAware user avatar

Joined: Tue Sep 30, 2014 7:14 pm
Posts: 3203
wittrup58 wrote:
Forbidden You don't have permission to access /posting.php on this server.

I've never seen this myself but I've asked ops if anything changed recently.

edit: apparently it is mod_security interfering, seeing if I can get those rules relaxed.


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Mon Jan 09, 2017 5:47 pm 
Offline
FlightAware Member
idh - FlightAware user avatar

Joined: Tue Oct 01, 2013 12:08 pm
Posts: 109
Quote:
"Forbidden You don't have permission to access /posting.php on this server."

Yes that is what I was getting in another thread over several days when trying to post with multiple "[code][/code]" statements.

In the end had to remove all the "code" and it posted it immediately - not pretty any more but at least I got my response into the thread.


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Tue Jan 10, 2017 3:47 pm 
Offline
FlightAware Member
keithma - FlightAware user avatar

Joined: Fri Nov 04, 2016 1:59 pm
Posts: 93
Location: JO01ou
wittrup58, if you can manage it, I'd really like to see a version of your script for mutability.


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Wed Jan 11, 2017 8:03 pm 
Offline
FlightAware Member
andrew6721 - FlightAware user avatar

Joined: Sun May 10, 2015 5:06 am
Posts: 64
Location: Sydney, Australia
abcd567 wrote:
Hi Jeff
Generally for ProStick + Filter, the settings are in the range 42 to 49.6 (max). Most use values around 45.


Received my two ProSticks today, now the fun begins. I run 1090-mutability and have always had the gain set to AGC on both my permanent and portable setups. I saw a noticeable difference on the portable setup changing from the DVB-T stick to the ProStick + external filter combo; didn't really see any great change in the permanent setup (DVB-T stick + filter to ProStick + filter).

Will start playing with the gain figures on the portable setup first to get a feel for things - lucky I read through this thread as I was wondering what a good starting point would be :-)

Andrew


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Sat Jan 14, 2017 7:13 am 
Offline
biekerc - FlightAware user avatar

Joined: Sat Jul 16, 2016 2:17 am
Posts: 40
What does gain -10 actually mean?

I have two feeders side by side, .111 and .137. The only difference is the antenna, both have a FA stick a la orange, albeit different generations...

Running the gain scripts on .137, the best gain value by far is -10, no other gain value gives results that come even close. Now I thought -10 means automatic, and the device itself chooses one of the possible gain values, but that can't be the case when all other lead to worse results?!

On the other device, .111, the best gain value seems to be 22.9, the results are far better than with -10 or any other value.

However, when I change the gain to this value and reboot, the results (planes, messages) fall off compared to the other feeder and are lower compared to setting the value to -10, even though -10 should give far worse results according to the script.


Last edited by biekerc on Sat Jan 14, 2017 8:40 am, edited 1 time in total.

Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Sat Jan 14, 2017 8:24 am 
Offline
FlightAware Staff
obj - FlightAware user avatar

Joined: Tue Sep 30, 2014 7:14 pm
Posts: 3203
biekerc wrote:
What does gain -10 actually mean?

Enable the tuner AGC (which does not act as an AGC for ADS-B)


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Sat Jan 14, 2017 11:29 pm 
Offline
FlightAware Member
andrew6721 - FlightAware user avatar

Joined: Sun May 10, 2015 5:06 am
Posts: 64
Location: Sydney, Australia
obj wrote:
biekerc wrote:
What does gain -10 actually mean?

Enable the tuner AGC (which does not act as an AGC for ADS-B)


@obj - so that is the same as having GAIN="agc" in /etc/default/dump1090-mutability ??

Cheers,

Andrew


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Sun Jan 15, 2017 7:25 am 
Offline
FlightAware Staff
obj - FlightAware user avatar

Joined: Tue Sep 30, 2014 7:14 pm
Posts: 3203
Yes, if you look at the init script you will see that AGC is translated to -10 and if you look at the dump1090 source then you will see that -10 turns on AGC.


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Sun Jan 15, 2017 12:28 pm 
Offline
biekerc - FlightAware user avatar

Joined: Sat Jul 16, 2016 2:17 am
Posts: 40
obj wrote:
biekerc wrote:
What does gain -10 actually mean?

Enable the tuner AGC (which does not act as an AGC for ADS-B)


Ok, so it enables AGC, but does not act as an AGC due to the nature of ADS-B signals. In another thread I read that it usually sets a gain value that is higher than 49.6, the hightest value you can set manually. At that value, however, I almost receive nothing. Just looking at the results of the gain script, the results of gain=-10 are comparable to gain values around 20+.

So des that mean that the AGC tunes itself to something other than the ADS-B signals?


Top
 Profile  
 
 Post subject: Re: Gain Adjustment...
PostPosted: Sun Jan 15, 2017 1:13 pm 
Offline
FlightAware Staff
obj - FlightAware user avatar

Joined: Tue Sep 30, 2014 7:14 pm
Posts: 3203
Turning on the tuner AGC sets a fixed gain for one gain stage and hands control of the other stages over to the AGC.

The AGC works by measuring the amplified signal power over a period and using that in a feedback loop to control the gain to try to get an amplified signal power that is somewhere in the middle of the usable range - i.e. if the amplified power is too low, it increases the gain, and vice versa. The AGC is designed for the dongle's original purpose - receiving digital TV signals. For a digital TV signal you are receiving a continuous signal from a single transmitter so this works fine.

ADS-B signals are bursts that are much shorter than the measurement period, and they originate from many transmitters of different power at different ranges. So the AGC measurement is basically useless and in practice the AGC sees it as a very weak signal and ends up setting the gain as high as it will go.

Due to a quirk in librtlsdr, the combination of the manually set gain stage + the AGC-controlled gain stages results in a total gain that is higher than what librtlsdr will let you set directly.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 182 posts ]  Go to page Previous  1 ... 4, 5, 6, 7, 8  Next

All times are UTC - 6 hours [ DST ]


Who is online

Users browsing this forum: homestea, navzptc and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to: