How-to Serialize Dongles for ES1090 + UAT978

I think you should read this howto: How to Install and Configure Piaware 5.0 SD card image - Quickstart Guide

This error when running rtl_test is always due to some program using it still running.
dump1090-fa and dump978-fa get restarted by piaware, so you need to stop that as well.
Also when you reboot the services obviously start up again.