Can't disable an alert without deleting them.

I used to be able to disable alerts and keep them “in queue” for when they’d be needed later. Now it seems to trim alerts and stay under 5 I have to delete them. Any way to bring back a way to just disable the alert without deleting? Thanks.