Not sure.
Perhaps use the "Thread limit" option. The number sets the maximum number of threads. Before you want to add one more thread, raise the number by one. Be aware that if you set it to a lower number than the current amount of threads on board, this will delete the surplus, so don't make the mistake of losing threads.
One thing: don't mistake it with the "Hourly thread limit" option.
In addition to the thread limit, to avoid bumping off threads, you might wanna pin all what you have.
So combining these two tools in theory you could achieve what you want.