IRS payment plans and agreements are not reported to the credit agencies so by setting up an IRS payment plan will not affect your credit or credit score.
The IRS does not report any of your payments or delinquent IRS taxes to the credit agencies as it's not credit that you are owing to them.
Both owing the IRS or making payments to the IRS on your income taxes will not affect your credit score.
So if you need to setup a payment plan with the IRS to pay your back taxes you owe then there's no need to worry about your credit score being affected.
Just continue to make the IRS income tax payments as you agreed to on and they cannot take anything you own or seize your bank accounts etc.
But if you default on the payments the IRS could seize your property, bank accounts, garnish wages etc.