The rule treats every IRA as one account
When you convert money from a traditional IRA to a Roth, the tax is not worked out against the account the money left. Section 408(d)(2) says all of your traditional, SEP and SIMPLE IRAs are treated as a single contract for this purpose, and the conversion is taxed in proportion to how much of that combined balance is pre-tax.
So the account matters less than the total. Opening a fresh IRA for a non-deductible contribution, converting it the next day, and leaving every other account untouched changes nothing about the arithmetic. The new account is added to the same pile before the proportion is worked out.
Why this surprises people doing a backdoor Roth
A backdoor Roth is a non-deductible contribution converted straight away. The appeal is that a contribution you got no deduction for should convert without tax, because there is nothing untaxed to tax. That holds only while you have no other pre-tax IRA money.
One rollover from a former employer’s plan, sitting in a traditional IRA and forgotten about, changes the proportion. If that balance is large next to the contribution, almost all of the conversion becomes taxable, and it stays that way every year the balance is there. Nothing about the transaction says so at the time: the conversion goes through, and the figure appears on a Form 1099-R the following January.
The basis is not lost
The part of your basis that is not used carries forward on line 14 of Form 8606. It is still yours and it still comes out tax-free eventually. What it does not do is come out this year: it is spread across every future conversion while the pre-tax balance exists, in smaller and smaller proportions.
That is why the cost is usually described per year rather than once. Our pro-rata rule calculator works out what the conversion is taxed on this year and what it would be taxed on if the pre-tax money were somewhere the rule cannot reach.
The date that decides it
The balance that counts is what your IRAs hold on 31 December, not on the day you convert. Converting in March and moving the rest of the balance out in November does not help for that year, because line 6 of Form 8606 asks what was left at the end of it. This is the detail most often got wrong, and it is got wrong in the direction that costs money.