Your refund from California no matter how large would only be taxable if you filed an itemized tax return, that is including a schedule A, and reported a deduction for the California state taxes paid.
If you did not itemize on your 2005 tax return and elected to claim the standard deduction, the good news is the IRS would not tax you on your California tax refund!!!