I remember my 8th grade science teacher telling us all about Plutonium's toxicity and I believed it for many years, so I don't blame you. Apparently some researchers had a "Hot Particle" theory that said that because of the way plutonium was taken in by the body it was much more dangerous than just looking at its radioactivity would lead you to believe - concentrating in certain areas of the lungs. Its not surprising that media figures like Nader confused this with chemical toxicity, or didn't notice when experiments contradicted this theory.
EDIT: Edited for horrible grammar caused by rewriting something without proofreading after.
This is Plutonium Oxide. IIRC, metallic plutonium is a lot more dangerous. Also almost impossible to come by in an oxygen atmosphere. Kind of like Aluminum.
Plutonium is chemically extremely toxic. Even throwing some powdered plutonium in air somewhere would cause serious harm.
Edit : In fact it's not :)