I am a reasonably mobile techie. Network stuff mostly but have been dealing with linux often lately. I am even making yet another bootable persistent drive right now, so it would be downright rude for me not to help out. I'm going to mostly stick to the 'disposable system on removable media' idea as the other obvious stuff has already been gone over reasonably well.
Firstly, it may be possible to boot windows from a usb stick, though I have not tried it. It would require flipping the RMB (removable Media Bit). Doing that will make the system treat usb drives like internal ones.
This tool is what I have used to do this. Be warned, it is entirely not supported, wont work on all drives, and may bork your drive. Another quirky method that I have gotten to work is using windows 7 disk management to repartition a usb stick. Something windows does not like at all and again is not supported, yet after a couple volumes are made, then wiped and made whole again, the RMB is zeroed for whatever reason.
I have been relatively vague about all of that as it is a silly process and prone to screwing stuff up.
To get a nice, portable, bootable linux system up and running simply
follow these instructions. The only things you really need to know in particular about this method is that it is a live build with potential for persistence. Basically if you set the persistence file size to zero, you will get a fresh system every time you boot. All your activities, installations, and whatever else is gone forever just like a livecd. Should you want to keep settings, files, programs, and so on then you need to set the persistence file to some size up to 4gb. Now linux is really small. A reasonable desktop can be had for just a few gigs. Should you need more space, you can always install to a bigger drive and mount to it, essentially negating that 4gb limit, but only for data. Additionally you could simply install in full directly onto any removable drive and call it good, as many people have already pointed out.
I use lots of stuff depending on the specific purpose. I do absolutely love virtual machines. They are great for so much and even a half assed setup can mean the end of catastrophic data loss or downtime. My favorite is
Oracle's VirtualBox. I also like unetbootin, but usually roll multiboot with
yumi.
For a really secure setup I wold definitely use Truecrypt within a 4096 fde live system with all my important emails ran through
this, stored on unnumbered,
catnip laced ibm 5081s in a room that my cat has almost but not quite entirely figured out how to get in to.