'95 Prism Pulley Question

I have a '95 Geo Prism (1.6 L) with 156,000 miles on it. Yesterday as I left for work, It suddenly siezed up and my battery light popped on. I immediately opened the hood to discover that my two largest belts had fallen off their pulleys (the third, smallest belt is still on with high tension). I examined it more closely upon returning from work and noticed that all of the pulleys except the bottom-most moved freely when I turned them. This lead me to assume that this bottom pulley is the drive shaft pulley (I'm making up terms here). What concerns me is that this bottom pulley was also about 1/2 to 3/4 of an inch dislocated (as in it had started sliding off of the mounting) and I had to work it back towards its original position. Since returning it to the base of the mounting, it will not slide back off, but I am afraid once I replace the belts and increase tension etc, they will all come flying off again. My question is should this pulley slide at all and if no, can a layman fix it or should I be taking it in? Thanks for your responses on this one!