Manual Transmission Won't Shift Into 1st or 2nd

My shifter is acting really weird, not that it locks out any particular gears, but rather it replaces 1st and 2nd with 3rd and 4th. An example, from Neutral, it shifts very smoothly into 3rd, 4th, 5th, and Reverse. But when I try to shift it into 1st, it shifts smoothly, but when I let off the clutch, it's very obvious it's in 3rd. I can proceed to shift into 4th by putting the shifter in the second slot, 5th with the shifter in the 3rd slot, and Reverse with the shifter in the 4th slot. Not only that, but Neutral becomes very offset to the left, that is until I shove my shifter over until I hear an unsettling Clunk. After I hear the clunk, I can put it in the normal 5th and Reverse slots, and Neutral becomes the middle once again.
Basically, what I get from that is when I move my shifter to put it in 1st or 2nd (even if I don't even put it in gear) it brings third with it, putting it in position to go into 3rd gear with the shifter in the 1st slot.
I'm not sure what to try, nobody I know has ever heard of this. Help?