Driver window only goes down, doesn't go up.

I've had the wiring reversed once so that pressing the button down makes it go up, then I had it wired back to normal and the guy at the shop told me I needed to go to a parts sale and replace my button unit. I did just that but the window still only goes down, it won't go up -- so I'm starting to think this wasn't a problem with the unit to begin with.

The motor is probably OK since the window goes down easily and reversing the wires made it go up, so I'm thinking it's a wiring problem. But if it's not the driver-side button unit, then what could it be?