2000 Lincoln Town Car Power window switch

The switch that controls all the windows on the drivers side needs to be replaced. It will put the window down but not up. (That is not without coaxing on my part).
The part is pricy...that's understandable. The R & R is too. That, I would like to save on. I can't imagine it's more than removing a switch assembly and replacing it. Getting the panel off that holds the switch has got me puzzled. So, am I on the right track, and if so how do I go about accomplishing this task.