Fan System Stopped Working Suddenly

When I bought the car a couple years ago, the AC and speed 4 for the vents never worked. No biggy, I can live without both. Out of no where, the speeds 1-3 stopped working. I found a "sweet spot" on the dash that if I gave it a "love tap" the vents would come back on again. This leads me to believe the issue is a short of some kind. I have checked the fuse and it looks fine, it's not cracked. I no longer can get the vents to come back on with the "love tap." When I change the speeds from 1-4, nothing happens. Any ideas on what may be causing this issue? I know it could be a ton of things but I wanted a few ideas so I could possibly fix it myself. If you need more information, let me know. I know this is pretty vague, I just don't know how to explain it any other way. Thanks for all your great help!