Car wouldn't even crank

One day went to leave and the car wouldn't even crank. The dash lights and digital clock stayed on and bright. All I heard was the buzzer alarm pre ignition and a single click. I switched from park to neutral to no avail. I tapped the ignition, jiggled the keys in the ignition, locked the steering wheel, checked the fuses and ignition relays all with no luck. A few minutes later my wife goes out and jiggles the the keys and bam to my disbelief it starts! Then after two starts it's doing the exact same thing. I even replaced the ignition switch cause it looked tired. I don't hear anything at all from the starter, not even the bendix. Just a single click when I try it. What do I do next. Starter replacement looks tedious. Any ideas?