84' Trans Am starter tries to turn the engine over with out the key in.

First off, the car has had problems with keeping the battery charged. Then the other day after I drove home, I turned the car off, took the key out got out off the ignition. Then I heard the starter trying to turn the engine over. Don't know if bad starter, or if bad ignition switch, or if there is faulty wiring a foot maybe to the batter itself. Please Help!