1986 Oldsmobile Cutlass Battery Drain

ok ive had this problem last year but it seem to go away now its back. i go to start the car in the morning and i get nothing ..everythings dead i check the battery and theres like 4 volts running through it. then i charge it for like a half an hour with a battery charger or get my pops to jump it "takes about 5 mins to jump" and it starts and ill drive it all day with no problems. then the next day depending on how long i drove it is how long i have till it wount start again because after changing the ground cable on the battery and various testing i notice theres a very slow drain on the battery that i cant seem to fix. usually if i drive it all day to work it will start at night for me to get home from work but it seems now that the drain is going a little quicker and the starter does hesitate after it turns over once " i get
like a one second pause then it turns over this happens usually after its charged or when the batteries dead but when the batteries dead its followed by a ticking sound then nothing. so what i did is removed my stereo and tested and still getting a drain and i did the same with the alarm. now my
question is can a bad starter cause the drain because im getting very frustrated and dont have alot of money to fix it and need to get to work everyday and this is my only transportation so any help would be very appreciated thanks.