Starter did not react at all no crank nothing?

A few months ago, after a hot day and standing still for that day my car did not want to start at all the starter did not react at all, no crank anything. After taking a look at the battery by charging and testing it with a multimeter it showed 12V as it should. After reinstalling it and trying it out a few more times it cranked and started again. And it was fine for a while then problems started. The crank fails and the car only started after the third time.
Thus, I replaced the starter motor with a brand new one. Now it works. 50% of the time the car starts immediately, 100% after it had just been driven. The rest of the time the starter does nothing. Turning the key does nothing, I have to remove the key and re-plug it.
I looked at all the cables, they are new and installed properly. I tried my second key still the same. And different ways of turning it might have slight impact on the result. What would you recommend checking and why? It should not be the fuses, right? It could be the relays.