Low voltage in cold weather

I am having an issue that I hope I just the alternator.
When the vehicle is cold in the morning, the battery light comes on, and I get a "Check Charging System" message after starting the car. When I put the car in reverse the power steering locks up, which I believe is all electrical since there is no where to put power steering fluid in. I have checked the voltage on the battery terminals before starting (12.6 volts), and after this message comes on(about 11.6 volts). I think this alone indicates a bad alternator.
However, these failures are short lived, they go away after running the car in park for about 3-4 minutes. The battery light turns off, the Check Charging System message goes away, the power steering starts working normally, and the voltages goes up to about 14.5.
I have had this vehicle for almost 4 years and this started happening in the middle of last winter, so almost a year ago. A month or so after it started, I got a new battery believing that would solve the problem, it did not and the problem persisted until the summer. However it is starting to get cold again, and the problem is back in full swing. Every single morning, same problem. Other then it being cold in the cab, I don't think I need to "warm up" the engine for more than a minute or so before putting it in gear. Annoyed, I had my wife take it to the dealer, hoping they would diagnose and tell me exactly what the deal was, no luck. They sold my wife on a "Faulty Leaking Battery that is probably the issue, but we can't tell until we clean and properly dispose this mess." I really don't know if that was true or not, but the battery was not even 10 months old. Will a bad alternator ruin the battery?

So, is this the alternator, something else, or do I need to just suck it up and deal with it being annoying?