Starting issues with dimming headlights

My car is having issues. It's very cold here. When I start my car I turn the key and all I hear is a click then when I turn the key again it starts just fine. Sometimes it takes a few tries and sometimes it starts the first try When I turn it the 1st time and it does nothing but click I still have power to the radio, heater, dash and headlights. Also, while I'm driving when I idle down the headlights dim for a second and the heater blower power decreases for a second also. We bought a new battery hoping that was the issue. It was not. Also the new battery is reading great but when I rev up the engine the battery voltage goes way down. Also the check engine light is on reading a bad o2 sensor.