2005 Honda Accord alignment

How hard or easy is it to actually knock a car out of alignment? Is general road damage--with no more than about 1 or 2 inch depth of pot holes going to knock your car out of alignment if you hit them at about 40mph or above? Also--what about manhold covers or water service pipe covers--that are recessed up to 3 inches below the tire level of the asphalt/concrete road--will that be enough?

i usually try and avoid all of those or slow down--but more often than not--it really upsets drivers behind me. Am I just being overly cautious for no reason?

thank you