Noise occurred in heavy rain when I let off the gas

I had a sound under hood when we had a heavy rain, it occurred when I let off the gas, when I hit the gas it went away, let off noise again. I have not heard the noise since. Checked the belt later on and I was able to move it enough to where it almost touches another part of the belt kind of like a rubber band. Do I replace the belt or the tension-er? Do loose belts make a noise or the tension-er?