Citroen Xsara 2001 making noise when driving but accelorator not pressed

My 1.4 Xsara is fine when under power, no matter at what speed but as soon as I take my foot off the accelorator it makes a groaning,low rumble sound on the front passenger side. If I put foot on accelorator again it goes, and it also goes when I put my foot on the clutch.