A/C Compressor Clutch knocking loud but still cooling. Can I keep on driving it.

When you start vehicle the compressor clutch knocks but when you give it gas the noise quiets down until you come to a stop then you can hear it knocking again. What will it do if i keep driving it and using the a/c since it still works and gets cold. What damage could i cause and could it break my belt if it freezes up?