We were faced with this same dilemma a few years ago. We went with the "don't worry, you'll be fine" theory and kept our adjoining/connecting door cabin. During the cruise our neighbors weren't being loud but we could hear every word of their conversations and assumed that they could hear ours. We used towels and tape to cut down the noise through the doors and that helped a little.
We'll never book adjoining cabins again...