We have been in Retreat for quite a few trips with X but only on Edge class ships. As for SS, we have only been on the Silver Moon (newer ship) twice with several more bookings next year.
Really enjoy the Retreat on the modern ships with so much space both in the cabins, your own restaurant, swimming area, bar, large comfortable lounge with so few people.
SS although a smaller ship seems more crowded. But we love SALT kitchen and are happy to share that space with other chefs.
It is really hard to compare the two because the vibe is so different. X ships have more action around the ship late into the night and SS is much more sedate but with great speakers on each destination (scientists, archeologists, writers, etc).
We enjoy each but for different reasons.