By happenstance, there was 10-12 of us from our cruise that did this. We did not meet each other until either sharing a taxi or getting on the ferry. Once there we kinda kept an eye out for each other so nobody got left behind. We all did our own thing.
To be honest, while planning this I was stressing about getting back to ship on time (we packed a change of clothes/passports and port agent’s phone number, etc…). But after the fact, I am so glad we did it this way. It was cheaper, longer and had more freedom.