It depends what you are doing in port. If we take a cruise ship excursion, we leave them in the safe. If we do a private excursion, or just go out on our own, we take them in case of an emergency. If you don't already have one, I highly suggest getting a pouch that attaches to your belt and can be placed inside your shorts/pants...... in our case, my husbands! We have one that is the size of passports and he carries them there along with some extra cash.
Amazon.com | Lewis N. Clark RFID Blocking Money Belt Travel Pouch + Credit Card, ID, Passport Holder for Women & Men, Black, One Size | Travel Wallets