You will be mixing directly with Isle of Wight traffic aiming to get to Ocean Terminal. I have known it take an hour from West Quay car park to Town Quay a 10 min walk due to grid locked traffic. Any Southampton regulars will advise traffic can be unpredictable especially on days with more than a couple of ships in port and factor in Isle of Wight festival. It’s your choice at the end of the day but If you miss the ship onward travel to meet at the next port will be at your own expense. You couldn’t’ t claim on your travel insurance either. Mine is with a Nationwide and there is a clause about leaving home in good time to avoid being late.