The 2 cruises we went on, the coach driver told everyone on the way down to ignore the disembark time given and either request an early one or just get off early. Also someone from P&O got onto the coach when we arrived at the terminal and told us to ask for early disembarkation times.
Of course, what happened both times is similar to OP's story, with us waiting nearly 2 hours for the stragglers to arrive from the buffet, with our coach being the last one out of the car park. It is pretty much the only thing that I could find fault with on either holiday.
I don't see how difficult it is for P&O booking system to have a tick box for all on coach transfers and then ensure that those passengers are all given a similar leaving slot.