If you are VIFP Platinum or Diamond, or have FTTF, you can open that access door and drop your carry-ons off in your cabin.
Personally, I am the same way. The only time we rush to get on the ship early is if we're flying in that morning and were in town anyway (or checking out of a hotel nearby). Otherwise, we aim to get to the pier around Noon or 12:30. The terminal isn't as busy, by the time we do muster and eat our rooms are ready. There just isn't much going on around the ship until sail away anyway.