No.  The main requirements for the drill are that passengers are accounted for all together (as they would need to be during an emergency) and that the drill be "as realistic as possible".  Additionally, unless you restricted those for the later muster to their cabins, this would cause chaos as crew attempted to direct passengers to their muster, only to have the passenger say "I'm not in this drill", along with chaos in the public spaces as some need to go to muster and some don't. While