The lounge is for elite tier, which those people EARNED. If everyone is allowed to bring 2 people in who aren't elite tier it would crowded. What's the value of earning a reward or perk if anyone can have it without having earned it. You can join your friends in the general boarding area.
This type of question also comes up regarding club class dining. If you didn't pay for it or get it as a perk for loyalty tiers then it's not yours to use.
Thanks to the OP for asking so that they know they don't have to leave them and can join them in the general boarding area. If you arrive early you could get into the first group to board after the B2Ber's and elite tier which is still early.