Some tables may agree a time, others won't.
We shared a table with a couple who liked to eat early so we often arrived when they were having dessert or coffee. It worked more like a table of 4 than 6.
Lots of people eat elsewhere for breakfast and lunch, there's no awkwardness, the staff are used to it.