I spent a little less than a week in Taormina a few years ago, the town itself is quite small and the only real "site" is the Greek theatre, which takes about an hour to see thoroughly.  The rest is shopping and views, so I think five and a half hours would be fine.