You can do Cinque Terre in a relaxed pace, it’s so easy to do. Suggestion, take the train to Vernazza and wander around town there and then have lunch in Vernazza there. Then after lunch take the train to Manarola and check out that town, then finally train to Riomaggiore to check it out, and then train back to La Spezia.
The port city of La Spezia is nice, but not a ton to do or see in my opinion. My recommendation, don’t skip Cinque Terre, you would regret it forever.