Cyncyn's idea of the beach is great. I'd keeping following the coast Eastward (away from the ship) turn left at the Mile 0 sign which will bring you to Beacon Hill Park. On a hot day there's a fun spray park at this side of the park, a couple of great playgrounds farther into the park, a petting zoo and pretty good gardens for those who enjoy such things. From here it's easy to walk into Victoria to check out the museum, the Bug Zoo or whatever else tickles your fancy.
Butchart Gardens is beautiful but my boys (now 13 & 16) have never really enjoyed it. They would rather go somewhere they can run and horse around to burn off some energy