The drive from Miami to Key West via Highway 1/Overseas Highway is very scenic and takes at least 4 hours on a good day and considerably longer on a not good day. So in my opinion, four days is about right. One day to get there, one day to get back, and two whole days to spend there. Don't forget a visit to the very nearby Everglades National Park.