I went with my wife for a baby moon a few years ago. I had always wanted to drive out across the keys.
I didn't hate it but I didn't love it. Here were my impressions.
1) The drive out felt like driving on South Padre Island. I always had this impression that it would be a lot more over water, but you on land most of the time. I had always wanted to do this and it didn't really meet my unrealistic expectation.
2) Key West isn't a beach destination, if you're looking for a sandy beach to lay on. There is a pay-to-access beach that is pretty well maintained, and if your group is really wanting beach time, it'll scratch that itch, but I assumed there would be nice beaches all over, and most of the shoreline is rocky.
Those were the negatives. But there were postives, too:
Overall, it lives up to the laid back vibe you probably expect. Once you get to the historic center, it's very walkable. Plenty of good food and drink.
Our trip was short, if you are there longer, I'd recommend that you plan out some excursion that might take you to some of the other keys. There are better beaches on some of them. If you are into fishing, there are great charter options on several of the keys.
There is a fun rum distillery that you can tour and get a locally made bottle of rum. I'd do that. The Hemmingway stuff is cool if it interests you.