I avoid driving a rental car through New Jersey and the NYC area at all costs. I would take a train from Philadelphia to Hartford CT and then rent a car to go see your friends and go from there to see Maine. Take a swing through Newport RI either on the way or on the way back to return the car to Hartford. You probably can get a flight back to Florida from Hartford.

Philadelphia (in the city or the suburbs?) to Fairfield is not a really long trip so taking a train out of the way to Hartford (which is not where you were planning to go) seems unnecessary to me. It will take longer and you will just be adding the cost of the train ride since you'll be paying for the same number of days of car rentals either way.

If you really want to stop on route by car, you could detour a bit to Princeton which is small enough to see couple things and get back on the road.

Fairfield to Berkshires and Fairfield to Newport are in opposite directions from each other. Are both must sees for you? If you skip one of them, the itinerary would have less backtracking.

Philadelphia to Fairfield, to Great Barrington or Lenox (berkshires) to Montpelier (or somewhere else in Vermont? ) To Portland to Bar Harbor would be good, and there are places between any of those 2 places that would also fill in extra days if you want.

If you want to include Newport, then Berkshires and Vermont are a detour.

A mostly coastal route the whole way would be Philadelphia, Fairfield, Newport, Portland , Bar Harbor--with many options for additional stops along this route too

End of May --Early June is a little early for some of the summer music and theater festivals, but after Memorial Day, you might be able to find a couple in those areas where the summer tourist season begins that early. Maine doesn't get into full swing until July 4th.

We haven't been to Bar Harbor area quite that early. Lupines bloom mid-June but here's info about bloom times for lilacs etc. There's an azalea garden on Mt Desert. Maine Coastal Gardens near Boothbay Harbor would also make a nice walk on your way south from Bar Harbor.
http://www.mainetravelmaven.com/2011/05/22/maines-azalea-and-lilac-gardens-blossom-with-springs-tidings/

But, to start off from Philadelphia, aim to for Albany NY and then to Montpelier VT. I don't think you will get near NYC. On your way back south you can take the Tappanzee bridge north of NYC and avoid the city again. No need to go thru NJ.

From Montpelier I would go a little south. You can take the famous Kancamagus Hwy thru NH's White Mountains and then Rt 2 over to Portland ME. Obviously, you know to spend some time there since it's on your list. Then take I95 the quickest route to Bar Harbor. We usually spend time in the mid-coast area. When you head south from Bar Harbor, you can take Rt 1. It will be slower but you'll also go thru beautiful Camden and lovely Wiscasset. Boothbay would be a slight detour. In Brunswick you can get on I95 again for a quicker trip south to Newport.

Then to Berkshires and CT but I would eliminate the Berkshires if you are short on time and can't find a music performance.