Both but more so Gardiners Island.
You have great enthusiasm for the sport but at times put yourself in possibly bad situations.
Gardiners Island is more of a open water situation. Weather forecast are great but can change drastically in short order. Boats, waves, BIG 
with big teeth. You need better recovery skills and instincts were a situation can become worse faster than a bay.
If you need more experienced kayakers with you then you are dependent on them. You put them at risk also. Partners are good but everyone should be able to handle situations themselves and not burden others especially in open waters and strong currents. Because you did Shelter Island twice doesn’t make Gardiners Island a breeze.
I think in open waters you need a bombproof roll in rough water or at very least a fast scramble self rescue ability in rough waters. You need the ability to feel calm in rough waters and be able to brace without even thinking about it.
I know my limitations and stay below them in most situations. I mean I did go out in 35 mph winds to play and see what it was like. It was rough but if I failed it was a short swim or blow to shore in summer bay water in daylight.
Balance risk like going out at night I do it in the summer but on clam warm water nights in the bays lit up like a Christmas tree. Still had a few close incidents. To try to paddle around Manhattan at night especially if you never did it in the daylight would not be my choice as you discussed doing. Boat traffic wakes you can’t see and all types of debris are present around Manhattan you’ll never see at night. Currents are not very visible at night either like daylight. I wouldn’t take my boat around Manhattan at night.
Crossing the Long Island Sound with no VHF I wouldn’t do either. I went out from Norwalk Connecticut weather changed fast we made it to an island to hold up a bit. I forget the name of the group of islands. Found ourself in some fairly choppy waters with breaking waves. Partner was panicking stiff winds hard to turn 22 Libra XT tandem. She basically refused to help me edge the kayak probably was really fighting against me. Hardest I ever had to paddle with sweep strokes. As that all was happening we were watching water spouts towards Long Island side. Weather forecast was all wrong for they day.
In one of your video’s you got caught by a small quartering stern wave and let go of the paddle on one hand to try and keep you balance. Natural instinct if your walking down the road but not in a kayak. No paddle in your hands your toast more experience and muscle memory that won’t happen. You need to be more comfortable in waves before sticking your neck out in open water.
I personally would not feel comfortable going around Gardiners Island with my skills. I can amuse myself in Jones Inlet with an incoming tide in warmer water when boats are present if I want a few bigger waves. Plus CG station is three minutes away. 
Something popped up next to my kayak in the inlet and I was 
probably at least a porpoise but I could have touched. I wouldn’t want to be around a 
whale surfacing even in my boat.
Things can go smoothly for ten times then the eleventh who knows. Good luck on your choices it’s about the risks you are comfortable with.
Joke