You can swim with a paddle jacket or dry top. About the only circumstances that would really impede swimming would be if you had a full dry suit and the neck gasket blew out, or you were wearing a bib with attached footies or ankle gaskets. In those instances, the suit or bib could fill with water and make swimming difficult. The weight of the water in the suit would not drag you down so long as your body was in the water, but it could be very difficult to get yourself up out of the water.
I used to wear a dry top frequently for whitewater kayaking accepting the possibility that I would get wet if I swam, which did happen a few times. But I found that is I was wearing layers of decent synthetic clothing or fleece under the paddling jacket or dry top I would remain as warm as if I was wearing a wet suit and after a few minutes, I would be about as warm as if I had been wearing a dry suit. So long as the outer garment is an efficient wind stop, your body will warm the water in the clothing pretty quickly, as if you were wearing a wet suit. If you are fully immersed for more than a second or two, I don’t think it will make much difference if you are wearing a paddling jacket or a dry top. Water will get in the dry top just about as quickly.
Two piece dry suits can work quite well if both the top and the bottom have tunnels and you know how to mate them together properly. A pair of dry pants that just has a wide neoprene waistband will not keep water out.
I would not have any reservations about wearing a farmer John style wet suit over a decent synthetic long sleeve top with a good paddling jacket or dry top for the conditions you describe.