Yes you can paddle there. All of the basic rules of paddling in a busy motorboat and commercial vessel infested waterway apply. Best to stick to the early mornings imho…You can launch from ABay
You can visit Boldt Castle without a passport if launching from the US. As it is a port of entry, it is advisable to have some form of ID such as a passport, REAL ID driver’s license, or other documentation that indicates citizenship. In fact it is always a good idea to have some form of ID showing citizenship if traveling anywhere near a national border. Immigration is known to set up roadside and other checkpoints.
There is a place to land a kayak at Boldt Castle on the southwest side of the island, not far from the admissions office. I don’t know if there is one at Singer Castle. Boldt Castle is open from 10:30 - 6:30 and the entrance fee for adults is $11. Plan on spending at least a couple of hours. Well worth the visit.
Beware of numerous tour boats and other boat traffic, which can be heavy.