Learn your state’s boating laws. In my home state (CO) the booklet includes how far to stay away from both shoreline fishermen and those in stationary boats.
Boaters—which includes paddlers—are required to stay at least 150 feet away from fishermen. A big distance, but I suppose it provides margin for the fact that the person fishing is much more visible than the line itself. The exception is when a fisherman is deliberately hiding his location, his “secret” fishing spot. They shouldn’t get bent out of shape when their concealment leads a passing boater to get too close before seeing that someone is fishing, but some of these old geezers are super-territorial and nasty.
When I’ve realized at the last minute where they are, I visibly veer away and say “Sorry!” That pretty much heads off an unpleasant encounter. After all, most people aren’t trying to hurt their fishing experience; it is usually a case of not seeing, not paying attention, or ignorance of the laws.
What throws things off is when a fishing boat that WAS under way suddenly stops. But I haven’t had much trouble with this, probably because I go places that usually aren’t jammed with fishing boats. In crowded waters, everybody gets irritable.
These days I stay farther from shore due to dropping water depths. I don’t want my ski’s rudder hitting rocks or mud or underwater bushes.
The last time I went out, something odd happened. After I had passed several stationary or slowly trolling boats, I looked back in about a tenth to a quarter mile. The boats all seemed to have followed where I went and started fishing in one spot. There had been many fish jumping as I paddled along, and I wondered if the fishermen noticed that. They’re curious anyway, at being passed when they are under way. Every fisherman has been polite, almost every boater has taken care not to pass by close. The couple of times there were jerks involved exclusively jet skiiers or wakeboaters.
So be courteous to fishermen. Around here, they’re potential allies against jetskiiers, who they dislike as much as paddlers do.