There are several options and, based upon conditions, shore terrain, surf, bottom conditions, currents, you have to make the best assessment at the time.
As someone said, first, learn to swim in the surf. To recognize tidal rips, to assess the waves, patterns, and longshore current before choosing a course of action. As Celia said, never get between the boat (or any heavy object) and shore. Some major injuries are possible when a heavy weight is dumped upon a swimmer.
Condition 1: Surf 4-5 feet, surf is regular and consistent, wind moderate (>5 MPH), water as you describe, separated from boat, still with paddle, shore safely approachable, boat not swamped and windblown:
Probably your best bet is to swim to shore and track the boat, since you will probably not catch up to the boat. Using the paddle to swim is quite possible in these conditions, but as stated, keep your hands below your shoulders to avoid dislocation injuries.
Condition 2: Same as condition 1 with following changes: dumping surf (all the wave energy is dumped in small area because of the steep shore), steep beach ending at a high bluff
Don’t let the boat get away. Very different situation with moderate changes to the variables. This is where I do use a paddle leash from paddle to boat (never to my person - I don’t like being attached to objects for a lot of reasons, especially a boat in the surf zone - so hold the paddle and the boat goes where you do). I’ve done this many times and your options are very limited. The beach offers little to no sanctuary. The dumping surf is dangerous and can break bones (even vertebra). You cannot use a paddle for a swim assist because when you hit the surf zone, you will probably have no control of your body and the paddle becomes a risk in itself. The shoreline is probably rocky and will not provide much buffering when the wave dumps upon your fragile body. This is not a beach where you want to land, even with the boat, so you need to get back in and find another spot to go. I’ve done plenty of entries through 3-4 foot dumping surf and I actually prefer to start in surfing the face of the wave, let the boat broach, and ride the foam into shore. Staying on the face of the wave will often lead to pealing (where the bow hits the bottom, often with catastrophic results (to boat, person, or both). If the boat is intact, you can recover from this event with a well done brace, but you are better off broaching to the wave before this can happen.
Condition 3: Swamp the boat, same as condition #1:
Recover the boat and re-enter. Easy decision as the swamped boat should be very retrievable.
So, just a few examples of what folks are trying to say here. You use your judgement to make the best call based upon your experience with the environment. Some additional points. In high surf, you will probably capsize on the wrong side of the boat (boat further out to sea than you are - you will only capsize when leaning into a brace if the brace fails for some reason which probably will not happen if the paddle remains intact). Move immediately either a) out of the way of the boat or b) underwater so the boat passes overhead. Often, the boat will pivot over you in a capsize (you are effectively a sea anchor when upside down if you are still in the seat and the boat will float right over you), but if you come out of the seat during the capsize (I see this a lot), the boat will move completely independent of you and become a very heavy threat to life and limb. Even swamped boats should float high enough to pass over you, so staying well underwater for a few moments can reduce the threat. Floating objects are often best avoided by diving under them (though, again, each case must be assessed and judgement applied).
So, what to do becomes a complicated question and it changes with even modest variations to conditions. As the saying goes, errors are caused by bad judgment; bad judgement is avoided by experience; experience is obtained from bad judgement.