My advice, much of which has already been offered by various others:
Learn to time your paddle strokes to counteract the turning effects of the waves. Forget about maintaining your usual cadence, just time your strokes with the waves.
Don’t try to paddle too hard; it’s more important to keep your balance. If you try too hard you can end up mis-timing your stroke and catching air, which is definitely destabilizing.
Your skeg looks a little different from mine, but as a general rule adjustable skegs should be in the full-down position to go downwind in strong winds, mostly-down to go almost-downwind, half-down to go perpendicular to the wind, etc. This is to prevent the situation where you have to paddle harder on one side than the other just to keep going in the direction you’ve chosen. It’s not specifically about the big-wave effects you asked about, but NOT doing it just makes your life harder.
If your bow seems to bury in the troughs, as someone else here mentioned, try putting some extra weight in the rear of your kayak. If that doesn’t work, you may have to choose a different direction of travel to gain a better experience.
To reduce any tendency for the bow to bury in the troughs, you could consider getting a kayak with more bow volume. I paddle a Current Designs Caribou, which has pretty good bow volume (the shape is convex all the way to the bow) and handles downwind travel very nicely. Its length may also be a factor; it does a lot less gyrating than does my Hobie Revolution 13. The Hobie’s gyrations are not a real problem for me, though, I just keep the distant horizon in view and stay flexible at the hips and nothing bad happens. It has a lot of bow volume, too.
Last weekend I found myself paddling with a tailwind that was at times at a bit of an angle and although the waves weren’t monsters, they were big enough to be adding some push, but not really big enough to surf to any benefit. What I found most useful was to just paddle at my normal pace and shift the paddle a few inches to add more power where needed to keep the boat on course.
I have learned that when trying to cover a distance, trying to catch rides is somewhat tiring and picking just the right waves is too frustrating. I get a better ride from just running over the waves. The boat I use the most in these conditions is long enough that it just spans the troughs and essentially just takes off and skips across the tops of the waves. I keep paddling, by grabbing a paddle full on top of each wave as I’m running over it. Most of the time, the wave sets peter out after a bit and I have to wait for another set to build. Well, really I don’t wait; I just keep at my same pace and pretty soon another good set catches up.
My NC Expedition has no skeg, and no rudder, but about the last three feet of the stern is a tail fin and it does a great job of keeping the boat on course.
I have broached the boat from time to time when I got sloppy and was very happy that the boat is fine sliding sideways until I get it turned back on course.
The one time when my normal downwind technique in confused seas didn’t work was on Lake Superior at the end of a three day trip to some of the Apostle islands. My son and I were each paddling NDK Explorers. We had left our camp site on Devil’s Island in a hurry as there was a sudden weather forecast of a front coming with strong northerly winds. We reckoned we just had time to get back to Sand Bay before the front hit us. We nearly made it. Half a mile out from the beach we were in shallow water, and I could see and hear the front approaching behind us, and we were soon surfing in confused short steep seas coming from several directions. At one point my paddle happened to be immersed on my right side when a short steep peaked crest leapt up into a peak on that side and threw me leftwards down the face of the wave, sending me looking down into the deep trough on the left. I couldn’t brace to the right as I was tipped over on on my left side and couldn’t reach the water. If I tried to brace on the left, the weight transfer would certainly have capsized me, and in any case the water flow would have been up the wave face, a recipe for instant capsize. I had no idea how to deal with the situation, so I just froze very still, and hoped. When the wave subsided, the kayak righted herself, and I paddled safely to the take-out in a torrential tropical style downpour. It pays to have a well-designed sea kayak under you.
Nick.