I can notice a big difference with the rudder down with speed. I often use the rudder due to my inexperience and the windy conditions. I have a dagger and the rudder is very long. Would a smaller rudder make a difference?
From my experience - yes
I have a Perception Eclipse with a large rudder, and I have a QCC-700 with a small rudder.
I experimented with the Eclipse and noted like you did that it definately slowed the yak down. When I raced with it I used to keep it up on the straight aways and down coming into a turn, (providing I could waste a stroke to get it up or down)
With the smaller seal line rudder on the QCC I cannot notice any difference in speed.
The trade off is in racing on bouy turns I could turn the Eclipse on a dime without any body motion, while I have to use a lot with the QCC rudder.
With the QCC, I keep the rudder down from start to finish.
This is just my take and others might feel different.
I used to paddle a Javelin, K-1 with a small understern rudder. It felt much more responsive & quicker, than the same boat, with a larger overstern. I always cut down my oversterns. I now paddle a 21' "Thunderbolt" with a small understern (approx 3.5x2) about 4' from the stern. I may be sitting home this weekend because I didn't put on the overstern, but I don't feel the boat would handle near as well with the overstern. You will need a deeper rudder in open water, if there is a chance it might come out of the water. Bigger rudder=more drag! Good luck.