How rough is too rough.

[JasonFish]

I am heading down to San Diego this weekend from northern california. I was planning to bring my kayak and venture out of La Jolla to try my luck with yellowtail I see that the forecasts are calling for mixed swell 4-5 and wind from SW/SE building to 10-20 knts on the days I was planning to fish. It sounds a bit rough to me, but not having any experience with the area I didn't know if perhaps there is local relief from wind/swell off La Jolla. I also don't know about accuracy of forecasts. I know up here they are subject to drastic changes. The big question is do I leave the kayak at home? It would be a big bummer to go through the hassle of hauling it if I wasn't able to get out, but I have been really looking forward to giving this a go. Any advise is much appreciated.