Do you do any exercising at a gym or anything? The reason I ask is you can do some workouts that will help build your core, and help with your balance while on the Kayaks. You want to keep you center of gravity in the middle of the boat. Lean too far one way or the other, and you could go in, but not likely. As for the Gym, or at home, get a stability ball, one that fits your height. If you buy one it will come with instructions on various exercises or you can search youtube for exercises. Doing crunches, sitting, and other things that force you to use your abs will build your core, also increasing stability.
As for the Kayaking part. Most of us have fished on boats, and rule number one is not to fall out of the boat, but in a Kayak you have to walk into the water before you can get into the boat. The first time getting in is always a little weird. Build your confidence take your kayak to areas in Mission bay that the speed is controlled. Practice your paddling and moving around and getting comfy. Check out these videos they will help you with your technique, and make your day more enjoyable.
http://www.youtube.com/watch?v=5YeduKtjFxA
http://www.youtube.com/watch?v=qZ9axwFqK0o
http://www.youtube.com/watch?v=VlsiHlRSYKA
http://www.youtube.com/watch?v=g2a2xu2VRxc
If you fall in
http://www.youtube.com/watch?v=NmaNBXEa_yQ
There are tons of videos that you can watch to help you. Trust me you want to know the tricks and it will help you feel more comfortable.