Don't let the gossip bother you brother, everyone makes tough decisions in their life. I'll stand behind anyone who puts their family first. At the end of the day this is a hobby for 99.99% of all of us and when life gets tough the luxury items are the first to be sold.
I have a loaner yak for you for those times you need to get on the water.