I don't know if I'll ever write another sci-fi again for the reasons you cite. Fantasy after this, definitely. Either that or romance.
SERIOUSLY. What is my problem?? I'm great with characters/relationships. Worldbuilding is where I'm weakest (although I'm getting better at a painfully slow rate). So why the heck do I torture myself with ridiculously involved sci-fi stories?
I've had several people ask me why I bother (when I'm complaining to them about worldbuilding and plot consistency). It makes me think of another thing my favorite teacher said: write the story you'd want someone to tell you. I realized that that's exactly what I'm doing. For better or worse, lol.