I stopped writing for years because I was made to feel guilty about that solitary time. I felt that it was a stupid, useless hobby and a venture not worth undertaking, because I was told so, and I believed him (and others) that I had more important things to do with my time. I resented it. I resented myself. and everyone who ever told me so, or treated me in a way that resembled that.
I divorced one of those naysayer voices in fact. (that wasn't the only reason). But I let someone do that to me,
and I did it to myself. I didn't defend my right to do what I loved. I believed him when he said it wasn't worth it and stupid.
Well I started again, and I do have a lot of important things to do, a regular job to keep because it provides the insurance and money we need to support our family. My house needs to be cleaned, I let it slip when I'm in writer mode. Can't help it. When it's time to feed the manic I have to do it. But you know what? I will never feel guilty about writing again. Thankfully I'm with someone now who respects my need to write and encourages it. He deals with my "writer mood swings".
But it took a lot of convincing myself even after leaving a bad situation where I allowed myself to be convinced it was "wrong" of me to enjoy writing. (in fact it took some total dipsh#t at work to tell me that he was writing a novel. and I thought, Trish, if this guy can do it, why the hell aren't you?)
I did it to myself, I blame no one else. I let it happen. Years of "yea trish, those are some pretty high hopes" had worn me down, but I've shut them out.
Not now. Not ever. I don't care if anything I ever write never gets published, I hope it will, but it's the joy of the craft, baby! Pure and simple. I'll write till my hands or brain can't do it anymore.
Life's too short not to do what you love, and my imagination requires a page.
The guilt you feel is that part of you that tells you that you're not good enough. Don't listen. Write. Often. However it suits you.
Icerose, I have two kids, young ones. I take my time for them. I love to spend time with them. I write when they go to bed, before they wake up in the morning, at lunch during my day job, at nap time on the weekends, and sometimes, while they play in the family room, I bring along my notebook and write 1000 words watching them play. YOu don't have to neglect family to write. Don't let anyone tell you different.
My husband loves to hunt and fish. I told him that he leaves the house for his passionate hobbies, and that I am here for mine, but he needs to pretend at least for a little while every couple of days that I'm not even here. If he can go away for a whole saturday to fish or go bow hunting and sit in the woods for a whole weekend, I can write.

Don't let anyone tell you not to do it. There is time, you just have to find it.