Rights
Rights simply are not a problem when pubishing a chapter from your novel in a magzine. Rights aren't even an issue. When you sell rights to a magazine, all you sell are First North American Rights, or possibly First North American Serial Rights.
These rights affect your novel in no way at all. They conflict with novel rights in no way at all, publishers could care less about them, and will, in fact, be grateful that the work is in the magazine because there's always a bleedover. . .people who read the chapter in the magazine are more likely to then go out and buy the novel. Book publishers LOVE it when a writer manages to have a novel excerpt placed in a good magazine.
I've been on this side of the business, and I can't say it any plainer. Rights are not a problem in any way, shape, or form. Zero, nada, zip.
And trademark has nothing at all to do with the issue. Nothing you write is trademarked until and unless you go through the process of applying for a trademark. And even if you do this, you will own the trademark. And no writer is going to use someone else's trademarked material in his own fiction. It just doesn't work this way. Your novel had better be your novel.
Trademarks simply play no part whatsoever in the process of first publishing a chapter of your novel in a magazine, and then publishing the novel.
Nor do any of the subsidiary rights, derivative rights, or any other form of rights apply to the very easy and straightforward process of having a chapter from a novel published in a magazine.
Really, this is not something out of the ordinary. Novel chapters are published in many, many magazines, particularly literary and mainstream magazines, on a regular basis, and any publisher loves the devil out of it. Having a chapter of your novel published in a magazine, if you can swing it, is a huge boost to you, a super plus for the publisher, and a selling point for the magazine.
Th rights in your novel contract are something to look at carefully, which is why it's dumb to go about it without an agent, but when talking about publishing a chapter of the novel in a magazine, rights are not only NOT a problem in any way, absolutely none of the rights involved conflict with each other at all. Period.