First draft, you can't always tell. Leave it in.
In revisions, view everything with a critical eye. Is it really adding anything, or are you just so in love with your character's hobby of knitting asbestos socks for orphaned dragons that you completely forget your story's a medical thriller? Sometimes, a little sidetracking is good. It can establish character or create some needed down time. But if it goes on long enough that you (or your beta readers) start skimming, cut it back, or cut it out.
If you yourself are calling it a "tangent," though, that's not a great sign. It's like calling something an "infodump." They're terms that generally indicate something's gone wrong...