I hear "edgy" a lot in reference to my work. I think it generally means different from a societal norm - sarcastic, dark, twisted, about people who are "outcasts" in some form or another, or "urban".
Sometimes you see "edgy" referring to writers (primarily men) who are deliberately trying to be shocking, garish, or hard-edged. Chuck Palahniuk (author of Fight Club) would be a prime example of that, as would Bret Easton Ellis (American Psycho), Andrew Vachss (dark dark DARK), etc. But I've also heard the word in relation to anyone writing sharp, stylish prose in sort of a noir style, about the grittier elements of society, or even just the dark lining of the glamorous life. If Raymond Chandler were writing today he'd probably be called edgy.
The term "transgressive", I hear more referring to people writing about criminal stuff... not necessarily just writing a dark or pessimistic view of the world. Though both mean pretty much the same thing to different audiences.