Isn't it "My brother and I's loss?" I'm getting pretty confused myself.
You wouldn't say "I loss," or "I's loss." You would say "My loss." The rule for me, my, and I in cases of referring to oneself and a companion can be a bit confusing.
For instance:
My brother and I went to the movies. (Correct, because you can take out the brother and it reads: "I went to the movies.")
"My brother and me went to the movies" or "Me and my brother went to the movies." (Both of these are incorrect, as you wouldn't say "Me went to the movies.")
Are you coming with John and me? (Correct, because it can be reduced to "Are you coming with me?")
Are you coming with John and I? (Incorrect, because you would not say, "Are you coming with I?")
If you're confused, always take out the reference to the other person and see if "I" or "me" belongs.
Hope that helps!