自分 does not exist. He is but a figment of imagination. He is an existence that is fictional to fictional characters. He truly does not exist. 自分 cannot affect 相手 nor can 相手 affect 自分, since 自分 truly does not exist in any way shape or form. Since 自分 does not exist even to fictional beings, it is only natural that 相手 cannot affect 自分. But that doesn't matter since 自分 also can't affect 相手.