It is possible, and does happen. They are able to do this by taking advantage of our psychological weaknesses.

But usually, the cause of the loss of vitality is our own psychological processing, whether we are aware of it or not. Habits of lust, anger, envy, fear, etc. waste enormous energy. Also, having egotistical expectations or desires can interfere with sexual intimacy.

So, whichever way it is happening, the answer is the same: change your psyche. If you remove your psychological vulnerability, the black magician cannot manipulate you, nor can your ego.