It's not always you. Sometimes it's simply, just, them.
If you swam near a school of sharks and got attacked, would you tie it to some esoteric cause involving you?
You got attacked because you were near sharks and sharks bite.
Certain people are like that, too. Get too close to them and you'll get eaten alive.
Yes, of course, often we learn from that kind of an experience.
But why find reasons why you were mysteriously at fault or how you "attracted" the calamity?