shouldn't I at least wonder - why me, every single time, every single place, from every single group?

Not if you know anything about history or human nature.

Persecution of minority groups within a society - especially religious minorities - is so ridiculously prevalent in human history as to be basically a universal constant of human society. If you are a religious minority you would expect that every single time, every single place, you run the risk of being persecuted at some point. Any minority that attempts to maintain its own separate religious, cultural, or ethnic identity faces that.

Jews aren't unique in facing such persecution. Whether the Yazidis, the Huguenots, the Romani, Zoroastarians....countless groups across the eons and across the miles have been persecuted. Many of those groups simply get destroyed, like the smaller "heretic" religious groups that the formal Church just slaughtered all their adherents, or all of the pagan religions that were stamped out of existence over centuries through the more "civilized" mechanism of tormenting them until their adherents just gave up.

The Jews are "special" solely by dint of our longevity and geographical dispersion. We're one of the oldest religious minorities. While many religious minorities were also universally persecuted, they just didn't survive. Related to that, we've gone through numerous expulsions which dispersed us throughout the globe - the Babylonian captivity, the expulsion by the Romans from Judea, the expulsion from England in 1290, the expulsion from Spain in 1492, the forced relocation to the Pale of Settlement in the 1700's...all the way through the mass expulsion of most Jewish communities from north Africa and the Middle east in the 1940's. Because we got around more than the Zoroastrians or the Yazidi, we've just encountered more majority populations to persecute us.