Almost everybody has ancestors that were enslaved. E.g. most Germanic people where enslaved by other Germanic people, or the Romans at some point in time. The only question is how far back you have to look and how far back you can look.

Thing is... nobody really has a valid reason to be offended. In this case context matters and the English language is an evolving construct.