The group says his presence would be "an affront to veterans of the civil rights movement."