Most of the time I decline, especially for stores I rarely frequent.

I really appreciate your invitation, but I must decline it.

He offered to be her escort, but she declined it.

If the host does not do this, the people may already have something planned and therefore have to decline the invitation.

I shall have to refuse your invitation because of a prior engagement.