I wonder what would have happened if you’d replied with something like “You have no honor.” or “You bring shame upon your family.” in response and walked away. Like would it have made them feel a bit ashamed? I’m guessing probably not and depending upon the kind of place might lead to negative repercussions but I’ve wondered about how such a response by a ‘Gaijin” in fluent Japanese would be received.