The main problem with the slur filter is that it does neither consider context, nor considers delimiters. For example, when someone were to say the word for the purple fruit used to make wine, they’re not intending to use a word for the non-consensual sexual violation, yet the algorithm can’t see the difference when it detects the UTF-8 sequence 0x72 0x61 0x70 0x65, doing a hard-replace with the substring “removed” regardless of the context. Tom Scott once made a video about this phenomenon: it’s called String("The Sc" + "unthorpe problem").
Also, as you can see through this reply of mine, filters are pointless when there’s a plethora of ways to say the same thing without saying the actual word. Naive filters (e.g. RegExp-based match and replace) will just curb those who aren’t creative and/or knowledgeable enough, while affecting the experience for everyone who aren’t intending to do slurs, leading them to start using coded language and, thus, making it even harder to detect slurs as the slurers will eventually learn, through the non-slurers, that they can express the same thing without triggering the filter, until we get to a point in which the entire platform pivots to AI moderation, and even then there’s so many ways to express the thing without the LLM detecting, it’s called “steganography” and curbing this requires technical approaches known to be a Hard NP problem in computer science.
The main problem with the slur filter is that it does neither consider context, nor considers delimiters. For example, when someone were to say the word for the purple fruit used to make wine, they’re not intending to use a word for the non-consensual sexual violation, yet the algorithm can’t see the difference when it detects the UTF-8 sequence 0x72 0x61 0x70 0x65, doing a hard-replace with the substring “removed” regardless of the context. Tom Scott once made a video about this phenomenon: it’s called
String("The Sc" + "unthorpe problem").Also, as you can see through this reply of mine, filters are pointless when there’s a plethora of ways to say the same thing without saying the actual word. Naive filters (e.g. RegExp-based match and replace) will just curb those who aren’t creative and/or knowledgeable enough, while affecting the experience for everyone who aren’t intending to do slurs, leading them to start using coded language and, thus, making it even harder to detect slurs as the slurers will eventually learn, through the non-slurers, that they can express the same thing without triggering the filter, until we get to a point in which the entire platform pivots to AI moderation, and even then there’s so many ways to express the thing without the LLM detecting, it’s called “steganography” and curbing this requires technical approaches known to be a Hard NP problem in computer science.
!fedimemes@feddit.uk