>> f = NaiveFilter() >>> f.add( ..."/>
您好,登錄后才能下訂單哦!
一個(gè)簡單的實(shí)現(xiàn)
class NaiveFilter(): '''Filter Messages from keywords very simple filter implementation >>> f = NaiveFilter() >>> f.add("sexy") >>> f.filter("hello sexy baby") hello **** baby ''' def __init__(self): self.keywords = set([]) def parse(self, path): for keyword in open(path): self.keywords.add(keyword.strip().decode('utf-8').lower()) def filter(self, message, repl="*"): message = str(message).lower() for kw in self.keywords: message = message.replace(kw, repl) return message
其中strip() 函數(shù) 刪除附近的一些空格,解碼采用utf-8的形式,然后將其轉(zhuǎn)為小寫。
parse()函數(shù)就是打開文件,然后從中取各個(gè)關(guān)鍵詞,然后將其存在關(guān)鍵詞集合中。
filter()函數(shù)是一個(gè)過濾器函數(shù),其中將消息轉(zhuǎn)化為小寫,然后將關(guān)鍵詞替換成*。、
class BSFilter: '''Filter Messages from keywords Use Back Sorted Mapping to reduce replacement times >>> f = BSFilter() >>> f.add("sexy") >>> f.filter("hello sexy baby") hello **** baby ''' def __init__(self): self.keywords = [] self.kwsets = set([]) self.bsdict = defaultdict(set) self.pat_en = re.compile(r'^[0-9a-zA-Z]+$') # english phrase or not def add(self, keyword): if not isinstance(keyword, str): keyword = keyword.decode('utf-8') keyword = keyword.lower() if keyword not in self.kwsets: self.keywords.append(keyword) self.kwsets.add(keyword) index = len(self.keywords) - 1 for word in keyword.split(): if self.pat_en.search(word): self.bsdict[word].add(index) else: for char in word: self.bsdict[char].add(index) def parse(self, path): with open(path, "r") as f: for keyword in f: self.add(keyword.strip()) def filter(self, message, repl="*"): if not isinstance(message, str): message = message.decode('utf-8') message = message.lower() for word in message.split(): if self.pat_en.search(word): for index in self.bsdict[word]: message = message.replace(self.keywords[index], repl) else: for char in word: for index in self.bsdict[char]: message = message.replace(self.keywords[index], repl) return message
在上面的實(shí)現(xiàn)例子中,對于搜索查找進(jìn)行了優(yōu)化,對于英語單詞,直接進(jìn)行了按詞索引字典查找。對于其他語言模式,我們采用逐字符查找匹配的一種模式。
BFS:寬度優(yōu)先搜索方式。
class DFAFilter(): '''Filter Messages from keywords Use DFA to keep algorithm perform constantly >>> f = DFAFilter() >>> f.add("sexy") >>> f.filter("hello sexy baby") hello **** baby ''' def __init__(self): self.keyword_chains = {} self.delimit = '\x00' def add(self, keyword): if not isinstance(keyword, str): keyword = keyword.decode('utf-8') keyword = keyword.lower() chars = keyword.strip() if not chars: return level = self.keyword_chains for i in range(len(chars)): if chars[i] in level: level = level[chars[i]] else: if not isinstance(level, dict): break for j in range(i, len(chars)): level[chars[j]] = {} last_level, last_char = level, chars[j] level = level[chars[j]] last_level[last_char] = {self.delimit: 0} break if i == len(chars) - 1: level[self.delimit] = 0 def parse(self, path): with open(path,encoding='UTF-8') as f: for keyword in f: self.add(keyword.strip()) def filter(self, message, repl="*"): if not isinstance(message, str): message = message.decode('utf-8') message = message.lower() ret = [] start = 0 while start < len(message): level = self.keyword_chains step_ins = 0 for char in message[start:]: if char in level: step_ins += 1 if self.delimit not in level[char]: level = level[char] else: ret.append(repl * step_ins) start += step_ins - 1 break else: ret.append(message[start]) break else: ret.append(message[start]) start += 1 return ''.join(ret)
DFA即Deterministic Finite Automaton,也就是確定有窮自動(dòng)機(jī)。
使用了嵌套的字典來實(shí)現(xiàn)。
參考
Github:敏感詞過濾系統(tǒng)
以上就是本文的全部內(nèi)容,希望對大家的學(xué)習(xí)有所幫助,也希望大家多多支持億速云。
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點(diǎn)不代表本網(wǎng)站立場,如果涉及侵權(quán)請聯(lián)系站長郵箱:is@yisu.com進(jìn)行舉報(bào),并提供相關(guān)證據(jù),一經(jīng)查實(shí),將立刻刪除涉嫌侵權(quán)內(nèi)容。