您好,登錄后才能下訂單哦!
python TF-IDF算法實現(xiàn)文本關(guān)鍵詞提?。肯嘈藕芏鄾]有經(jīng)驗的人對此束手無策,為此本文總結(jié)了問題出現(xiàn)的原因和解決方法,通過這篇文章希望你能解決這個問題。
TF-IDF算法步驟:
(1)、計算詞頻:
詞頻 = 某個詞在文章中出現(xiàn)的次數(shù)
考慮到文章有長短之分,考慮到不同文章之間的比較,將詞頻進行標準化
詞頻 = 某個詞在文章中出現(xiàn)的次數(shù)/文章的總詞數(shù)
詞頻 = 某個詞在文章中出現(xiàn)的次數(shù)/該文出現(xiàn)次數(shù)最多的詞出現(xiàn)的次數(shù)
(2)、計算逆文檔頻率
需要一個語料庫(corpus)來模擬語言的使用環(huán)境。
逆文檔頻率 = log(語料庫的文檔總數(shù)/(包含該詞的文檔數(shù) + 1))
(3)、計算TF-IDF
TF-IDF = 詞頻(TF)* 逆文檔頻率(IDF)
詳細代碼如下:
#!/usr/bin/env python #-*- coding:utf-8 -*- ''' 計算文檔的TF-IDF ''' import codecs import os import math import shutil #讀取文本文件 def readtxt(path): with codecs.open(path,"r",encoding="utf-8") as f: content = f.read().strip() return content #統(tǒng)計詞頻 def count_word(content): word_dic ={} words_list = content.split("/") del_word = ["\r\n","/s"," ","/n"] for word in words_list: if word not in del_word: if word in word_dic: word_dic[word] = word_dic[word]+1 else: word_dic[word] = 1 return word_dic #遍歷文件夾 def funfolder(path): filesArray = [] for root,dirs,files in os.walk(path): for file in files: each_file = str(root+"//"+file) filesArray.append(each_file) return filesArray #計算TF-IDF def count_tfidf(word_dic,words_dic,files_Array): word_idf={} word_tfidf = {} num_files = len(files_Array) for word in word_dic: for words in words_dic: if word in words: if word in word_idf: word_idf[word] = word_idf[word] + 1 else: word_idf[word] = 1 for key,value in word_dic.items(): if key !=" ": word_tfidf[key] = value * math.log(num_files/(word_idf[key]+1)) #降序排序 values_list = sorted(word_tfidf.items(),key = lambda item:item[1],reverse=True) return values_list #新建文件夾 def buildfolder(path): if os.path.exists(path): shutil.rmtree(path) os.makedirs(path) print("成功創(chuàng)建文件夾!") #寫入文件 def out_file(path,content_list): with codecs.open(path,"a",encoding="utf-8") as f: for content in content_list: f.write(str(content[0]) + ":" + str(content[1])+"\r\n") print("well done!") def main(): #遍歷文件夾 folder_path = r"分詞結(jié)果" files_array = funfolder(folder_path) #生成語料庫 files_dic = [] for file_path in files_array: file = readtxt(file_path) word_dic = count_word(file) files_dic.append(word_dic) #新建文件夾 new_folder = r"tfidf計算結(jié)果" buildfolder(new_folder) #計算tf-idf,并將結(jié)果存入txt i=0 for file in files_dic: tf_idf = count_tfidf(file,files_dic,files_array) files_path = files_array[i].split("//") #print(files_path) outfile_name = files_path[1] #print(outfile_name) out_path = r"%s//%s_tfidf.txt"%(new_folder,outfile_name) out_file(out_path,tf_idf) i=i+1 if __name__ == '__main__': main()
看完上述內(nèi)容,你們掌握python TF-IDF算法實現(xiàn)文本關(guān)鍵詞提取的方法了嗎?如果還想學到更多技能或想了解更多相關(guān)內(nèi)容,歡迎關(guān)注億速云行業(yè)資訊頻道,感謝各位的閱讀!
免責聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點不代表本網(wǎng)站立場,如果涉及侵權(quán)請聯(lián)系站長郵箱:is@yisu.com進行舉報,并提供相關(guān)證據(jù),一經(jīng)查實,將立刻刪除涉嫌侵權(quán)內(nèi)容。