您好,登錄后才能下訂單哦!
今天就跟大家聊聊有關(guān)Java如何實(shí)現(xiàn)基于模板的網(wǎng)頁結(jié)構(gòu)化信息精準(zhǔn)抽取組件HtmlExtractor,可能很多人都不太了解,為了讓大家更加了解,小編給大家總結(jié)了以下內(nèi)容,希望大家根據(jù)這篇文章可以有所收獲。
HtmlExtractor是一個Java實(shí)現(xiàn)的基于模板的網(wǎng)頁結(jié)構(gòu)化信息精準(zhǔn)抽取組件,本身并不包含爬蟲功能,但可被爬蟲或其他程序調(diào)用以便更精準(zhǔn)地對網(wǎng)頁結(jié)構(gòu)化信息進(jìn)行抽取。
HtmlExtractor由2個子項(xiàng)目構(gòu)成,html-extractor和html-extractor-web。 html-extractor實(shí)現(xiàn)了數(shù)據(jù)抽取邏輯,是從節(jié)點(diǎn),html-extractor-web提供web界面來維護(hù)抽取規(guī)則,是主節(jié)點(diǎn)。 html-extractor是一個jar包,可通過maven引用:
<dependency> <groupId>org.apdplat</groupId> <artifactId>html-extractor</artifactId> <version>1.0</version> </dependency>
html-extractor-web是一個war包,需要部署到Servlet/Jsp容器上。
//1、構(gòu)造抽取規(guī)則 List<UrlPattern> urlPatterns = new ArrayList<>(); //1.1、構(gòu)造URL模式 UrlPattern urlPattern = new UrlPattern(); urlPattern.setUrlPattern("http://money.163.com/\\d{2}/\\d{4}/\\d{2}/[0-9A-Z]{16}.html"); //1.2、構(gòu)造HTML模板 HtmlTemplate htmlTemplate = new HtmlTemplate(); htmlTemplate.setTemplateName("網(wǎng)易財(cái)經(jīng)頻道"); htmlTemplate.setTableName("finance"); //1.3、將URL模式和HTML模板建立關(guān)聯(lián) urlPattern.addHtmlTemplate(htmlTemplate); //1.4、構(gòu)造CSS路徑 CssPath cssPath = new CssPath(); cssPath.setCssPath("h2"); cssPath.setFieldName("title"); cssPath.setFieldDescription("標(biāo)題"); //1.5、將CSS路徑和模板建立關(guān)聯(lián) htmlTemplate.addCssPath(cssPath); //1.6、構(gòu)造CSS路徑 cssPath = new CssPath(); cssPath.setCssPath("div#endText"); cssPath.setFieldName("content"); cssPath.setFieldDescription("正文"); //1.7、將CSS路徑和模板建立關(guān)聯(lián) htmlTemplate.addCssPath(cssPath); //可象上面那樣構(gòu)造多個URLURL模式 urlPatterns.add(urlPattern); //2、獲取抽取規(guī)則對象 ExtractRegular extractRegular = ExtractRegular.getInstance(urlPatterns); //注意:可通過如下3個方法動態(tài)地改變抽取規(guī)則 //extractRegular.addUrlPatterns(urlPatterns); //extractRegular.addUrlPattern(urlPattern); //extractRegular.removeUrlPattern(urlPattern.getUrlPattern()); //3、獲取HTML抽取工具 HtmlExtractor htmlExtractor = HtmlExtractor.getInstance(extractRegular); //4、抽取網(wǎng)頁 String url = "http://money.163.com/08/1219/16/4THR2TMP002533QK.html"; List<ExtractResult> extractResults = htmlExtractor.extract(url, "gb2312"); //5、輸出結(jié)果 int i = 1; for (ExtractResult extractResult : extractResults) { System.out.println((i++) + "、網(wǎng)頁 " + extractResult.getUrl() + " 的抽取結(jié)果"); for(ExtractResultItem extractResultItem : extractResult.getExtractResultItems()){ System.out.print("\t"+extractResultItem.getField()+" = "+extractResultItem.getValue()); } System.out.println("\tdescription = "+extractResult.getDescription()); System.out.println("\tkeywords = "+extractResult.getKeywords()); }
1、運(yùn)行主節(jié)點(diǎn),負(fù)責(zé)維護(hù)抽取規(guī)則: 將子項(xiàng)目html-extractor-web打成War包然后部署到Tomcat。 2、獲取一個HtmlExtractor的實(shí)例(從節(jié)點(diǎn)),示例代碼如下:
String allExtractRegularUrl = "http://localhost:8080/HtmlExtractorServer/api/all_extract_regular.jsp"; String redisHost = "localhost"; int redisPort = 6379; HtmlExtractor htmlExtractor = HtmlExtractor.getInstance(allExtractRegularUrl, redisHost, redisPort);
3、抽取信息,示例代碼如下:
String url = "http://money.163.com/08/1219/16/4THR2TMP002533QK.html"; List<ExtractResult> extractResults = htmlExtractor.extract(url, "gb2312"); int i = 1; for (ExtractResult extractResult : extractResults) { System.out.println((i++) + "、網(wǎng)頁 " + extractResult.getUrl() + " 的抽取結(jié)果"); for(ExtractResultItem extractResultItem : extractResult.getExtractResultItems()){ System.out.print("\t"+extractResultItem.getField()+" = "+extractResultItem.getValue()); } System.out.println("\tdescription = "+extractResult.getDescription()); System.out.println("\tkeywords = "+extractResult.getKeywords()); }
看完上述內(nèi)容,你們對Java如何實(shí)現(xiàn)基于模板的網(wǎng)頁結(jié)構(gòu)化信息精準(zhǔn)抽取組件HtmlExtractor有進(jìn)一步的了解嗎?如果還想了解更多知識或者相關(guān)內(nèi)容,請關(guān)注億速云行業(yè)資訊頻道,感謝大家的支持。
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點(diǎn)不代表本網(wǎng)站立場,如果涉及侵權(quán)請聯(lián)系站長郵箱:is@yisu.com進(jìn)行舉報,并提供相關(guān)證據(jù),一經(jīng)查實(shí),將立刻刪除涉嫌侵權(quán)內(nèi)容。