您好,登錄后才能下訂單哦!
本篇文章為大家展示了Pig怎么實現(xiàn)關(guān)鍵詞匹配,內(nèi)容簡明扼要并且容易理解,絕對能使你眼前一亮,通過這篇文章的詳細(xì)介紹希望你能有所收獲。
1. 問題描述
收集日志avro數(shù)據(jù)中有兩個Map字段appInstall、appUse分別表示已安裝的app、正在使用的app,且key值為app的名稱,value值為app使用信息?,F(xiàn)在要得到一份匹配上購物類app支付寶|京東|淘寶|天貓的用戶名單;MapReduce 解決辦法如下:
public static class M extends Mapper{ Text text = new Text(); @SuppressWarnings("unchecked") @Override protected void map(String key, Pair value, Context context) throws IOException, InterruptedException { Map data = value.fields.data; String dvc = data.get("dvc").toString(); MapappInstall = (Map) data.get("appInstall"); MapappUse = (Map) data.get("appUse"); for(String app: appInstall.keySet()) { if(app.matches("支付寶|京東|淘寶|天貓")) { text.set(appInstall.keySet().toString()); context.write(dvc, text); return; } } for(String app: appUse.keySet()) { if(app.matches("支付寶|京東|淘寶|天貓")) { text.set(appUse.keySet().toString()); context.write(dvc, text); return; } } } }
但是,如果要匹配游戲類的app、金融類的app類呢?如果匹配關(guān)鍵詞發(fā)生了變化呢?顯然,我們應(yīng)該將匹配關(guān)鍵詞開放成API,可以自由地匹配正則表達(dá)式。這時,pig派上了用場。
2. Bag正則匹配
A = load '//' using org.apache.pig.piggybank.storage.avro.AvroStorage(); -- A: {key: chararray,value: (fields: (data: map[]))} B = foreach A generate value.fields.data#'dvc' as dvc, value.fields.data#'appInstall' as ins:map[], value.fields.data#'appUse' as use:map[]; -- B: {dvc: bytearray,ins: map[],use: map[]} C = foreach B generate dvc, KEYSET(ins) as insk, KEYSET(use) as usek; -- C: {dvc: bytearray,insk: {(chararray)},usek: {(chararray)}}
在上述代碼中,load 數(shù)據(jù)轉(zhuǎn)換得到bag類型的app-set(insk與usek);但是,應(yīng)如何遍歷bag中的tuple與正則表達(dá)式做匹配呢?答案是UDF。
Apache DataFu Pig 提供了豐富的UDF,其中關(guān)于bags的UDF可以參看這里。TupleFromBag 提供根據(jù)index從bag提取tuple,支持三個輸入?yún)?shù)。依葫蘆畫瓢,遍歷bag匹配正則表達(dá)式的UDF如下:
package com.pig.udf.bag; /** * This UDF will return true if one tuple from a bag matches regex. * * There are two input parameter: * 1. DataBag * 2. Regex String */ public class BagMatchRegex extends FilterFunc { @Override public Boolean exec(Tuple tinput) throws IOException { try{ DataBag samples = (DataBag) tinput.get(0); String regex = (String) tinput.get(1); for (Tuple tuple : samples) { if(((String) tuple.get(0)).matches(regex)){ return true; } } } catch (Exception e) { return false; } return false; } }
其中,F(xiàn)ilterFunc為過濾UDF的基類,繼承于EvalFunc,即exec(Tuple tinput)的返回值必為Boolean類型。bag正則匹配的pig 腳本如下:
REGISTER ../piglib/udf-0.0.1-SNAPSHOT-jar-with-dependencies.jar define BagMatchRegex com.pig.udf.bag.BagMatchRegex(); A = load '/user/../current/*.avro' using org.apache.pig.piggybank.storage.avro.AvroStorage(); B = foreach A generate value.fields.data#'dvc' as dvc, value.fields.data#'appInstall' as ins:map[], value.fields.data#'appUse' as use:map[]; C = foreach B generate dvc, KEYSET(ins) as insk, KEYSET(use) as usek; D = filter C by BagMatchRegex(insk, '支付寶|京東|淘寶|天貓') or BagMatchRegex(usek, '支付寶|京東|淘寶|天貓');
3. 優(yōu)化
還有沒有可以做優(yōu)化的地方呢?我們先來看看pig中的KEYSET實現(xiàn):
package org.apache.pig.builtin; public class KEYSET extends EvalFunc{ private static final TupleFactory TUPLE_FACTORY = TupleFactory.getInstance(); @SuppressWarnings("unchecked") @Override public DataBag exec(Tuple input) throws IOException { if(input == null || input.size() == 0) { return null; } Mapm = null; // Input must be of type Map. This is verified at compile time m = (Map)(input.get(0)); if(m == null) { return null; } DataBag bag = new NonSpillableDataBag(m.size()); for (String s : m.keySet()) { Tuple t = TUPLE_FACTORY.newTuple(s); bag.add(t); } return bag; } ... }
需要指出的一點(diǎn)——pig的map數(shù)據(jù)類型是由Java類Map實現(xiàn)的。從KEYSET源碼中可以看出在調(diào)用時已經(jīng)將map遍歷了一次,然后在調(diào)用BagMatchRegex時又需要將key-set的bag再遍歷一次。其實,完全可以只用一次遍歷做map-key值的正則匹配:
package com.pig.udf.map; /** * This UDF will return true if map's key matches regex. * * There are two input parameter: * 1. Map * 2. Regex String */ public class KeyMatchRegex extends FilterFunc { @SuppressWarnings("unchecked") @Override public Boolean exec(Tuple input) throws IOException { try{ Mapm = null; // Input must be of type Map. This is verified at compile time m = (Map)(input.get(0)); String regex = (String) input.get(1); for (String key : m.keySet()) { if(key.matches(regex)){ return true; } } } catch (Exception e) { return false; } return false; } }
上述內(nèi)容就是Pig怎么實現(xiàn)關(guān)鍵詞匹配,你們學(xué)到知識或技能了嗎?如果還想學(xué)到更多技能或者豐富自己的知識儲備,歡迎關(guān)注億速云行業(yè)資訊頻道。
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點(diǎn)不代表本網(wǎng)站立場,如果涉及侵權(quán)請聯(lián)系站長郵箱:is@yisu.com進(jìn)行舉報,并提供相關(guān)證據(jù),一經(jīng)查實,將立刻刪除涉嫌侵權(quán)內(nèi)容。