Elasticsearch集群40億級優(yōu)化

目前架構(gòu)：

n臺filebeat客戶端來將每臺應(yīng)用上的日志傳到kafka，3臺kafka做集群用于日志隊列，四臺ES做集群，前兩臺存放近兩天熱數(shù)據(jù)日志，后兩臺存放兩天前的歷史日志，數(shù)據(jù)保存一個月，目前總數(shù)據(jù)量44億，大小為6T。logstash與kibana與ES在一臺機(jī)器上，kibana域名指向后端三個kibana做輪詢。

出現(xiàn)性能問題：

1、集群中只有第一臺負(fù)載很高，其他節(jié)點負(fù)載一直都很低，偶爾同為hot數(shù)據(jù)節(jié)點的第二臺負(fù)載也會稍微有點升高。

2、隊列經(jīng)常堵塞，kafka中uat,pet,prd三個環(huán)境的topic同在一個默認(rèn)的logstash消費組。只要其中一個環(huán)境的列隊積壓，其他環(huán)境的隊列就無法消費了。

3、Kibana登陸后首頁打開，需要至少半分鐘，日志查詢也很慢，至少幾分鐘才會出結(jié)果。

4、有時候ES常因負(fù)載高而脫離集群，導(dǎo)致集群節(jié)點數(shù)據(jù)重新分配，集群狀態(tài)顏色為RED，同時kibana頁面打開時顯示Red報錯。kibana頁面間斷無法打開的情況約持續(xù)一兩周。

目前ELK中發(fā)現(xiàn)有些索引查詢有點慢，于是打開ES索引查詢?nèi)罩緛碛涗浡樵?，進(jìn)而對慢查詢?nèi)罩具M(jìn)行分析，定位問題。慢日志內(nèi)容如下：

[2017-08-28T11:21:02,377][WARN ][index.search.slowlog.query] [node-3] [logstash-nginx-2017.08.01][4] took[15s], took_millis[15029], types[], stats[], search
_type[QUERY_THEN_FETCH], total_shards[140], source[{"size":0,"query":{"bool":{"filter":[{"match_none":{"boost":1.0}},{"query_string":{"query":"NOT status:200  OR  NOT
status:304","fields":[],"use_dis_max":true,"tie_breaker":0.0,"default_operator":"or","auto_generate_phrase_queries":false,"max_determined_states":10000,"enable_position
_increment":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"analyze_wildcard":true,"escape":false,"split_on_whitespace":true,
"boost":1.0}}],"disable_coord":false,"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"3":{"terms":{"field":"status","size":5,"min_doc_count":0,"shard_min_doc_
count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_term":"asc"}]},"aggregations":{"2":{"date_histogram":{"field":"@timestamp","format":"epoch_mill
is","interval":"20m","offset":0,"order":{"_key":"asc"},"keyed":false,"min_doc_count":0,"extended_bounds":{"min":"1503886846372","max":"1503890446372"}}}}}}}],
[2017-08-28T11:21:02,377][WARN ][index.search.slowlog.query] [node-3] [logstash-nginx-2017.08.01][2] took[15.7s], took_millis[15787], types[], stats[], sear
ch_type[QUERY_THEN_FETCH], total_shards[140], source[{"size":0,"query":{"bool":{"filter":[{"match_none":{"boost":1.0}},{"query_string":{"query":"NOT status:200  OR  NOT
  status:304","fields":[],"use_dis_max":true,"tie_breaker":0.0,"default_operator":"or","auto_generate_phrase_queries":false,"max_determined_states":10000,"enable_positi
on_increment":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"analyze_wildcard":true,"escape":false,"split_on_whitespace":tru
e,"boost":1.0}}],"disable_coord":false,"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"3":{"terms":{"field":"status","size":5,"min_doc_count":0,"shard_min_do
c_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_term":"asc"}]},"aggregations":{"2":{"date_histogram":{"field":"@timestamp","format":"epoch_mi
llis","interval":"20m","offset":0,"order":{"_key":"asc"},"keyed":false,"min_doc_count":0,"extended_bounds":{"min":"1503886846372","max":"1503890446372"}}}}}}}],

下面進(jìn)行分析：

待續(xù)