怎么理解MySQL中半同步引起Master實(shí)例Crash

怎么理解MySQL中半同步引起Master實(shí)例Crash，很多新手對(duì)此不是很清楚，為了幫助大家解決這個(gè)難題，下面小編將為大家詳細(xì)講解，有這方面需求的人可以來(lái)學(xué)習(xí)下，希望你能有所收獲。

場(chǎng)景 : 
Crash發(fā)生時(shí)的數(shù)據(jù)庫(kù)版本: MySQL-5.7.12, 官方標(biāo)注在5.7.17進(jìn)行了fix;
開(kāi)啟半同步的主從架構(gòu)中, 從庫(kù)開(kāi)啟半同步, 啟動(dòng)/重啟slave線程導(dǎo)致Master實(shí)例Crash;

結(jié)論 :
mysql bug, 附上bug單鏈接: https://bugs.mysql.com/bug.php?id=79865

問(wèn)題描述(摘抄):

Description: From 5.7，semi-sync add Ack_receiver thread for listening slave ack，which use select(). But select() can only listen socket fd between 1 and __FD_SET_SIZE(my os is 1024),  when socket fd is bigger than __FD_SET_SIZE, select() has no effect, and can never get  ack from slave,then semi-sync can't run normally.even more,select() use array store fds, when use FD_SET store fd which is bigger than __FD_SET_SIZE, array will overflow，so mysqld may crash。

主要問(wèn)題就出在tcp連接的select方法, 通常, 操作系統(tǒng)通過(guò)宏FD_SET_SIZE來(lái)聲明一個(gè)進(jìn)程中select能操作的文件描述符的最大數(shù)據(jù), 然而通常情況下, 這個(gè)FD_SET_SIZE的值僅為1024;

實(shí)際上, 用epoll或者poll會(huì)比較少, select貌似是用的很少的;


問(wèn)題復(fù)現(xiàn):

準(zhǔn)備一套MySQL-5.7.12的主從架構(gòu), 開(kāi)啟半同步:


為了能盡量簡(jiǎn)單的啟用大量的文件描述符, 這里利用MyISAM分區(qū)表的"特性";


這時(shí)候在主庫(kù)上連續(xù)執(zhí)行select語(yǔ)句多次(>5);


這時(shí)候看一下主庫(kù)的文件描述符數(shù)量;


那么現(xiàn)在在開(kāi)啟半同步的從庫(kù)上重啟一下slave, 同時(shí)tail一下主庫(kù)的日志;


在重啟線程幾秒鐘之后, 主庫(kù)就發(fā)生了Crash;

PS: 在測(cè)試的過(guò)程中, 多次執(zhí)行了select語(yǔ)句, 然后確認(rèn)主庫(kù)的半同步狀態(tài)也是ON的情況下迅速在從庫(kù)上重啟slave, 基本是必現(xiàn)的;
PPS: MyISAM表在open的時(shí)候會(huì)同時(shí)打開(kāi)所有的分區(qū)文件, 所以能比較方便的模擬占用大量文件描述符的情景;
(MyISAM分區(qū)表: http://blog.itpub.net/29510932/viewspace-2134679/)
PPPPPPPS:  _(:з」∠)_


附上測(cè)試用的腳本與Crash的信息

1. CREATE TABLE `myisam_t` (
   

2.   `id` int(11) DEFAULT NULL
   

3. ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
   

4. /*!50100 PARTITION BY HASH (id)
   

5. PARTITIONS 2000 */

點(diǎn)擊(此處)折疊或打開(kāi)

2017-04-28T22:10:00.731611+08:00 5092 [Note] Start binlog_dump to master_thread_id(5092) slave_server(13043), pos(, 4)

2017-04-28T22:10:01.648365+08:00 5092 [Note] Start semi-sync binlog_dump to slave (server_id: 13043), pos(, 4)

*** buffer overflow detected ***: /usr/sbin/mysqld terminated

 Backtrace: 

/lib/x86_64-linux-gnu/libc.so.6(+0x731af)[0x7fcdfc7981af]

/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7fcdfc81dcf7]

/lib/x86_64-linux-gnu/libc.so.6(+0xf6f10)[0x7fcdfc81bf10]

/lib/x86_64-linux-gnu/libc.so.6(+0xf8c67)[0x7fcdfc81dc67]

/usr/lib/mysql/plugin/semisync_master.so(_ZN12Ack_receiver17get_slave_socketsEP6fd_set+0x83)[0x7fcc73d4a493]

/usr/lib/mysql/plugin/semisync_master.so(_ZN12Ack_receiver3runEv+0x603)[0x7fcc73d4aaf3]

/usr/lib/mysql/plugin/semisync_master.so(ack_receive_handler+0x19)[0x7fcc73d4aba9]

/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0xe90784]

/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7fcdfdf650a4]

/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fcdfc80d87d]

點(diǎn)擊(此處)折疊或打開(kāi)

1. 14:10:01 UTC - mysqld got signal 6 ;
   

2. This could be because you hit a bug. It is also possible that this binary
   

3. or one of the libraries it was linked against is corrupt, improperly built,
   

4. or misconfigured. This error can also be caused by malfunctioning hardware.
   

5. Attempting to collect some information that could help diagnose the problem.
   

6. As this is a crash and something is definitely wrong, the information
   

7. collection process might fail.
   

   
   

8. key_buffer_size=8388608
   

9. read_buffer_size=131072
   

10. max_used_connections=5
    

11. max_threads=9999
    

12. thread_count=8
    

13. connection_count=2
    

14. It is possible that mysqld could use up to
    

15. key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 21899362 K  bytes of memory
    

16. Hope that's ok; if not, decrease some variables in the equation.
    

    
    

17. Thread pointer: 0x0
    

18. Attempting backtrace. You can use the following information to find out
    

19. where mysqld died. If you see no messages after this, something went
    

20. terribly wrong...
    

21. stack_bottom = 0 thread_stack 0x40000
    

22. /usr/sbin/mysqld(my_print_stacktrace+0x2c)[0xe77fec]
    

23. /usr/sbin/mysqld(handle_fatal_signal+0x459)[0x7a7019]
    

24. /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7fcdfdf6c8d0]
    

25. /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7fcdfc75a067]
    

26. /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fcdfc75b448]
    

27. /lib/x86_64-linux-gnu/libc.so.6(+0x731b4)[0x7fcdfc7981b4]
    

28. /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7fcdfc81dcf7]
    

29. /lib/x86_64-linux-gnu/libc.so.6(+0xf6f10)[0x7fcdfc81bf10]
    

30. /lib/x86_64-linux-gnu/libc.so.6(+0xf8c67)[0x7fcdfc81dc67]
    

31. /usr/lib/mysql/plugin/semisync_master.so(_ZN12Ack_receiver17get_slave_socketsEP6fd_set+0x83)[0x7fcc73d4a493]
    

32. /usr/lib/mysql/plugin/semisync_master.so(_ZN12Ack_receiver3runEv+0x603)[0x7fcc73d4aaf3]
    

33. /usr/lib/mysql/plugin/semisync_master.so(ack_receive_handler+0x19)[0x7fcc73d4aba9]
    

34. /usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0xe90784]
    

35. /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7fcdfdf650a4]
    

36. /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fcdfc80d87d]
    

37. The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
    

38. information that should help you find out what is causing the crash.

看完上述內(nèi)容是否對(duì)您有幫助呢？如果還想對(duì)相關(guān)知識(shí)有進(jìn)一步的了解或閱讀更多相關(guān)文章，請(qǐng)關(guān)注億速云行業(yè)資訊頻道，感謝您對(duì)億速云的支持。