PostgreSQL pg_ctl start超時分析

一、問題
pg_ctl start啟動時報錯退出：pg_ctl：server did not start in time。超時時間是多少？從什么時候到哪個階段算超時？

二、分析：該信息打印位置，從后面代碼段do_start函數(shù)中可以看出
1、pg_ctl start調(diào)用start_postmaster啟動PG的主進程后，每隔0.1ms檢查一次postmaster.pid文件，是否已寫入ready/standby
2、總共會檢查600次，即從啟動主進程后，最多等待60s，如果沒有寫入ready/standby則打印上述日志并退出
3、默認等待時間是60s，如果pg_ctl start -t指定等待時間，則等待時間為該指定時間

三、什么時候postmaster.pid文件寫入ready/standby
1、如果是主機不管有沒有設置hot standby
?? ?1）當startup進程恢復完成退出時，調(diào)用proc_exit函數(shù)向主進程發(fā)送SIGCHLD信號并退出
?? ?2）主進程接收到信號后，signal處理函數(shù)reaper調(diào)用AddToDataDirLockFile向postmaster.pid文件寫入ready
2、如果是備機即data目錄下有recovery.cnf文件，且設置了hot standby，在實際恢復前沒有到達一致性位置
?? ?1）startup進程向主進程發(fā)送PMSIGNAL_RECOVERY_STARTED信號，主進程調(diào)用信號處理函數(shù)sigusr1_handler，將pmState=PM_RECOVERY
?? ?2）每次讀取下一個xlog前都會調(diào)用CheckRecoveryConsistency函數(shù)進行一致性檢查：
?? ??? ?2.1 進入一致性狀態(tài)，starup進程向主進程發(fā)送PMSIGNAL_BEGIN_HOT_STANDBY信號，主進程接收到信號后調(diào)用sigusr1_handler->AddToDataDirLockFile向postmaster.pid文件寫入ready
3、如果是備機即data目錄下有recovery.cnf文件，且設置了hot standby，在實際恢復前沒有到達一致性位置
?? ?1）startup進程向主進程發(fā)送PMSIGNAL_RECOVERY_STARTED信號，主進程調(diào)用信號處理函數(shù)sigusr1_handler，將pmState=PM_RECOVERY
?? ?2）每次讀取下一個xlog前都會調(diào)用CheckRecoveryConsistency函數(shù)進行一致性檢查。如果沒有進入一致性狀態(tài)
?? ?3）本地日志恢復完成，切換日志源時同樣調(diào)用CheckRecoveryConsistency函數(shù)進行一致性檢查
?? ??? ?3.1 進入一致性狀態(tài)，starup進程向主進程發(fā)送PMSIGNAL_BEGIN_HOT_STANDBY信號，主進程接收到信號后調(diào)用sigusr1_handler->AddToDataDirLockFile向postmaster.pid文件寫入ready
4、如果是備機即data目錄下有recovery.cnf文件，且設置了hot standby，在實際恢復前到達一致性位置
?? ?1）startup進程向主進程發(fā)送PMSIGNAL_RECOVERY_STARTED信號，主進程調(diào)用信號處理函數(shù)sigusr1_handler，將pmState=PM_RECOVERY
?? ?2）CheckRecoveryConsistency函數(shù)進行一致性檢查，向主進程發(fā)送PMSIGNAL_BEGIN_HOT_STANDBY信號，主進程接收到信號后調(diào)用sigusr1_handler->AddToDataDirLockFile向postmaster.pid文件寫入ready
5、如果是備機即data目錄下有recovery.cnf文件，沒有設置hot standby
?? ?1）startup進程向主進程發(fā)送PMSIGNAL_RECOVERY_STARTED信號
?? ?2）主進程接收到信號后，向postmaster.將pmState=PM_RECOVERY

四、代碼分析
1、pg_ctl start流程

do_start->
    pm_pid = start_postmaster();
    if (do_wait){
        print_msg(_("waiting for server to start..."));
        switch (wait_for_postmaster(pm_pid, false)){
            case POSTMASTER_READY:
                print_msg(_(" done\n"));
                print_msg(_("server started\n"));
                break;
            case POSTMASTER_STILL_STARTING:
                print_msg(_(" stopped waiting\n"));
                write_stderr(_("%s: server did not start in time\n"), progname);
                exit(1);
                break;
            case POSTMASTER_FAILED:
                print_msg(_(" stopped waiting\n"));
                write_stderr(_("%s: could not start server\n" "Examine the log output.\n"), progname);
                exit(1);
                break;
        }
    }else
        print_msg(_("server starting\n"));

wait_for_postmaster->
    for (i = 0; i < wait_seconds * WAITS_PER_SEC; i++){
        if ((optlines = readfile(pid_file, &numlines)) != NULL && numlines >= LOCK_FILE_LINE_PM_STATUS){
            pmpid = atol(optlines[LOCK_FILE_LINE_PID - 1]);
            pmstart = atol(optlines[LOCK_FILE_LINE_START_TIME - 1]);
            if (pmstart >= start_time - 2 && pmpid == pm_pid){
                char       *pmstatus = optlines[LOCK_FILE_LINE_PM_STATUS - 1];
                if (strcmp(pmstatus, PM_STATUS_READY) == 0 || strcmp(pmstatus, PM_STATUS_STANDBY) == 0){
                    /* postmaster is done starting up */
                    free_readfile(optlines);
                    return POSTMASTER_READY;
                }
            }
        }
        free_readfile(optlines);
        if (waitpid((pid_t) pm_pid, &exitstatus, WNOHANG) == (pid_t) pm_pid)
            return POSTMASTER_FAILED;
        pg_usleep(USEC_PER_SEC / WAITS_PER_SEC);
    }
    /* out of patience; report that postmaster is still starting up */
    return POSTMASTER_STILL_STARTING;

2、server主進程及信號處理函數(shù)

PostmasterMain->
    pqsignal_no_restart(SIGUSR1, sigusr1_handler);  /* message from child process */
    pqsignal_no_restart(SIGCHLD, reaper);   /* handle child termination */
    ...
    StartupXLOG();
    ...
    proc_exit(0);//exit函數(shù)向主進程發(fā)送SIGCHLD信號

reaper->//進程終止或者停止的信號
    AddToDataDirLockFile(LOCK_FILE_LINE_PM_STATUS, PM_STATUS_READY);

postmaster進程接收信號：
sigusr1_handler->
    if (CheckPostmasterSignal(PMSIGNAL_RECOVERY_STARTED) &&
        pmState == PM_STARTUP && Shutdown == NoShutdown){
        CheckpointerPID = StartCheckpointer();
        BgWriterPID = StartBackgroundWriter();
        if (XLogArchivingAlways())
            PgArchPID = pgarch_start();
        //hot_standby在postgresql.conf文件中配置TRUE
        //表示在恢復的時候允許連接
        if (!EnableHotStandby){
            //將standby寫入postmaster.pid文件，表示up但不允許連接
            AddToDataDirLockFile(LOCK_FILE_LINE_PM_STATUS, PM_STATUS_STANDBY);
        }
        pmState = PM_RECOVERY;
    }
    if (CheckPostmasterSignal(PMSIGNAL_BEGIN_HOT_STANDBY) &&
        pmState == PM_RECOVERY && Shutdown == NoShutdown){
        PgStatPID = pgstat_start();
        //將ready寫入postmaster.pid文件，允許連接
        AddToDataDirLockFile(LOCK_FILE_LINE_PM_STATUS, PM_STATUS_READY);
        pmState = PM_HOT_STANDBY;
    }
    ...

3、Startup進程

StartupXLOG->
    ReadCheckpointRecord
    if (ArchiveRecoveryRequested && IsUnderPostmaster){//有recovery.conf文件則ArchiveRecoveryRequested為TRUE
        //有recovery.conf文件則ArchiveRecoveryRequested為TRUE
        PublishStartupProcessInformation();
        SetForwardFsyncRequests();
        //向master進程發(fā)送PMSIGNAL_RECOVERY_STARTED信號
        SendPostmasterSignal(PMSIGNAL_RECOVERY_STARTED);
        bgwriterLaunched = true;
    }
    CheckRecoveryConsistency();-->...
    |-- if (standbyState == STANDBY_SNAPSHOT_READY && !LocalHotStandbyActive &&
    |       reachedConsistency && IsUnderPostmaster){
    |       SpinLockAcquire(&XLogCtl->info_lck);
    |       XLogCtl->SharedHotStandbyActive = true;
    |       SpinLockRelease(&XLogCtl->info_lck);
    |       LocalHotStandbyActive = true;
    |       SendPostmasterSignal(PMSIGNAL_BEGIN_HOT_STANDBY);
    |-- }
    ...
    回放一個record后，每次讀取下一個record前都會調(diào)用CheckRecoveryConsistency