您好,登錄后才能下訂單哦!
這篇文章主要介紹了thinkphp5.1怎么實(shí)現(xiàn)多線程爬蟲,具有一定借鑒價(jià)值,感興趣的朋友可以參考下,希望大家閱讀完這篇文章之后大有收獲,下面讓小編帶著大家一起了解一下。
創(chuàng)建一個(gè)cli命令
php think make:command Thread thread
測(cè)試能否成功執(zhí)行
php think thread
文檔地址:guzzle文檔地址(https://guzzle-cn.readthedocs.io/zh_CN/latest/quickstart.html)
<?php /** * Created by. * User: Jim * Date: 2020/9/29 * Time: 14:31 */ namespace app\command; use GuzzleHttp\Client; use GuzzleHttp\Pool; use think\console\Command; use think\console\Input; use think\console\Output; /** * Guzzle * Class Thread * @package app\command * 文檔地址 https://guzzle-cn.readthedocs.io/zh_CN/latest/quickstart.html */ class Thread extends Command { /** * 請(qǐng)求的總次數(shù) * @var int */ protected $totalPageCount = 50; /** * 當(dāng)前請(qǐng)求的次數(shù) * @var int */ protected static $counter = 1; /** * 線程的數(shù)量 * @var int */ protected $threads = 20; protected function configure() { // 指令配置 $this->setName('thread'); // 設(shè)置參數(shù) } protected function execute(Input $input, Output $output) { $client = new Client(); $requests = function ($total) use ($client) { foreach (range(1, $total) as $r) { $uri = 'https://apinew.juejin.im/content_api/v1/short_msg/detail'; yield function () use ($client, $uri) { return $client->postAsync($uri, [ 'verify' => false, 'json' => [ 'msg_id' => '6845185452727599118' ] ]); }; } }; $pool = new Pool($client, $requests($this->totalPageCount), [ 'concurrency' => $this->threads, // 請(qǐng)求成功 'fulfilled' => function ($response, $index) use ($output) { $res = $response->getBody()->getContents(); $output->writeln($res); $output->writeln("正在執(zhí)行第{$index}個(gè)·····"); if ($this->checkThreadIsEnd() == true) { $output->writeln("------------請(qǐng)求結(jié)束---------"); return false; } }, // 請(qǐng)求失敗 'rejected' => function ($reason, $index) use ($output) { $output->writeln("執(zhí)行失敗,{$reason}"); }, ]); $promise = $pool->promise(); $promise->wait(); } /** * 檢測(cè)任務(wù)是否結(jié)束 * @return bool */ private function checkThreadIsEnd() { if (self::$counter < $this->totalPageCount) { self::$counter++; return false; } else { return true; } } }
php think thread
感謝你能夠認(rèn)真閱讀完這篇文章,希望小編分享的“thinkphp5.1怎么實(shí)現(xiàn)多線程爬蟲”這篇文章對(duì)大家有幫助,同時(shí)也希望大家多多支持億速云,關(guān)注億速云行業(yè)資訊頻道,更多相關(guān)知識(shí)等著你來(lái)學(xué)習(xí)!
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng),如果涉及侵權(quán)請(qǐng)聯(lián)系站長(zhǎng)郵箱:is@yisu.com進(jìn)行舉報(bào),并提供相關(guān)證據(jù),一經(jīng)查實(shí),將立刻刪除涉嫌侵權(quán)內(nèi)容。