2022-08-04 06:59:33 +00:00
<!DOCTYPE HTML>
< html lang = "zh" class = "sidebar-visible no-js light" >
< head >
<!-- Book generated using mdBook -->
< meta charset = "UTF-8" >
< title > 示例: 并发的Web爬虫 - Go语言圣经< / title >
<!-- Custom HTML head -->
< meta content = "text/html; charset=utf-8" http-equiv = "Content-Type" >
< meta name = "description" content = "<The Go Programming Language>中文版" >
< meta name = "viewport" content = "width=device-width, initial-scale=1" >
< meta name = "theme-color" content = "#ffffff" / >
< link rel = "icon" href = "../favicon.svg" >
< link rel = "shortcut icon" href = "../favicon.png" >
< link rel = "stylesheet" href = "../css/variables.css" >
< link rel = "stylesheet" href = "../css/general.css" >
< link rel = "stylesheet" href = "../css/chrome.css" >
< link rel = "stylesheet" href = "../css/print.css" media = "print" >
<!-- Fonts -->
< link rel = "stylesheet" href = "../FontAwesome/css/font-awesome.css" >
< link rel = "stylesheet" href = "../fonts/fonts.css" >
<!-- Highlight.js Stylesheets -->
< link rel = "stylesheet" href = "../highlight.css" >
< link rel = "stylesheet" href = "../tomorrow-night.css" >
< link rel = "stylesheet" href = "../ayu-highlight.css" >
<!-- Custom theme stylesheets -->
< link rel = "stylesheet" href = "../style.css" >
< / head >
< body >
<!-- Provide site root to javascript -->
< script type = "text/javascript" >
var path_to_root = "../";
var default_theme = window.matchMedia("(prefers-color-scheme: dark)").matches ? "navy" : "light";
< / script >
<!-- Work around some values being stored in localStorage wrapped in quotes -->
< script type = "text/javascript" >
try {
var theme = localStorage.getItem('mdbook-theme');
var sidebar = localStorage.getItem('mdbook-sidebar');
if (theme.startsWith('"') & & theme.endsWith('"')) {
localStorage.setItem('mdbook-theme', theme.slice(1, theme.length - 1));
}
if (sidebar.startsWith('"') & & sidebar.endsWith('"')) {
localStorage.setItem('mdbook-sidebar', sidebar.slice(1, sidebar.length - 1));
}
} catch (e) { }
< / script >
<!-- Set the theme before any content is loaded, prevents flash -->
< script type = "text/javascript" >
var theme;
try { theme = localStorage.getItem('mdbook-theme'); } catch(e) { }
if (theme === null || theme === undefined) { theme = default_theme; }
var html = document.querySelector('html');
html.classList.remove('no-js')
html.classList.remove('light')
html.classList.add(theme);
html.classList.add('js');
< / script >
<!-- Hide / unhide sidebar before it is displayed -->
< script type = "text/javascript" >
var html = document.querySelector('html');
var sidebar = 'hidden';
if (document.body.clientWidth >= 1080) {
try { sidebar = localStorage.getItem('mdbook-sidebar'); } catch(e) { }
sidebar = sidebar || 'visible';
}
html.classList.remove('sidebar-visible');
html.classList.add("sidebar-" + sidebar);
< / script >
< nav id = "sidebar" class = "sidebar" aria-label = "Table of contents" >
< div class = "sidebar-scrollbox" >
< ol class = "chapter" > < li class = "chapter-item expanded affix " > < a href = "../index.html" > Go语言圣经< / a > < / li > < li class = "chapter-item expanded affix " > < a href = "../preface.html" > 前言< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch1/ch1.html" > < strong aria-hidden = "true" > 1.< / strong > 入门< / a > < / li > < li > < ol class = "section" > < li class = "chapter-item expanded " > < a href = "../ch1/ch1-01.html" > < strong aria-hidden = "true" > 1.1.< / strong > Hello, World< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch1/ch1-02.html" > < strong aria-hidden = "true" > 1.2.< / strong > 命令行参数< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch1/ch1-03.html" > < strong aria-hidden = "true" > 1.3.< / strong > 查找重复的行< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch1/ch1-04.html" > < strong aria-hidden = "true" > 1.4.< / strong > GIF动画< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch1/ch1-05.html" > < strong aria-hidden = "true" > 1.5.< / strong > 获取URL< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch1/ch1-06.html" > < strong aria-hidden = "true" > 1.6.< / strong > 并发获取多个URL< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch1/ch1-07.html" > < strong aria-hidden = "true" > 1.7.< / strong > Web服务< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch1/ch1-08.html" > < strong aria-hidden = "true" > 1.8.< / strong > 本章要点< / a > < / li > < / ol > < / li > < li class = "chapter-item expanded " > < a href = "../ch2/ch2.html" > < strong aria-hidden = "true" > 2.< / strong > 程序结构< / a > < / li > < li > < ol class = "section" > < li class = "chapter-item expanded " > < a href = "../ch2/ch2-01.html" > < strong aria-hidden = "true" > 2.1.< / strong > 命名< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch2/ch2-02.html" > < strong aria-hidden = "true" > 2.2.< / strong > 声明< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch2/ch2-03.html" > < strong aria-hidden = "true" > 2.3.< / strong > 变量< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch2/ch2-04.html" > < strong aria-hidden = "true" > 2.4.< / strong > 赋值< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch2/ch2-05.html" > < strong aria-hidden = "true" > 2.5.< / strong > 类型< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch2/ch2-06.html" > < strong aria-hidden = "true" > 2.6.< / strong > 包和文件< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch2/ch2-07.html" > < strong aria-hidden = "true" > 2.7.< / strong > 作用域< / a > < / li > < / ol > < / li > < li class = "chapter-item expanded " > < a href = "../ch3/ch3.html" > < strong aria-hidden = "true" > 3.< / strong > 基础数据类型< / a > < / li > < li > < ol class = "section" > < li class = "chapter-item expanded " > < a href = "../ch3/ch3-01.html" > < strong aria-hidden = "true" > 3.1.< / strong > 整型< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch3/ch3-02.html" > < strong aria-hidden = "true" > 3.2.< / strong > 浮点数< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch3/ch3-03.html" > < strong aria-hidden = "true" > 3.3.< / strong > 复数< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch3/ch3-04.html" > < strong aria-hidden = "true" > 3.4.< / strong > 布尔型< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch3/ch3-05.html" > < strong aria-hidden = "true" > 3.5.< / strong > 字符串< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch3/ch3-06.html" > < strong aria-hidden = "true" > 3.6.< / strong > 常量< / a > < / li > < / ol > < / li > < li class = "chapter-item expanded " > < a href = "../ch4/ch4.html" > < strong aria-hidden = "true" > 4.< / strong > 复合数据类型< / a > < / li > < li > < ol class = "section" > < li class = "chapter-item expanded " > < a href = "../ch4/ch4-01.html" > < strong aria-hidden = "true" > 4.1.< / strong > 数组< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch4/ch4-02.html" > < strong aria-hidden = "true" > 4.2.< / strong > Slice< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch4/ch4-03.html" > < strong aria-hidden = "true" > 4.3.< / strong > Map< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch4/ch4-04.html" > < strong aria-hidden = "true" > 4.4.< / strong > 结构体< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch4/ch4-05.html" > < strong aria-hidden = "true" > 4.5.< / strong > JSON< / a > < / li > < li class = "chapter-item expanded " > < a href = "../ch4/ch4-06.html" > < strong aria-hidd
< div id = "sidebar-resize-handle" class = "sidebar-resize-handle" > < / div >
< / nav >
< div id = "page-wrapper" class = "page-wrapper" >
< div class = "page" >
< div id = "menu-bar-hover-placeholder" > < / div >
< div id = "menu-bar" class = "menu-bar sticky bordered" >
< div class = "left-buttons" >
< button id = "sidebar-toggle" class = "icon-button" type = "button" title = "Toggle Table of Contents" aria-label = "Toggle Table of Contents" aria-controls = "sidebar" >
< i class = "fa fa-bars" > < / i >
< / button >
< button id = "theme-toggle" class = "icon-button" type = "button" title = "Change theme" aria-label = "Change theme" aria-haspopup = "true" aria-expanded = "false" aria-controls = "theme-list" >
< i class = "fa fa-paint-brush" > < / i >
< / button >
< ul id = "theme-list" class = "theme-popup" aria-label = "Themes" role = "menu" >
< li role = "none" > < button role = "menuitem" class = "theme" id = "light" > Light (default)< / button > < / li >
< li role = "none" > < button role = "menuitem" class = "theme" id = "rust" > Rust< / button > < / li >
< li role = "none" > < button role = "menuitem" class = "theme" id = "coal" > Coal< / button > < / li >
< li role = "none" > < button role = "menuitem" class = "theme" id = "navy" > Navy< / button > < / li >
< li role = "none" > < button role = "menuitem" class = "theme" id = "ayu" > Ayu< / button > < / li >
< / ul >
< button id = "search-toggle" class = "icon-button" type = "button" title = "Search. (Shortkey: s)" aria-label = "Toggle Searchbar" aria-expanded = "false" aria-keyshortcuts = "S" aria-controls = "searchbar" >
< i class = "fa fa-search" > < / i >
< / button >
< / div >
< h1 class = "menu-title" > Go语言圣经< / h1 >
< div class = "right-buttons" >
< a href = "../print.html" title = "Print this book" aria-label = "Print this book" >
< i id = "print-button" class = "fa fa-print" > < / i >
< / a >
< a href = "https://github.com/gopl-zh/gopl-zh.github.com" title = "Git repository" aria-label = "Git repository" >
< i id = "git-repository-button" class = "fa fa-github" > < / i >
< / a >
< a href = "https://github.com/gopl-zh/gopl-zh.github.com/edit/master/./ch8/ch8-06.md" title = "Suggest an edit" aria-label = "Suggest an edit" >
< i id = "git-edit-button" class = "fa fa-edit" > < / i >
< / a >
< / div >
< / div >
< div id = "search-wrapper" class = "hidden" >
< form id = "searchbar-outer" class = "searchbar-outer" >
< input type = "search" id = "searchbar" name = "searchbar" placeholder = "Search this book ..." aria-controls = "searchresults-outer" aria-describedby = "searchresults-header" >
< / form >
< div id = "searchresults-outer" class = "searchresults-outer hidden" >
< div id = "searchresults-header" class = "searchresults-header" > < / div >
< ul id = "searchresults" >
< / ul >
< / div >
< / div >
<!-- Apply ARIA attributes after the sidebar and the sidebar toggle button are added to the DOM -->
< script type = "text/javascript" >
document.getElementById('sidebar-toggle').setAttribute('aria-expanded', sidebar === 'visible');
document.getElementById('sidebar').setAttribute('aria-hidden', sidebar !== 'visible');
Array.from(document.querySelectorAll('#sidebar a')).forEach(function(link) {
link.setAttribute('tabIndex', sidebar === 'visible' ? 0 : -1);
});
< / script >
< div id = "content" class = "content" >
<!-- Page table of contents -->
< div class = "sidetoc" > < nav class = "pagetoc" > < / nav > < / div >
< main >
<!-- 头部 -->
< ul dir = "auto" >
2022-08-24 10:48:15 +00:00
< li > < em > KusonStack一站式可编程配置技术栈(Go): < a href = "https://github.com/KusionStack/kusion" > https://github.com/KusionStack/kusion< / a > < / em > < / li >
< li > < em > KCL 配置编程语言(Rust): < a href = "https://github.com/KusionStack/KCLVM" > https://github.com/KusionStack/KCLVM< / a > < / em > < / li >
< li > < em > 凹语言™: < a href = "https://github.com/wa-lang/wa" > https://github.com/wa-lang/wa< / a > < / em > < / li >
2022-08-04 06:59:33 +00:00
< / ul >
< hr >
< h2 id = "86-示例-并发的web爬虫" > < a class = "header" href = "#86-示例-并发的web爬虫" > 8.6. 示例: 并发的Web爬虫< / a > < / h2 >
< p > 在5.6节中， ， ， ， ， < / p >
< p > < u > < i > gopl.io/ch8/crawl1< / i > < / u > < / p >
< pre > < code class = "language-go" > func crawl(url string) []string {
fmt.Println(url)
list, err := links.Extract(url)
if err != nil {
log.Print(err)
}
return list
}
< / code > < / pre >
< p > 主函数和5.6节中的breadthFirst(广度优先)类似。像之前一样， ， ， < / p >
< pre > < code class = "language-go" > func main() {
worklist := make(chan []string)
// Start with the command-line arguments.
go func() { worklist < - os.Args[1:] }()
// Crawl the web concurrently.
seen := make(map[string]bool)
for list := range worklist {
for _, link := range list {
if !seen[link] {
seen[link] = true
go func(link string) {
worklist < - crawl(link)
}(link)
}
}
}
}
< / code > < / pre >
< p > 注意这里的crawl所在的goroutine会将link作为一个显式的参数传入， （ ） ， ， ， ， < / p >
< p > 现在爬虫可以高并发地运行起来， ， ： < / p >
< pre > < code > $ go build gopl.io/ch8/crawl1
$ ./crawl1 http://gopl.io/
http://gopl.io/
https://golang.org/help/
https://golang.org/doc/
https://golang.org/blog/
...
2015/07/15 18:22:12 Get ...: dial tcp: lookup blog.golang.org: no such host
2015/07/15 18:22:12 Get ...: dial tcp 23.21.222.120:443: socket: too many open files
...
< / code > < / pre >
< p > 最初的错误信息是一个让人莫名的DNS查找失败， ： ， ， < / p >
< p > 这个程序实在是太他妈并行了。无穷无尽地并行化并不是什么好事情， ， ， ， ， ， ， ， ， ， ， ， < / p >
< p > 我们可以用一个有容量限制的buffered channel来控制并发， ， （ ） ， ， （ ， ） ， < / p >
< p > 让我们重写crawl函数， ， < / p >
< p > < u > < i > gopl.io/ch8/crawl2< / i > < / u > < / p >
< pre > < code class = "language-go" > // tokens is a counting semaphore used to
// enforce a limit of 20 concurrent requests.
var tokens = make(chan struct{}, 20)
func crawl(url string) []string {
fmt.Println(url)
tokens < - struct{}{} // acquire a token
list, err := links.Extract(url)
< -tokens // release the token
if err != nil {
log.Print(err)
}
return list
}
< / code > < / pre >
< p > 第二个问题是这个程序永远都不会终止， （ ， ， ） ， < / p >
< pre > < code class = "language-go" > func main() {
worklist := make(chan []string)
var n int // number of pending sends to worklist
// Start with the command-line arguments.
n++
go func() { worklist < - os.Args[1:] }()
// Crawl the web concurrently.
seen := make(map[string]bool)
for ; n > 0; n-- {
list := < -worklist
for _, link := range list {
if !seen[link] {
seen[link] = true
n++
go func(link string) {
worklist < - crawl(link)
}(link)
}
}
}
}
< / code > < / pre >
< p > 这个版本中， ， ， ， ， < / p >
< p > 现在这个并发爬虫会比5.6节中的深度优先搜索版快上20倍， ， < / p >
< p > 下面的程序是避免过度并发的另一种思路。这个版本使用了原来的crawl函数， ， ， < / p >
< pre > < code class = "language-go" > func main() {
worklist := make(chan []string) // lists of URLs, may have duplicates
unseenLinks := make(chan string) // de-duplicated URLs
// Add command-line arguments to worklist.
go func() { worklist < - os.Args[1:] }()
// Create 20 crawler goroutines to fetch each unseen link.
for i := 0; i < 20; i++ {
go func() {
for link := range unseenLinks {
foundLinks := crawl(link)
go func() { worklist < - foundLinks }()
}
}()
}
// The main goroutine de-duplicates worklist items
// and sends the unseen ones to the crawlers.
seen := make(map[string]bool)
for list := range worklist {
for _, link := range list {
if !seen[link] {
seen[link] = true
unseenLinks < - link
}
}
}
}
< / code > < / pre >
< p > 所有的爬虫goroutine现在都是被同一个channel - unseenLinks喂饱的了。主goroutine负责拆分它从worklist里拿到的元素， < / p >
< p > seen这个map被限定在main goroutine中； ， ， ； （ < / p >
< p > crawl函数爬到的链接在一个专有的goroutine中被发送到worklist中来避免死锁。为了节省篇幅， < / p >
< p > < strong > 练习 8.6： < / strong > 为并发爬虫增加深度限制。也就是说， ， < / p >
< p > < strong > 练习 8.7： < / strong > 完成一个并发程序来创建一个线上网站的本地镜像， ， （ ， ： ） ， ， ， < / p >
< p > < strong > 译注：< / strong >
拓展阅读 < a href = "http://marcio.io/2015/07/handling-1-million-requests-per-minute-with-golang/" > Handling 1 Million Requests per Minute with Go< / a > 。< / p >
<!-- 公众号 -->
< hr >
< table >
< tr >
< td >
< img width = "222px" src = "https://chai2010.cn/advanced-go-programming-book/css.png" >
< / td >
< td >
< img width = "222px" src = "https://chai2010.cn/advanced-go-programming-book/cch.png" >
< / td >
< / tr >
< / table >
< div id = "giscus-container" > < / div >
< footer class = "page-footer" >
< span > © 2015-2016 | < a href = "https://github.com/gopl-zh" > Go语言圣经中文版< / a > , 仅学习交流使用< / span >
< / footer >
< / main >
< nav class = "nav-wrapper" aria-label = "Page navigation" >
<!-- Mobile navigation buttons -->
< a rel = "prev" href = "../ch8/ch8-05.html" class = "mobile-nav-chapters previous" title = "Previous chapter" aria-label = "Previous chapter" aria-keyshortcuts = "Left" >
< i class = "fa fa-angle-left" > < / i >
< / a >
< a rel = "next" href = "../ch8/ch8-07.html" class = "mobile-nav-chapters next" title = "Next chapter" aria-label = "Next chapter" aria-keyshortcuts = "Right" >
< i class = "fa fa-angle-right" > < / i >
< / a >
< div style = "clear: both" > < / div >
< / nav >
< / div >
< / div >
< nav class = "nav-wide-wrapper" aria-label = "Page navigation" >
< a rel = "prev" href = "../ch8/ch8-05.html" class = "nav-chapters previous" title = "Previous chapter" aria-label = "Previous chapter" aria-keyshortcuts = "Left" >
< i class = "fa fa-angle-left" > < / i >
< / a >
< a rel = "next" href = "../ch8/ch8-07.html" class = "nav-chapters next" title = "Next chapter" aria-label = "Next chapter" aria-keyshortcuts = "Right" >
< i class = "fa fa-angle-right" > < / i >
< / a >
< / nav >
< / div >
< script type = "text/javascript" >
window.playground_copyable = true;
< / script >
< script src = "../elasticlunr.min.js" type = "text/javascript" charset = "utf-8" > < / script >
< script src = "../mark.min.js" type = "text/javascript" charset = "utf-8" > < / script >
< script src = "../searcher.js" type = "text/javascript" charset = "utf-8" > < / script >
< script src = "../clipboard.min.js" type = "text/javascript" charset = "utf-8" > < / script >
< script src = "../highlight.js" type = "text/javascript" charset = "utf-8" > < / script >
< script src = "../book.js" type = "text/javascript" charset = "utf-8" > < / script >
< script type = "text/javascript" charset = "utf-8" >
var pagePath = "ch8/ch8-06.md"
< / script >
<!-- Custom JS scripts -->
< script type = "text/javascript" src = "../js/custom.js" > < / script >
< script type = "text/javascript" src = "../js/bigPicture.js" > < / script >
< / body >
< / html >
