在学习铁锈的同时,我正在尝试建立一个简单的网络刮刀。我的目标是刮https://news.ycombinator.com/,得到标题,超链接,选票和用户名。为此,我使用了外部库reqwest和scraper,并编写了一个程序,用于从该站点抓取HTML链接。
Cargo.toml
[package]
name = "stackoverflow_scraper"
version = "0.1.0"
edition = "2018"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
scraper = "0.12.0"
reqwest = "0.11.2"
tokio = { version = "1", features = ["full"] }
futures = "0.3.13"src/main.rs
use scraper::{Html, Selector};
use reqwest;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let url = "https://news.ycombinator.com/";
let html = reqwest::get(url).await?.text().await?;
let fragment = Html::parse_fragment(html.as_str());
let selector = Selector::parse("a.storylink").unwrap();
for element in fragment.select(&selector) {
println!("{:?}",element.value().attr("href").unwrap());
// todo println!("Title");
// todo println!("Votes");
// todo println!("User");
}
Ok(())
}我如何获得相应的标题,选票和用户名?
发布于 2021-03-19 17:38:35
首页上的项目存储在带有类table的.itemlist中。
由于每一项都是由三个连续的<tr>组成的,所以您必须以三个块的形式迭代它们。我选择了首先收集所有的节点。
第一行包含:
第二行包含:
第三行是一个应该忽略的间隔。
注:
tr.morespace和包含a.morelink的tr应该被忽略。这就是为什么我选择首先对节点进行.collect(),然后使用.chunks_exact()。use reqwest;
use scraper::{Html, Selector};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let url = "https://news.ycombinator.com/";
let html = reqwest::get(url).await?.text().await?;
let fragment = Html::parse_fragment(html.as_str());
let selector_items = Selector::parse(".itemlist tr").unwrap();
let selector_title = Selector::parse("a.storylink").unwrap();
let selector_score = Selector::parse("span.score").unwrap();
let selector_user = Selector::parse("a.hnuser").unwrap();
let nodes = fragment.select(&selector_items).collect::<Vec<_>>();
let list = nodes
.chunks_exact(3)
.map(|rows| {
let title_elem = rows[0].select(&selector_title).next().unwrap();
let title_text = title_elem.text().nth(0).unwrap();
let title_href = title_elem.value().attr("href").unwrap();
let score_text = rows[1]
.select(&selector_score)
.next()
.and_then(|n| n.text().nth(0))
.unwrap_or("0 points");
let user_text = rows[1]
.select(&selector_user)
.next()
.and_then(|n| n.text().nth(0))
.unwrap_or("Unknown user");
[title_text, title_href, score_text, user_text]
})
.collect::<Vec<_>>();
println!("links: {:#?}", list);
Ok(())
}这将为您提供以下列表:
[
[
"Docker for Mac M1 RC",
"https://docs.docker.com/docker-for-mac/apple-m1/",
"327 points",
"mikkelam",
],
[
"A Mind Is Born – A 256 byte demo for the Commodore 64 (2017)",
"https://linusakesson.net/scene/a-mind-is-born/",
"226 points",
"matthewsinclair",
],
[
"Show HN: Video Game in a Font",
"https://www.coderelay.io/fontemon.html",
"416 points",
"ghub-mmulet",
],
...
]另一种方法是,有一个可以使用的API:
发布于 2021-03-19 17:36:09
这是一个更多的选择器问题,这取决于被刮掉的站点的html。在这种情况下,很容易得到标题,但更难得到的分数和用户。由于使用的选择器选择包含href和title的链接,所以可以使用.text()方法获得标题
let title = element.text().collect::<Vec<_>>();元素与href元素相同。
然而,要获得其他值,更改第一个选择器并从中获取数据就更容易了。由于news.ycombinator.com上新闻条目的标题和链接位于带有.athing类的元素中,而选票和用户位于下一个元素中,而下一个元素没有类(这使选择变得更加困难),所以最好选择"table.itemlist tr.athing"并迭代这些结果。从找到的每个元素中,您可以对"a.storylink"元素进行子选择,然后分别获得以下tr元素和子选择points和user元素
let select_item = Selector::parse("table.itemlist tr.athing").unwrap();
let select_link = Selector::parse("a.storylink").unwrap();
let select_score = Selector::parse("span.score").unwrap();
for element in fragment.select(&select_item) {
// Get the link element that contains the href and title
let link_el = element.select(&select_link).next().unwrap();
println!("{:?}", link_el.value().attr("href").unwrap());
// Get the next tr element that follows the first, with score and user
let details_el = ElementRef::wrap(element.next_sibling().unwrap()).unwrap();
// Get the score element from within the second row element
let score = details_el.select(&select_score).next().unwrap();
println!("{:?}", score.text().collect::<Vec<_>>());
}这只显示得到href和得分。我会让你从details_el上得到用户
https://stackoverflow.com/questions/66711259
复制相似问题