爬取网址：http://www.ultramanclub.com/allultraman/
使用工具：pycharm，requests
进入网页
打开开发者工具
点击 Network
刷服务器之家页，获取信息
其中的Request URL就是我们所爬取的网址
滑到最下有一个User-Agent，复制
向服务器发送请求
200意味着请求成功
使用 response.text 获取文本数据
可以看到有些乱码
使用encode转换
import requests
url = "http://www.ultramanclub.com/allultraman/"
headers = {
"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36"
}
response = requests.get(url = url,headers=headers)
html = response.text
Html=html.encode("iso-8859-1").decode("gbk")
print(Html)
接下来开始爬取需要的数据
使用Xpath获得网页链接
要使用Xpath必须先导入parsel包
import requests
import parsel
def get_response(html_url):
headers = {
"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36"
}
response = requests.get(url = html_url,headers=headers)
return response
url = "http://www.ultramanclub.com/allultraman/"
response = get_response(url)
html=response.text.encode("iso-8859-1").decode("gbk")
selector = parsel.Selector(html)
period_hrefs = selector.xpath("//div[@class="btn"]/a/@href") #获取三个时代的网页链接
for period_href in period_hrefs:
print(period_href.get())
可以看到网页链接不完整，我们手动给它添加上去period_href = 'http://www.ultramanclub.com/allultraman/' + period_href.get()
进入其中一个网页
跟之前的操作一样，用Xpath获取奥特曼的网页信息
for period_href in period_hrefs:
period_href = "http://www.ultramanclub.com/allultraman/" + period_href.get()
# print(period_href)
period_response = get_response(period_href).text
period_html = parsel.Selector(period_response)
lis = period_html.xpath("//div[@class="ultraheros-Contents_Generations"]/div/ul/li/a/@href")
for li in lis:
print(li.get())
运行后同样发现链接不完整
li = 'http://www.ultramanclub.com/allultraman/' + li.get().replace('./','')
拿到网址后继续套娃操作,就可以拿到图片数据
png_url = 'http://www.ultramanclub.com/allultraman/' + li_selector.xpath('//div[@class="left"]/figure/img/@src').get().replace('../','')
完整代码
import requests
import parsel
import os
dirname = "奥特曼"
if not os.path.exists(dirname): #判断是否存在名称为奥特曼的文件夹，没有就创建
os.mkdir(dirname)
def get_response(html_url):
headers = {
"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36"
}
response = requests.get(url = html_url,headers=headers)
return response
url = "http://www.ultramanclub.com/allultraman/"
response = get_response(url)
html=response.text.encode("iso-8859-1").decode("gbk")
selector = parsel.Selector(html)
period_hrefs = selector.xpath("//div[@class="btn"]/a/@href") #获取三个时代的网页链接
for period_href in period_hrefs:
period_href = "http://www.ultramanclub.com/allultraman/" + period_href.get()
period_html = get_response(period_href).text
period_selector = parsel.Selector(period_html)
lis = period_selector.xpath("//div[@class="ultraheros-Contents_Generations"]/div/ul/li/a/@href")
for li in lis:
li = "http://www.ultramanclub.com/allultraman/" + li.get().replace("./","") #获取每个奥特曼的网址
# print(li)
li_html = get_response(li).text
li_selector = parsel.Selector(li_html)
url = li_selector.xpath("//div[@class="left"]/figure/img/@src").get()
# print(url)
if url:
png_url = "http://www.ultramanclub.com/allultraman/" + url.replace(".", "")
png_title =li_selector.xpath("//ul[@class="lists"]/li[3]/text()").get()
png_title = png_title.encode("iso-8859-1").decode("gbk")
# print(li,png_title)
png_content = get_response(png_url).content
with open(f"{dirname}{png_title}.png","wb") as f:
f.write(png_content)
print(png_title,"图片下载完成")
else:
continue
当爬到 奈克斯特奥特曼的时候，就会返回None，调了半天，也没搞懂，所以用if url：语句跳过了奈克斯特奥特曼，有没有大佬知道原因
url = li_selector.xpath("//div[@class="left"]/figure/img/@src").get()
到此这篇关于用python实现爬取奥特曼图片实例的文章就介绍到这了,更多相关python爬取奥特曼图片内容请搜索服务器之家以前的文章或继续浏览下面的相关文章希望大家以后多多支持服务器之家！
原文链接：https://blog.csdn.net/K4239527/article/details/122842708
