使用Python爬虫库BeautifulSoup遍历文档树并对标签进行操作详解

更新时间：2020年01月25日 16:19:08 作者：BQW_

今天为大家介绍下Python爬虫库BeautifulSoup遍历文档树并对标签进行操作的详细方法与函数

下面就是使用Python爬虫库BeautifulSoup对文档树进行遍历并对标签进行操作的实例，都是最基础的内容

html_doc = """
<html><head><title>The Dormouse's story</title></head>

<p class="title"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>

<p class="story">...</p>
"""

from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc,'lxml')

一、子节点

一个Tag可能包含多个字符串或者其他Tag，这些都是这个Tag的子节点.BeautifulSoup提供了许多操作和遍历子结点的属性。

1.通过Tag的名字来获得Tag

print(soup.head)
print(soup.title)

<head><title>The Dormouse's story</title></head>
<title>The Dormouse's story</title>

通过名字的方法只能获得第一个Tag，如果要获得所有的某种Tag可以使用find_all方法

soup.find_all('a')

[<a class="sister" href="http://example.com/elsie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">Elsie</a>,
 <a class="sister" href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>,
 <a class="sister" href="http://example.com/tillie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>]

2.contents属性：将Tag的子节点通过列表的方式返回

head_tag = soup.head
head_tag.contents

[<title>The Dormouse's story</title>]

title_tag = head_tag.contents[0]
title_tag

<title>The Dormouse's story</title>

title_tag.contents

["The Dormouse's story"]

3.children：通过该属性对子节点进行循环

for child in title_tag.children:
  print(child)

The Dormouse's story

4.descendants：不论是contents还是children都是返回直接子节点，而descendants对所有tag的子孙节点进行递归循环

for child in head_tag.children:
  print(child)

<title>The Dormouse's story</title>

for child in head_tag.descendants:
  print(child)

<title>The Dormouse's story</title>
The Dormouse's story

5.string 如果tag只有一个NavigableString类型的子节点，那么tag可以使用.string得到该子节点

title_tag.string

"The Dormouse's story"

如果一个tag只有一个子节点，那么使用.string可以获得其唯一子结点的NavigableString.

head_tag.string

"The Dormouse's story"

如果tag有多个子节点，tag无法确定.string对应的是那个子结点的内容，故返回None

print(soup.html.string)

None

6.strings和stripped_strings

如果tag包含多个字符串，可以使用.strings循环获取

for string in soup.strings:
  print(string)

The Dormouse's story


The Dormouse's story


Once upon a time there were three little sisters; and their names were

Elsie
,

Lacie
 and

Tillie
;
and they lived at the bottom of a well.


...

.string输出的内容包含了许多空格和空行，使用strpped_strings去除这些空白内容

for string in soup.stripped_strings:
  print(string)

The Dormouse's story
The Dormouse's story
Once upon a time there were three little sisters; and their names were
Elsie
,
Lacie
and
Tillie
;
and they lived at the bottom of a well.
...

二、父节点

1.parent：获得某个元素的父节点

title_tag = soup.title
title_tag.parent

<head><title>The Dormouse's story</title></head>

字符串也有父节点

title_tag.string.parent

<title>The Dormouse's story</title>

2.parents：递归的获得所有父辈节点

link = soup.a
for parent in link.parents:
  if parent is None:
    print(parent)
  else:
    print(parent.name)

p
body
html
[document]

三、兄弟结点

sibling_soup = BeautifulSoup("<a><b>text1</b><c>text2</c></b></a>",'lxml')
print(sibling_soup.prettify())

<html>
 <body>
 <a>
  <b>
  text1
  </b>
  <c>
  text2
  </c>
 </a>
 </body>
</html>

1.next_sibling和previous_sibling

sibling_soup.b.next_sibling

<c>text2</c>

sibling_soup.c.previous_sibling

<b>text1</b>

在实际文档中.next_sibling和previous_sibling通常是字符串或者空白符

soup.find_all('a')

[<a class="sister" href="http://example.com/elsie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">Elsie</a>,
 <a class="sister" href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>,
 <a class="sister" href="http://example.com/tillie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>]

soup.a.next_sibling # 第一个<a></a>的next_sibling是,\n

',\n'

soup.a.next_sibling.next_sibling

<a class="sister" href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>

2.next_siblings和previous_siblings

for sibling in soup.a.next_siblings:
  print(repr(sibling))

',\n'
<a class="sister" href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>
' and\n'
<a class="sister" href="http://example.com/tillie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>
';\nand they lived at the bottom of a well.'

for sibling in soup.find(id="link3").previous_siblings:
  print(repr(sibling))

' and\n'
<a class="sister" href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>
',\n'
<a class="sister" href="http://example.com/elsie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">Elsie</a>
'Once upon a time there were three little sisters; and their names were\n'

四、回退与前进

1.next_element和previous_element

指向下一个或者前一个被解析的对象(字符串或tag)，即深度优先遍历的后序节点和前序节点

last_a_tag = soup.find("a", id="link3")
print(last_a_tag.next_sibling)
print(last_a_tag.next_element)

;
and they lived at the bottom of a well.
Tillie

last_a_tag.previous_element

' and\n'

2.next_elements和previous_elements

通过.next_elements和previous_elements可以向前或向后访问文档的解析内容，就好像文档正在被解析一样

for element in last_a_tag.next_elements:
  print(repr(element))

'Tillie'
';\nand they lived at the bottom of a well.'
'\n'
<p class="story">...</p>
'...'
'\n'

更多关于使用Python爬虫库BeautifulSoup遍历文档树并对标签进行操作的方法与文章大家可以点击下面的相关文章

您可能感兴趣的文章:

Python+OpenCV实战之拖拽虚拟方块的实现
这篇文章主要介绍了如何利用Python+OpenCV实现拖拽虚拟方块的效果，即根据手指坐标位置和矩形的坐标位置，判断手指点是否在矩形上，如果在则矩形跟随手指移动，感兴趣的可以了解一下
2022-08-08
python缩进区别分析
这篇文章主要介绍了python缩进区别分析,需要的朋友可以参考下
2014-02-02
Python实现Word和TXT文件格式之间的相互转换
Word文档（.doc或.docx）和纯文本文件（.txt）是两种常用的文件格式,本文将详细介绍如何使用Python实现Word和TXT文件格式之间的相互转换,文中有详细的代码示例供大家参考,需要的朋友可以参考下
2024-07-07
对python中词典的values值的修改或新增KEY详解
今天小编就为大家分享一篇对python中词典的values值的修改或新增KEY详解，具有很好的参考价值，希望对大家有所帮助。一起跟随小编过来看看吧
2019-01-01
python实现比较类的两个instance(对象)是否相等的方法分析
这篇文章主要介绍了python实现比较类的两个instance(对象)是否相等的方法,结合实例形式分析了Python判断类的实例是否相等的判断操作实现技巧,需要的朋友可以参考下
2019-06-06
Python实现图算法、堆操作和并查集代码实例
这篇文章主要介绍了Python实现图算法、堆操作和并查集代码实例,图算法、堆操作和并查集是计算机科学中常用的数据结构和算法，它们在解决各种实际问题中具有重要的应用价值,需要的朋友可以参考下
2023-08-08
Python中用户输入与while循环详情
这篇文章主要介绍了Python中用户输入与while循环详情，，包括如何接收用户输入并进行处理，在程序满足一定的条件时让程序一直运行，通过获取用户输入并学会控制程序在用户想要结束时退出循环，即可编写出交互式程序，下文详细内容介绍,需要的朋友可以参考一下
2022-03-03
Python运行报错UnicodeDecodeError的解决方法
本文给大家分享的是在Python项目中经常遇到的关于编码问题的一个小bug的解决方法以及分析方法，有相同遭遇的小伙伴可以来参考下
2016-06-06
python DataFrame转dict字典过程详解
这篇文章主要介绍了python DataFrame转dict字典过程详解,文中通过示例代码介绍的非常详细，对大家的学习或者工作具有一定的参考学习价值,需要的朋友可以参考下
2019-12-12
Python基于smtplib实现异步发送邮件服务
这篇文章主要介绍了Python基于smtplib实现异步发送邮件服务,需要的朋友可以参考下
2015-05-05

使用Python爬虫库BeautifulSoup遍历文档树并对标签进行操作详解

相关文章

最新评论

大家感兴趣的内容

最近更新的内容

常用在线小工具