Python神器之使用watchdog监控文件变化
一、官方文档
需要细节选择去官网,需要了解和应用范例看本文即可
https://pythonhosted.org/watchdog/官网上的:User’s Guide,主要看API相关这里即可
- 安装
- 快速开始
- API 相关
二、watchdog安装
watchdog需要在python2.6以上的版本工作,如果使用的是Linux//FreeBSD/Mac OS X 系统,python已经安装了,但是需要保证python的版本至少是python2.7(学习用最好还是使用3.6以上的版本吧)
下面有三种不同的安装方法:选择第一种最简单的就行了,可以简便的不要自求麻烦。
1. Installing from PyPI using pip
pip install watchdog
2. Installing from source tarballs
$ wget -c http://pypi.python.org/packages/source/w/watchdog/watchdog-0.8.2.tar.gz $ tar zxvf watchdog-0.8.2.tar.gz $ cd watchdog-0.8.2 $ python setup.py install
3. Installing from the code repository
$ git clone --recursive git://github.com/gorakhargosh/watchdog.git $ cd watchdog $ python setup.py install
在官方还说需要下载一些依赖,但是你使用第一种pip安装的方法基本都不会有这种需求。
三、快速开始(官方范例)
下面我们提供一个简单的示例,该示例以递归方式监视当前目录(这意味着它将遍历所有子目录)以检测更改。 这是我们将要使用的API:
1.创建watchdog.observers.Observer线程类的实例
2.实现watchdog.events.FileSystemEventHandler的子类(或者在我们的例子中,我们将使用内置的watchdog.events.LoggingEventHandler)。
3.通过附加事件处理程序的观察者实例来计划监视一些路径。
4.启动观察者线程并等待它生成事件而不会阻塞我们的主线程。
import sys import time import logging from watchdog.observers import Observer # 监控 from watchdog.events import LoggingEventHandler # 触发事件 if __name__=='__main__': logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(message)s', datefmt='%Y-%m-%d %H:%M:%S') path = sys.argv[1] if len(sys.argv) > 1 else '.' event_handler = LoggingEventHandler() # 监控处理事件的方法,这个类非常重要,可以根据自己的需要重写 observer = Observer() # 定义监控类,多线程类 thread class observer.schedule(event_handler,'./monitor_folder_1', recursive=True) # 指定监控路径/触发对应的监控事件类 observer.start()# 将observer运行在同一个线程之内,不阻塞主进程运行,可以调度observer来停止该线程 try: while True: time.sleep(1) # 监控频率(1s1次,根据自己的需求进行监控) except KeyboardInterrupt: observer.stop() observer.join()
四、API Reference(翻译官网)
观察者模型主要有三个角色:observer、event_handler、被监控的文件夹。三者原本是独立的,主要通过observer.schedule函数将三者串起来,意思为observer不断检测调用平台依赖代码对监控文件夹进行变动检测,当发现改变时,通知event_handler处理。
watchdog.event
模型:watchdog.events简介:文件系统事件和时间处理的程序原作者: yesudeep@google.com (Yesudeep Mangalapilly)
1. Event Class
首先考虑一下关于文件和文件夹的相关操作有那些?无外乎:
- 创建(FileCreatedEvent/DirCreateEvent)
- 移动(FileMovedEvent/DirMovedEvent)
- 删除(FildDeletedEvent/DirDeletedEvent)
- 修改(FileModifiedEvent/DirModifiedEvent)
对文件/文件夹的监控也就无非以上的操作了,watchdog对文件夹的监控也就是针对上述需求进行设计的Event Classes也就包含了大概八个类(都分别依赖于watchdog.events.FileSystemEvent 和watchdog.events.FileSystemMovedEvent):
- class watchdog.events.FileMovedEvent(src_path, dest_path)
- class watchdog.events.DirMovedEvent(src_path, dest_path)
- class watchdog.events.FileModifiedEvent(src_path)
- class watchdog.events.DirModifiedEvent(src_path)
- class watchdog.events.FileCreatedEvent(src_path)
- class watchdog.events.DirCreatedEvent(src_path)
- class watchdog.events.FileDeletedEvent(src_path)
- class watchdog.events.DirDeletedEvent(src_path)
2. Event handler Classes事件处理类
官方文件中对Event Handler Classes的说明:
class watchdog.events.FileSystemEventHandler[source]
Bases: object
Base file system event handler that you can override methods from.
dispatch(event)[source]
Dispatches events to the appropriate methods.
Parameters:event (FileSystemEvent) – The event object representing the file system event.
on_any_event(event)[source]
Catch-all event handler.
Parameters:event (FileSystemEvent) – The event object representing the file system event.
on_created(event)[source]
Called when a file or directory is created.
Parameters:event (DirCreatedEvent or FileCreatedEvent) – Event representing file/directory creation.
on_deleted(event)[source]
Called when a file or directory is deleted.
Parameters:event (DirDeletedEvent or FileDeletedEvent) – Event representing file/directory deletion.
on_modified(event)[source]
Called when a file or directory is modified.
Parameters:event (DirModifiedEvent or FileModifiedEvent) – Event representing file/directory modification.
on_moved(event)[source]
Called when a file or a directory is moved or renamed.
Parameters:event (DirMovedEvent or FileMovedEvent) – Event representing file/directory movement.
上述也就说明了watchdog的触发事件的方法为:
- dispatch
- on_any_event
- on_create
- on_delete
- on_modified
- on_moved
3. watchdog.observers.api
class watchdog.observers.api.ObservedWatch(path, recursive)
Bases: object
An scheduled watch.
Parameters:
path – Path string.
recursive:
True if watch is recursive;
False
otherwise.is_recursive:Determines whether subdirectories are watched for the path.
path:The path that this watch monitors
Observer用以监控文件的变化,根据触发事件调用相关联的事件处理方法来执行。
observer是threading.Thread的子类,通过observer.start()使之运行在一个线程之中,不阻塞主线程的运行,然后可以调用observer.stop()来停止该进程.
observer.schedule(event_handler,path,recursive=False) # event_handler:触发事件的操作方法 # path:监控的文件或者目录 # recursive:如果监控的是文件则需要设置为True
五、linux中使用watchdog监控文件夹各种操作会触发的监控事件
1. 移除文件
文件删除:File_deleted
文件夹修改 Directory_modified
2. 添加文件
文件新建:File_created
文件夹修改:Directory_modified
3.修改文件
两次文件修改:File_modified(显示针对一个备份文件,然后是针对原始文件)
删除备份文件:File_deleted
文件夹修改:Directory_modified
4.文件改名
文件移动:File_moved
文件夹修改:Directory_modified
5.使用cp命令复制文件到监控目录
文件新建:File_created
一大堆文件修改操作:Directory_modified
文件夹修改操作:File_modified
六、应用实例
1.监控文件夹中是否有后缀为".segy"的文件进入
#! /home/liufeng/project # -*- coding:utf-8 -*- # Created by jerry_liufeng # 本模块的功能:<检测文件夹变化> # 导入watchdog对应模块 from watchdog.observers import Observer from watchdog.events import * # 导入时间模块 import time class FileEventHandler(FileSystemEventHandler): # 初始化魔术方法 def __init__(self): FileSystemEventHandler.__init__(self) self.fileNameList = [] # 文件名列表(存储文件名) # 文件或文件夹移动 def on_moved(self, event): if event.is_directory: print("directory moved from {0} to {1}".format(event.src_path,event.dest_path)) else: print("file moved from {0} to {1}".format(event.src_path,event.dest_path)) # 创建文件或文件夹 def on_created(self, event): if event.is_directory: print("directory created:{0}".format(event.src_path)) else: print("file created:{0}".format(event.src_path)) fileAllName =str(event.src_path.split('/')[-1]) if fileAllName.endswith('.segy'): self.fileNameList.append(fileAllName) print(self.fileNameList) # 删除文件或文件夹 def on_deleted(self, event): if event.is_directory: print("directory deleted:{0}".format(event.src_path)) else: print("file deleted:{0}".format(event.src_path)) # # 修改文件或文件夹 # def on_modified(self, event): # if event.is_directory: # print("directory modified:{0}".format(event.src_path)) # else: # print("file modified:{0}".format(event.src_path)) if __name__ == "__main__": # 实例化Observer对象 observer = Observer() event_handler = FileEventHandler() # 设置监听目录 dis_dir = "./03_seis_data/" observer.schedule(event_handler,dis_dir,True) observer.start() try: while True: # 设置监听频率(间隔周期时间) time.sleep(1) except KeyboardInterrupt: observer.stop() observer.join()
说明:之所以注释掉modified这段代码是因为我关心的是文件的进入,而非文件内容的改变,太过频繁的modified操作会影响我对新增文件数量的判断。
2.监控多个文件夹
from watchdog.observers import Observer from watchdog.events import FileSystemEventHandler import time class FileEventHandler(FileSystemEventHandler): def __init__(self): FileSystemEventHandler.__init__(self) def on_moved(self, event): if event.is_directory: print("directory moved from {0} to {1}".format(event.src_path,event.dest_path)) else: print("file moved from {0} to {1}".format(event.src_path,event.dest_path)) def on_created(self, event): if event.is_directory: print("directory created:{0}".format(event.src_path)) else: print("file created:{0}".format(event.src_path)) def on_deleted(self, event): if event.is_directory: print("directory deleted:{0}".format(event.src_path)) else: print("file deleted:{0}".format(event.src_path)) def on_modified(self, event): if event.is_directory: print("directory modified:{0}".format(event.src_path)) else: print("file modified:{0}".format(event.src_path)) if __name__ == "__main__": observer = Observer() dirs = [r'./monitor_folder_1', r'./monitor_folder_1'] # dirs = [ r'./monitor_folder_1'] for dir in dirs: event_handler = FileEventHandler() observer.schedule(event_handler, dir, True) observer.start()
3.文件系统一次操作多次触发各类事件问题解决
问题:
文件操作引发的事件比我们想象的多了不少,而且难以在事件函数中做出针对性处理。
在 windows 平台上每一次的文件修改引发两次 modified 事件?在 windows 平台上,由于 watchdog 封装的是 windows 系统的 FileSystemWatcher Events,处理文件的过程中执行了多次文件系统操作,无法避免地触发了多次事件。
改进方案:
如果对监视文件的实时性要求不高,又懒得处理一大堆事件,那么,比较事件前后的文件夹快照就是一个值得尝试的改进方案。实现这个思路,有三个前提条件:
- 快速获取文件夹快照。幸运的是,watchdog 模块为我们提供了 DirectorySnapshot 功能
- 可以接受200毫秒的延时。文件操作引发的一大堆事件会集中在一个较短的时间内,一般情况下,在文件操作之后200毫秒获取文件夹快照,是一个不错的间隔
- 快速比较文件夹快照。这也不是问题,watchdog 模块有 DirectorySnapshotDiff 子模块
改进思路:设置一个定时器, 200毫秒后抓取快照,并与上一张快照比较。每当有事件发生时,检查定时器是否已经启动。如果未启动,则直接启动定时器;否则,说明该事件距离上次事件不足200毫秒,可视为是同一组事件,此时终止定时器,再次重启。具体代码如下:
import time import os, threading from watchdog.observers import Observer from watchdog.events import * from watchdog.utils.dirsnapshot import DirectorySnapshot, DirectorySnapshotDiff class FileEventHandler(FileSystemEventHandler): def __init__(self, aim_path): FileSystemEventHandler.__init__(self) self.aim_path = aim_path self.timer = None self.snapshot = DirectorySnapshot(self.aim_path) def on_any_event(self, event): if self.timer: self.timer.cancel() self.timer = threading.Timer(0.2, self.checkSnapshot) self.timer.start() def checkSnapshot(self): snapshot = DirectorySnapshot(self.aim_path) diff = DirectorySnapshotDiff(self.snapshot, snapshot) self.snapshot = snapshot self.timer = None print("files_created:", diff.files_created) print("files_deleted:", diff.files_deleted) print("files_modified:", diff.files_modified) print("files_moved:", diff.files_moved) print("dirs_modified:", diff.dirs_modified) print("dirs_moved:", diff.dirs_moved) print("dirs_deleted:", diff.dirs_deleted) print("dirs_created:", diff.dirs_created) # def on_moved(self, event): # if event.is_directory: # print("directory moved from {0} to {1}".format(event.src_path,event.dest_path)) # else: # print("file moved from {0} to {1}".format(event.src_path,event.dest_path)) # def on_created(self, event): # if event.is_directory: # print("directory created:{0}".format(event.src_path)) # else: # print("file created:{0}".format(event.src_path)) # def on_deleted(self, event): # if event.is_directory: # print("directory deleted:{0}".format(event.src_path)) # else: # print("file deleted:{0}".format(event.src_path)) # def on_modified(self, event): # if event.is_directory: # print("directory modified:{0}".format(event.src_path)) # else: # print("file modified:{0}".format(event.src_path)) class DirMonitor(object): """文件夹监视类""" def __init__(self, aim_path): """构造函数""" self.aim_path= aim_path self.observer = Observer() def start(self): """启动""" event_handler = FileEventHandler(self.aim_path) self.observer.schedule(event_handler, self.aim_path, True) self.observer.start() def stop(self): """停止""" self.observer.stop() if __name__ == "__main__": monitor = DirMonitor(r"./monitor_folder_1") monitor.start() try: while True: time.sleep(1) except KeyboardInterrupt: monitor.stop()
以上就是Python神器之使用watchdog监控文件变化的详细内容,更多关于Python watchdog监控文件变化的资料请关注脚本之家其它相关文章!
相关文章
Python中的np.vstack()和np.hstack()详解
这篇文章主要介绍了np.vstack()和np.hstack(),本文给大家介绍的非常详细,对大家的学习或工作具有一定的参考借鉴价值,需要的朋友可以参考下2023-04-04
最新评论