v0.3.5

anomalyco/opencodev0.3.5Jul 5, 2023by NaiboWang

AI Summary

Focuses on performance improvements and comprehensive variable support across the application. It adds task control features like pausing and resuming, along with expanded data export options to MySQL.

Key Highlights

  • Comprehensive `Field[]` variable support in all code inputs and links.
  • Task pause/resume functionality via keyboard shortcut.
  • Data export to MySQL database.
  • Greatly improved collection speed.
  • Cloudflare bypass support for Windows x64.

New Features

  • `Field[]` variable substitution in all code inputs.
  • Task pause/resume functionality.
  • Data export to MySQL.
  • XPath Helper integration for debugging.
  • Mobile browser simulation.
  • Cloudflare bypass support.
  • Random wait time (50%-150% of set time).
  • Source code included in package.
  • Cookie modification support.
  • New click simulation method.
  • General parameter settings (local write frequency, preview length).
  • Relative path saving support (`../../`).

Full Release Notes

如果下载速度慢,可以考虑中国境内下载地址:[中国境内下载地址](https://easyspider.cn/download.html)。

有关Windows x64位版本部分情况下无法采集链接地址的说明:https://github.com/NaiboWang/EasySpider/issues/128

Explanation about the issue where the link address cannot be collected in some cases on Windows x64 version: https://github.com/NaiboWang/EasySpider/issues/128

Windows x64版本支持64位的**Windows 10及以上系统**,Windows x86版本支持所有位数(32位和64位)的**Windows 7及以上系统**,即64位的Windows 7也要下载此版本。注意x86版本的EasySpider的Chrome浏览器永远都是109,不会随着Chrome版本更新而更新(为了兼容Win 7系统),因此如果想用最新版Chrome浏览器采集数据,请在Windows 10 x64及以上系统上运行x64版本的软件。

The Windows  x64 version supports **Windows 10 and above** with 64-bit, while the x86 version of Windows supports all versions (32-bit and 64-bit) of Windows 7 and above, meaning that the 64-bit version of Windows 7 should also download this version. Note that the Chrome browser in this x86 version of EasySpider is always version 109 and will not update with Chrome updates (to maintain compatibility with the Win 7 system). Therefore, if you want to collect data with the latest version of the Chrome browser, please run the x64 version of the software on Windows 10 x64 and above systems.

MacOS版请用系统自带的`归档使用工具`解压,MacOS版本支持所有芯片组,包括**Intel和M1,M2**等处理器,但操作系统最低版本要求为11.1,更低操作系统版本请下载[v0.2.0版本](https://github.com/NaiboWang/EasySpider/releases/tag/v0.2.0)的[Mac版](https://github.com/NaiboWang/EasySpider/releases/download/v0.2.0/EasySpider_0.2.0_MacOS_all_arch.zip)使用,或自行下载代码并编译,示例编译方式看[这个Issue](https://github.com/NaiboWang/EasySpider/issues/69)。

The MacOS version supports all chipsets, including **Intel, M1, M2**, and other processors. However, the minimum operating system requirement is 11.1. For lower operating system versions, please download the code and compile it yourself. An example compilation method can be found in [this issue](https://github.com/NaiboWang/EasySpider/issues/69). Please unzip the `.tar.gz` file by the `Arxiv Utility` software.

同理,Linux版只适用于**Ubuntu 20.04及以上版本、Deepin、Debian及其衍生版本**,如想使用其他Linux发行版采集数据,请自行下载代码并编译。

Similarly, the Linux version is only compatible with **Ubuntu 20.04 and above, Deepin, Debian, and their derivatives**. If you want to use other Linux distributions for data collection, please download the code and compile it yourself.

## 更新说明

1. 提速:极大的提升了大部分场景的采集速度。
2. **所有**写JavaScript/系统命令代码语句的地方以及打开网页的链接池,都可以用`Field["参数名"]`表示最近提取到的页面参数值/自定义操作返回,即实现了全面的`变量`功能。
3. 循环中可以在任意位置使用`自定义操作`的`退出循环`选项直接退出循环,即添加了`Break`功能。
<img src="https://github.com/NaiboWang/EasySpider/assets/30287768/c84d4675-5ea3-4836-854e-5d277ed174fe" width=30%/>

4. 可以提取在`<iframe>`标签内的数据。
5. 增加**暂停执行任务**功能,可**长按**键盘`p`键暂停和继续执行任务。
6. (Windows x64可用,其余系统请等待下个版本)增加“一直向下滚动直到页面内容无变化”的功能,同时循环点击下一页的操作的退出循环条件改为找不到下一页按钮**及检测不到页面内容变化**。
7. 执行阶段也可以使用`XPath Helper`扩展来调试XPath,配合上面的暂停功能使用。
8. 可导出为`Excel/TXT`文件,可写入`MySQL`数据库,可指定数据类型为`整数/小数/日期`等,[点此查看MySQL写入教程](https://www.bilibili.com/video/BV1os4y1679S)。
9. 调用任务时的输入参数值可以通过读取Excel文件替换。
10. 浏览器操作台可通过左上角拖动改变大小。
<img src="https://github.com/NaiboWang/EasySpider/assets/30287768/37367afd-f766-467a-836d-e8ad9b99a213" width=40% height=40%/>

11. 提取数据的字段可设置为不保存(适用于只想将此字段作为变量输入的情况)。
12. 输入文字操作后可用`<enter>`或`<ENTER>`表示硬回车,即输入完成后在当前文本框按回车。
13. 可以模拟手机端浏览器运行。
14. (只支持Windows x64版本)可处理和采集针对被Cloudflare的验证码保护的变态网站,[点此查看视频教程](https://www.bilibili.com/video/BV1Ph4y1E7R9/)。
15. 新增默认索引位置使用last()从后往前数的XPath提示。
16. 操作后等待时长可设置为设定时间的50%-150%的随机等待。
17. 软件包内自带python源代码以供专业人士修改任务流程和调试。
18. `打开网页`的高级操作支持获取当前页面Cookies,并可修改Cookies。
<img src="https://github.com/NaiboWang/EasySpider/assets/30287768/4d123ded-8279-47fc-b759-e795d2e35f4a" width=40%/>

19. 更改点击元素方式,真正模拟现实世界鼠标点击操作。
20. 通用参数设置:每采集多少条本地写入一次,默认为10;控制栏预览数据长度,默认为15等。
21. 压缩任务文件大小。
22. 保存名称和位置更改,默认文件保存路径是`Data/Task_ID`,想要保存到其他路径,可以用`../../`这种形式进行相对路径引用,比如`../../JS`表示保存的的文件名是`JS`,保存位置为和`Data`文件夹同一级目录的文件夹,即`EasySpider`主文件夹。
23. 流程图和选项配置自动刷新,无需点击`确定`按钮,但仍需手动保存任务。
24. 源代码优化,使二次开发更容易。
25. Bug修复:如执行系统命令如果失败会打印错误信息,修复了MacOS和Linux下系统命令执行失败的Bug;URL格式判断,累计增长的字段名索引值不正确等Bug。
26. 屏蔽无关日志信息,执行界面更清爽。

## Update Instruction

1. **Speed up**: Greatly improved the collection speed in most scenarios.
2. **Variable Functionality**: In all places where you write JavaScript/system command code statements and open web page links, you can use Field["parameter_name"] to represent the recently extracted page parameter value/custom operation return. This provides comprehensive variable functionality.
3. **Loop Control**: During a loop, you can use the `exit loop` option of `custom operation` at any position to directly exit the loop, that is, the `Break` function has been added.
4. **Data Extraction**: Data within `<iframe>` tags can be extracted.
5. **Task Control**: Added **pause execution task** feature, you can **long press** the `p` key on the keyboard to pause and continue execution.
6. (Windows x64 only now, other OS please wait for the next version) Add a "Keep scrolling until the page content does not change" feature, and modify the loop exit condition of repeatedly clicking the next page operation to "unable to find the next page button" and "page content doesn't change".
7. **XPath Debugging**: You can also use `XPath Helper` extension to debug XPath during the execution stage, which can be used in conjunction with the pause feature above.
8. **Data Export and Writing**: Can be exported to `Excel/TXT` files, can be written to `MySQL` databases, can specify data types as `integer/decimal/date`, etc., [click here to view MySQL writing tutorial](https://www.bilibili.com/video/BV1os4y1679S).
9. **Parameter Handling**: The input parameter values when calling tasks can be replaced by reading Excel files.
10. **Interface Adjustment**: The browser operation console can be resized by dragging the top left corner.
11. **Data Handling**: Fields for extracting data can be set to not be saved (suitable for cases where you only want to use this field as a variable input).
12. **Text Input**: After entering text operation, `<enter>` or `<ENTER>` can be used to represent a hard return, that is, press enter in the current text box after entering.
13. **Device Simulation**: Can simulate mobile browser running.
14. (Not Stable) **Cloudflare Handling**: Capable of handling and collecting data from websites protected by Cloudflare's captcha, [click here to view the video tutorial](https://www.bilibili.com/video/BV1Ph4y1E7R9/).
15. **XPath Indexing**: Added a hint for using last() from the back as the default index position in XPath.
16. **Wait Time Control**: The waiting time after the operation can be set to 50%-150% of the set time for random waiting.
17. **Source Code Included**: The software package comes with Python source code for professionals to modify the task process and debugging.
18. **Cookie Handling**: The advanced operations of `open webpage` support getting the current page Cookie and can modify Cookie.
19. **Click Simulation**: Change the way to click elements, truly simulating real-world mouse click operations.
20. **General Parameter Settings**: General parameter settings: how many times to write locally for each collection, the default is 10; control bar preview data length, the default is 15, etc.
21. **File Compression**: Compressed task file size.
22. **Name and Location Changes**: The default file save path is `Data/Task_ID`. If you want to save to a different path, use relative path referencing like `../../`. For example, if the file name is `JS` and you want to save it in a folder at the same level as the `Data` folder, which is the `EasySpider` main folder, you can use `../../JS` as the relative path.
23. **Flowchart Updates**: Automatic update and refresh of the flowchart, no need to click the `Confirm` button.
24. **Source Code Optimization**: Source code optimization, making secondary development easier.
25. **Bug Fixes**: Bug fixes: such as printing error information if the execution of system commands fails, fixing the bug of system command execution failure under MacOS and Linux; URL format judgment and other bugs.
26. Filter irrelevant log information for a cleaner interface execution.