> For the complete documentation index, see [llms.txt](https://euclid-jie.gitbook.io/euclid_jie_s_book/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://euclid-jie.gitbook.io/euclid_jie_s_book/pa-chong-xiang-guan.md).

# 网络爬虫

这里存储一些爬虫需要注意的点

### 爬虫不能忘记的几个点

* 标签查找要使用稳定的方法
* `selenium`设置隐式等待优于`sleep`

### Cookie

将cookie设置另做函数是一种推荐的做法

```python
def set_headers(cookieFileName):
    """
    设置请求头，主要是cookie的设置，需要登录企查查后，进入开发者工作获取。
    :param cookieFileName: 存储cookie的文件例如：'cookie.txt',str
    :return: 请求头header,dict
    """
    with open(cookieFileName, 'rt', encoding='utf-8') as f:
        cookie = f.read().strip()
    header = {
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:61.0) Gecko/20100101 Firefox/61.0',
        'Cookie': cookie
    }
    print('headers设置成功')
    return header
```

### 爬外网时需要注意自己的梯子

* `requests`设置代理

  ```python
  import requests
  #ssr配置代理，端口查看梯子软件
  proxies={'http': 'http://127.0.0.1:10619', 'https': 'http://127.0.0.1:10619'}
  response = requests.get(Url,proxies=proxies)
  html = response.content.decode('utf-8', 'ignore')
  soup = BeautifulSoup(html, features="lxml")
  ```
* `selenuim`设置代理

  ```python
  from selenium.webdriver import Chrome, ChromeOptions # 导入类库
  option = ChromeOptions() # 初始化类
  ip = "127.0.0.1"
  port = "10619"
  # 设置代理
  option.add_argument("--proxy-server=http://{}:{}".format(ip, port))
  driver = Chrome(options=option)  # 模拟开浏览器
  driver.get(Url) # 跳转网址
  ```