站長日誌

OpenAi 官方的Stream教學 https://platform.openai.com/docs/api-reference/chat/streaming
Gradio 官方的Stream教學 https://www.gradio.app/guides/creating-a-custom-chatbot-with-blocks#add-streaming-to-your-chatbot
兩者結合一下就可以完成Stream, 效果如下

Cloudflare blocks out threats and bad bots and, unfortunately for us, it also assumes all non-whitelisted bot traffic is malicious. This makes web scraping difficult since there’s a good chance our scraper will be denied access to a Cloudflare-protected web page.

One of the best ways to solve this is by using a headless browser, like Selenium, because it’s capable of imitating the activities of a real user. In this guide, we’ll discuss the most effective method to bypass Cloudflare with Selenium.

Let’s jump right in!

What Is Selenium

Selenium is a Python library used for automating web browsers and scraping web pages. Selenium extensions emulate user interaction and provide interactivity in various ways, like enabling the clicking of buttons, scrolling the page, executing custom JavaScript code, simulating user inputs and so on. It automates processes on several browsers, including Firefox and Chrome, using the Webdriver protocol.

What Is Cloudflare and How Does It Work

Cloudflare is a web performance and security company that works to secure and optimize websites and applications. When it comes to security, Cloudflare offers a Web Application Firewall (WAF) that is capable of defending against web attacks, such as cross-site scripting (XSS) and DDoS attacks.

Does Cloudflare Detect Selenium

Unfortunately for us, Cloudflare Bot Management is capable of detecting Selenium. Cloudflare stops malicious HTTP traffic from moving on to the server and it performs security checks to mitigate Layer 7 (application layer) of DDoS attacks. These security checks would identify a Selenium webDriver as a bot.

On a site running with Cloudflare, an interstitial page comes up for 5 seconds. If the checks on the HTTP traffic pass as genuine, the server redirects the user to the page. If not, the page stays there and shows a CAPTCHA.

How Does Cloudflare Block Bots?

Cloudflare’s bot detection techniques can be classified into two categories: passive and active. Passive bot detection uses backend server detection techniques, like TLS fingerprinting, HTTP request headers and IP address reputation. Active bot detection happens on the client side, including CAPTCHAs, event tracking, canvas fingerprinting and others.

Can Selenium Bypass Cloudflare?

Yes! It’s possible to bypass Cloudflare in Selenium. While using base Selenium might not be enough, it’s possible to install extended Selenium libraries to help you avoid Cloudflare detection.

First, let’s run through a quick example to show you why base Selenium isn’t enough. We’ll be using DataCamp, a website with Cloudflare anti-bot protection.

The following tools are necessary to follow along with this tutorial.

• Python3.
• Selenium.

Selenium WebDriver serves as a web automation tool, allowing you to manage web browsers. Previously, you needed to install WebDriver separately, but starting from Selenium version 4 and later, it comes bundled. If you’re using an older version, update to access the latest features and capabilities. Check your current version using pip show selenium and upgrade with pip install --upgrade selenium.

Open a terminal. In your desired directory, install Selenium.

pip install selenium 

In your Python file, copy and paste the code block below.

from selenium import webdriver 
from selenium.webdriver.chrome.options import Options 
import time 
 
options = Options() 
options.add_argument("--headless") # Headless mode

driver = webdriver.Chrome(options=options) 

driver.get("https://www.datacamp.com/users/sign_in") 
 
time.sleep(20) 
 
driver.save_screenshot("datacamp.png") 
 
driver.close()

Run the Python file.

After running the script, the request came back with an error preventing access to the HTML elements, and we can’t crawl the webpage without these elements. Good one, Cloudflare, you win this one!

So how do we tweak Selenium to bypass Cloudflare? We’re getting there.

How to Bypass Cloudflare with Selenium

As we’ve discussed and shown, using base Selenium for Cloudflare just doesn’t work since it isn’t capable of accessing sites with complex anti-bot services. That said, let’s take a look at a few tweaks and tricks available to bypass Cloudflare Selenium.

Selenium Cloudflare Bypass with undetected_chromedriver

undetected_chromedriver is a selenium.webdriver.Chrome replacement and it’s often used when there’s a need to access a site with anti-bot protection as it focuses on stealth. With undetected_chromedriver, a web-driver can be created and used to bypass bot detections, like Cloudflare.

Let’s proceed to use undetected_chromedriver to access the DataCamp sign-in page. Start by installing it.

pip install undetected-chromedriver

Create a Python file and import undetected_chromedriver.

import undetected_chromedriver as uc

Instantiate undetected_chromedriver.

driver = uc.Chrome(...)

With our instantiated driver, let’s make a successful call to DataCamp using .get() from the webdriver. time.sleep is used as an explicit wait condition to keep our window open until the process is done and the maximize_window() method is used to maximize the window if it’s not already maximized.

import undetected_chromedriver as uc 
import time 
 
options = uc.ChromeOptions() 
options.headless = False  # Set headless to False to run in non-headless mode

driver = uc.Chrome(use_subprocess=True, options=options) 
driver.get("https://www.datacamp.com/users/sign_in") 
driver.maximize_window() 

time.sleep(20) 
driver.save_screenshot("datacamp.png") 
driver.close()

Run the created file using the Python command and the name of the file:

python scraper.py

And that’s it, here’s what you’ll see:

Congratulations! You have avoided getting Selenium blocked by Cloudflare, well for now.

Limitations of undetected_chromedriver

If you’re looking to scrape a website with basic anti-bot protection, then undetected_chromedriver might be enough for you. But when it comes to websites that use advanced Cloudflare protection and other DDoS mitigation services, undetected_chromedriver can be unreliable.

For example, we tried to access the Asana page on g2.com using undetected_chromedriver and the steps shared above. Do you want to know what happened? We got blocked.

As you can see, undetected_chromedriver failed because the G2 anti-bot service detected that our browser session was automated. So how do we solve this problem? Easy: using ZenRows.

Selenium Cloudflare bypass using ZenRows

ZenRows is a web scraping tool capable of bypassing different types of antibots, even complex ones, with a simple API call. And yes, it can bypass Cloudflare without stress. Let’s see how.

To get started, create a free ZenRows account and navigate to the Request Builder. We’ll be using Zenrows API, so click on Python and select API from the options on the screen. Paste the URL to scrape, enable Javascript rendering, and Antibot. ZenRows will automatically generate a Python web scraping script for you.

Copy the code generated and paste it into your Python file.

# pip install requests
import requests

url = 'https://www.g2.com/products/asana/reviews'
apikey = '<YOUR_ZENROWS_API_KEY>'
params = {
    'url': url,
    'apikey': apikey,
    'js_render': 'true',
    'antibot': 'true',
    'premium_proxy': 'true',
}
response = requests.get('https://api.zenrows.com/v1/', params=params)
print(response.text)

The final step is to run the script in the terminal.

python scraper.py

Boom! Just like that, we have our output:

C’est fini, you’ve successfully bypassed Cloudflare with ZenRows!

近期在撰寫爬蟲機器人，發現許多網站都裝上了CloudFlare的防火牆，其中5秒盾是最難攻克的一環。
5秒盾的工作原理是，一旦他覺得你可疑，就會把頁面跳到challenge.html，透過Javascript等待5秒，並且重新送出表單逼你按確定。
才可以重新進入網站。

我試過建立完整的HTTPHEADER以及user-agent依然沒用，還是會被跳轉到challenge.html。
用網路找來的透過第三方網站幫你計算Javascript的五秒盾解密也是失敗。
就算用python的webbrowser先開啟Firefox來破解五秒盾後再進網站有時還是會失敗。

至於PHP，原本想法是先用A.php去取得cloudflare的cookie後，再把這個數值換到B.php身上，再訪問網站，依然失敗。
Cloudflare不虧是世界大站，防火牆等級很高XD

但即使如此，我還是發現了可用的方法！

原理說明：
1.訪問目標網站時，若對方開啟cloudflarfe的網站，這時瀏覽器會取得一組「app_session」的cookie

2.cloudflare會針對這組cookie進行檢測，判斷是否有正常訪問和更新。
3.如果「app_session」未有資料，或是進行不正常訪問，頁面就轉到challenge.html去做驗證。

根據上述，會發現關鍵在於「app_session」有持續更新，所以我的作法如下，以php Curl為例：

1.先用Firefox開啟目標網站，取得完整的cookie值以及User-Agent資料
2.把取得的cookie值，套進curl裡面的「curl_setopt($ch, CURLOPT_COOKIE, $setcookie);」$setcookie這個參數。
3.把取得的User-agent資料，套進curl裡面的「curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);」$headers這個參數。

基本上php修改成這樣就可以了。接下來才是關鍵!

4.Firefox安裝套件 Tab Reloader (page auto refresh)
5.Firefox在目標網站啟動Tab Reloader (page auto refresh)這個擴充套件，並且做下圖設定。

Enable Reloader for this tab 及 Bypass form submission: 要啟用，秒數設定10秒。

如此一來，剛剛得到的「app_session」這組資料就會不斷地被更新，也就比較不容易被轉到challenge.html去做驗證。
Firefox跟php不需要同一個IP，換句話說，你可以透過A電腦開啟firefox取得app_session後，拿到B電腦去用。
只要A電腦的有持續更新app_session就可以了。

===2020/01/03補充====
最好是拿兩種瀏覽器如 Chrome及Firefox 同時進行重新整理動作。

上面這個方法我已經測試了24H*14天，目前一切都正常，未有被檔死的經驗。
有被擋過數次，但很快透過Firefox的autorefresh功能後就會通過驗證了。

以上，如果您有更好的方法，也歡迎在下方留言交流!