亚马逊和 HTML AgiltiyPack

Question

似乎我的所有请求都遇到了 500 服务错误。

但是我可以通过网络浏览器访问该网站。我也通过 HAP 发送 UserAgent。

有这方面有经验的人知道亚马逊如何检测HAP请求是机器人吗？

  Dim aHtml As New HtmlWeb
    Dim UserAgent1 As String = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11"
    Dim iPadAgent As String = "Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B334b Safari/531.21.102011-10-16 20:23:10"
    Dim AndroidAgent As String = "Mozilla/5.0 (Linux; U; Android 3.0; en-us; Xoom Build/HRI39) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13"
    Dim iPhoneAgent As String = "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_2 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8H7 Safari/6533.18.5"
    Dim ChromeAgent As String = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11"
    aHtml.UserAgent = iPadAgent

    Dim Credentials As New System.Net.NetworkCredential("", "", "")

    Dim proxy As New System.Net.WebProxy


    Dim proxyAddress As New Uri("http://11.11.11.11")
    proxy.Address = proxyAddress

    Dim aDoc As HtmlDocument = aHtml.Load(AmazonURL, "GET")

    Dim aNode As HtmlAgilityPack.HtmlNode
    aNode = aDoc.DocumentNode.SelectSingleNode("//div[@id='olpDivId']/span[2]")

    If aNode.InnerText Is Nothing Then

    End If

    Dim UsedPrice1 As String = aNode.InnerText
    Dim i As Integer = UsedPrice1.IndexOf("$")
    Dim UsedPrice As Integer = UsedPrice1.Substring(i + 1)

    System.Threading.Thread.Sleep(Delay)

    Return UsedPrice

'

WebClient 方法（返回 503）：

 Dim filename As String = "amzContent.html"
    Client.Headers("User-Agent") = ChromeAgent
    Client.Proxy = proxy
    Client.Credentials = Credentials
    Client.DownloadFile(AmazonURL, (Server.MapPath("amz//") & _
                  filename))
    Dim aDoc As HtmlDocument = aHtml.Load(Server.MapPath("amz//") & _
                  filename)

Answer 1

使用上面的 WebClient 方法，并发送语言、字符集和接受标头现在似乎可以工作