以编程方式登录网站的技术

Question

我正在尝试自动登录到Photobucket的API使用的项目，需要使用存储的凭据自动下载照片。

API生成一个用于登录的URL，使用Firebug，我可以看到正在发送/接收哪些请求和响应。

我的问题是，如何使用HttpWebRequest和HttpWebResponse在C#中模拟浏览器中发生的事情？

是否可以在C#应用程序中使用web浏览器组件，填写用户名和密码字段并提交登录信息？

Answer 1

我以前做过这样的事情，最后得到了一个很好的工具包来编写这些类型的应用程序。我已经使用这个工具包来处理非平凡的来回web请求，所以这是完全可能的，并且不是非常困难。

我很快就发现，从头开始做HttpWebRequest/HttpWebResponse确实比我想要处理的级别要低。我的工具完全基于Simon Mourier的HtmlAgilityPack。这是一个很棒的工具集。它为您做了很多繁重的工作，并使解析所获取的HTML变得非常容易。如果您可以实现XPath查询，那么HtmlAgilityPack就是您想要开始的地方。它也能很好地处理格式不佳的HTML！

您仍然需要一个好的工具来帮助调试。除了您的调试器中所具有的功能之外，能够在网络上来回检查http/https流量也是无价的。由于发出这些请求的是您的代码，而不是您的浏览器，因此FireBug对调试您的代码没有多大帮助。有各种各样的包嗅探工具，但对于HTTP/HTTPS调试，我认为你无法超越Fiddler 2的易用性和强大功能。最新版本甚至还为firefox提供了一个插件，可以快速地在fiddler之间来回转移请求。因为它还可以充当无缝的HTTPS代理，所以您还可以检查HTTPS流量。

试一试，我相信它们会是你黑客攻击中不可缺少的两个工具。

更新：添加了以下代码示例。这是从一个不太大的“会话”类中提取出来的，它登录到一个网站，并为您保存相关的cookie。我之所以选择这个，是因为它做的不仅仅是简单的“请帮我取那个网页”代码，还有一两行针对最终目标页面的XPath查询。

public bool Connect() {
   if (string.IsNullOrEmpty(_Username)) { base.ThrowHelper(new SessionException("Username not specified.")); } 
   if (string.IsNullOrEmpty(_Password)) { base.ThrowHelper(new SessionException("Password not specified.")); }

   _Cookies = new CookieContainer();
   HtmlWeb webFetcher = new HtmlWeb();
   webFetcher.UsingCache = false;
   webFetcher.UseCookies = true;

   HtmlWeb.PreRequestHandler justSetCookies = delegate(HttpWebRequest webRequest) {
      SetRequestHeaders(webRequest, false);
      return true;
   };
   HtmlWeb.PreRequestHandler postLoginInformation = delegate(HttpWebRequest webRequest) {
      SetRequestHeaders(webRequest, false);

      // before we let webGrabber get the response from the server, we must POST the login form's data
      // This posted form data is *VERY* specific to the web site in question, and it must be exactly right,
      // and exactly what the remote server is expecting, otherwise it will not work!
      //
      // You need to use an HTTP proxy/debugger such as Fiddler in order to adequately inspect the 
      // posted form data. 
      ASCIIEncoding encoding = new ASCIIEncoding();
      string postDataString = string.Format("edit%5Bname%5D={0}&edit%5Bpass%5D={1}&edit%5Bform_id%5D=user_login&op=Log+in", _Username, _Password);
      byte[] postData = encoding.GetBytes(postDataString);
      webRequest.ContentType = "application/x-www-form-urlencoded";
      webRequest.ContentLength = postData.Length;
      webRequest.Referer = Util.MakeUrlCore("/user"); // builds a proper-for-this-website referer string

      using (Stream postStream = webRequest.GetRequestStream()) {
         postStream.Write(postData, 0, postData.Length);
         postStream.Close();
      }

      return true;
   };

   string loginUrl = Util.GetUrlCore(ProjectUrl.Login); 
   bool atEndOfRedirects = false;
   string method = "POST";
   webFetcher.PreRequest = postLoginInformation;

   // this is trimmed...this was trimmed in order to handle one of those 'interesting' 
   // login processes...
   webFetcher.PostResponse = delegate(HttpWebRequest webRequest, HttpWebResponse response) {
      if (response.StatusCode == HttpStatusCode.Found) {
         // the login process is forwarding us on...update the URL to move to...
         loginUrl = response.Headers["Location"] as String;
         method = "GET";
         webFetcher.PreRequest = justSetCookies; // we only need to post cookies now, not all the login info
      } else {
         atEndOfRedirects = true;
      }

      foreach (Cookie cookie in response.Cookies) {
         // *snip*
      }
   };

   // Real work starts here:
   HtmlDocument retrievedDocument = null;
   while (!atEndOfRedirects) {
      retrievedDocument = webFetcher.Load(loginUrl, method);
   }


   // ok, we're fully logged in.  Check the returned HTML to see if we're sitting at an error page, or
   // if we're successfully logged in.
   if (retrievedDocument != null) {
      HtmlNode errorNode = retrievedDocument.DocumentNode.SelectSingleNode("//div[contains(@class, 'error')]");
      if (errorNode != null) { return false; }
   }

   return true; 
}


public void SetRequestHeaders(HttpWebRequest webRequest) { SetRequestHeaders(webRequest, true); }
public void SetRequestHeaders(HttpWebRequest webRequest, bool allowAutoRedirect) {
   try {
      webRequest.AllowAutoRedirect = allowAutoRedirect;
      webRequest.CookieContainer = _Cookies;

      // the rest of this stuff is just to try and make our request *look* like FireFox. 
      webRequest.UserAgent = @"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3";
      webRequest.Accept = @"text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
      webRequest.KeepAlive = true;
      webRequest.Headers.Add(@"Accept-Language: en-us,en;q=0.5");
      //webRequest.Headers.Add(@"Accept-Encoding: gzip,deflate");
   }
   catch (Exception ex) { base.ThrowHelper(ex); }
}

Answer 2

下面是我解决这个问题的方法：

public partial class Form1 : Form {
        private string LoginUrl = "/apilogin/login";
        private string authorizeUrl = "/apilogin/authorize";
        private string doneUrl = "/apilogin/done";

        public Form1() {
            InitializeComponent();
            this.Load += new EventHandler(Form1_Load);
        }

        void Form1_Load(object sender, EventArgs e) {
            PhotobucketNet.Photobucket pb = new Photobucket("pubkey","privatekey");
            string url = pb.GenerateUserLoginUrl();
            webBrowser1.Url = new Uri(url);
            webBrowser1.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(webBrowser1_DocumentCompleted);
        }

        void webBrowser1_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e) {
            if (e.Url.AbsolutePath.StartsWith(LoginUrl))
            {
                webBrowser1.Document.GetElementById("usernameemail").SetAttribute("Value","some username");
                webBrowser1.Document.GetElementById("password").SetAttribute("Value","some password");
                webBrowser1.Document.GetElementById("login").InvokeMember("click");
            }

            if (e.Url.AbsolutePath.StartsWith(authorizeUrl))
            {
                webBrowser1.Document.GetElementById("allow").InvokeMember("click");
            }

            if (e.Url.AbsolutePath.StartsWith(doneUrl))
            {
                string token = webBrowser1.Document.GetElementById("oauth_token").GetAttribute("value");
            }
        }
    }

最后一个if块中捕获的令牌是继续使用API所需的。这种方法对我来说工作得很好，因为需要它的代码将在windows上运行，所以我可以毫不费力地产生一个进程来加载这个单独的应用程序来提取令牌。

Answer 3

可以使用本机WebbrowserControl登录网站。但正如您在示例中看到的，您必须在此之前识别控件的名称。

private void webBrowserLogin_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
    {

        if (webBrowserLogin.Url.ToString() == WebSiteUrl)
        {
            foreach (HtmlElement elem in webBrowserLogin.Document.All)
            {

                if (elem.Name == "user_name")              // name of the username input
                {
                    elem.InnerText = UserName;               
                }

                if (elem.Name == "password")               // name of the password input
                {
                    elem.InnerText = Password;                
                } 

            }

            foreach (HtmlElement elem in webBrowserLogin.Document.All)
            {

                if (elem.GetAttribute("value") == "Login")
                {
                    elem.InvokeMember("Click");
                }
            }
        }
    }