java 網絡編程-爬蟲+模擬瀏覽器

網絡爬蟲+模擬瀏覽器(獲取有權限網站資源)：
獲取URL
下載資源
分析
處理

public class http {

public static void main(String[]args) throws  Exception
{
    //http+s更安全
    //URL.openStream()打開于URL的連接，并返回一個InputStream用于從連接中讀取數據
    //獲取URL
    URL url=new URL("https://www.jd.com");
    //下載資源
    InputStream is = url.openStream();
    BufferedReader br=new BufferedReader(new InputStreamReader(is,"UTF-8"));;
    String msg=null;
    while((msg=br.readLine())!=null)
    {

        System.out.println(msg);
    }
    br.close();
}
}

獲取有權限網絡資源:

public class http {

public static void main(String[]args) throws  Exception
{
    //.openConnectio，,返回一個URLConnection實例表示由所引用的遠程對象的連接URL
    //URLConnection的子類有HttpURLConnection和JarURLConnection
    URL url=new URL("https://www.jd.com");
    //下載資源
    HttpURLConnection conn=(HttpURLConnection)url.openConnection();
    conn.setRequestMethod("GET");//模擬瀏覽器得get請求
    conn.setRequestProperty( "User-Agent","Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/18.17763");
    BufferedReader br=new BufferedReader(new InputStreamReader(conn.getInputStream(),"UTF-8"));
    String msg=null;
    while((msg=br.readLine())!=null)
    {

        System.out.println(msg);
    }
    br.close();
}
}