爬虫知识点总结

一，从网页上获取网页的html文件

使用webClient获取网页信息，这个方法传入url就可以获得html的字符串

public static string HttpPost(string url, string paraJsonStr)
        {
            WebClient webClient = new WebClient();
            webClient.Headers.Add("Content-Type", "application/x-www-form-urlencoded");
            byte[] postData = System.Text.Encoding.UTF8.GetBytes(paraJsonStr);
            byte[] responseData = webClient.UploadData(url, "POST", postData);
            string returnStr = System.Text.Encoding.UTF8.GetString(responseData);
            return returnStr;

        }

二，正则匹配

 MatchCollection matches = Regex.Matches(str, @">(.*)</h1>");
            foreach (Match match in matches)
            {
                GroupCollection groups = match.Groups;
               //group[1]是内容
                Console.WriteLine(groups[1].Value);
            }

字典形式 

Dictionary<string, string> resultDict = new Dictionary<string, string>();
foreach (Match matchedGroup in Regex.Matches(str, @"<p>(?<key>.*)</p>\n<ul>\n(?<value>[\w\W\n]*?)</ul>"))
            {
                Console.WriteLine(matchedGroup.Groups["key"].Value);
                if (matchedGroup.Groups["key"].Value.Contains("hahhaha"))
                {
                    resultDict.Add(matchedGroup.Groups["key"].Value, matchedGroup.Groups["value"].Value);
                }
                
            }

三，写入文件

写入txt文件

using (StreamWriter sw = new StreamWriter($"C:\\hhaha.txt"))
            {
                resultDict.ToList().ForEach(x => { sw.WriteLine($"{x.Key}"); sw.WriteLine($"{x.Value}"); });
            }

写入csv文件，其中new StreamWriter($"路径",true)这个true表示已经存在文件就不要新建，没有存在此文件，需要新建这个文件

 using (StreamWriter sw = new StreamWriter($"C:\\Users\\haha1.CSV", true))
                {
                   
                    if (type == "people")
                    {
                        
                        sw.WriteLine("name," + name);
//aValue是一个list,其中有逗号会跳到多个列
                        aValue.ToList().ForEach(x => { sw.WriteLine("hobby," + (x)); });
                    }else if(type== "animals")
                    {
                        
                        sValue.ToList().ForEach(x => { sw.WriteLine("长相," + (x)); });
                        sw.WriteLine("");
                     
                    }
                    else
                    {

                    }

                }

这样在文件上添加新的行

//File.AppendAllLines($"C:\\Users\\hobby.CSV", Value.ToArray());