前言
欢迎来到马士兵教育,和我们一起探究Python的奥妙。
本次课程共分为三天,你将学到爬虫基础知识、代码优化、数据的存储…
老师会手把手带你实操写代码,一起感受python给我们的生活和工作带来的高效😊
同时我们还有助教老师👧跟踪辅导,绝不让疑问和困难阻挡你探索的脚步!
开始学习前老师还有个小tips:将学习界面添加到收藏夹,学习更便捷😎
OK~一切准备就绪,let’s go💪💪💪
初识python爬虫
什么是爬虫?
- 爬取网络数据的虫子(Python程序)
爬虫实质是什么呢?
- 模拟浏览器的工作原理,向服务器请求相应的数据
浏览器的工作原理

- 浏览器在这个过程中还起到了翻译数据的作用哦
爬虫的工作原理如下图:

数据背后的秘密
找不到这双鞋子的销售数据怎么办?
- 曲线救国,通过评论数据间接得到鞋子的销售数据
如何找到评论区内容背后的URL?
-
(1)鼠标右击选择检查,打开程序员调试窗口,点击network(网络)
-
(2)刷新当前页面
-
(3)复制一小段评论区内容,然后在程序员调试窗口点击放大镜🔎,粘贴
-
(4)点击刷新小圆圈🔄查找
-
(5)点击查询结果的第二行,跳转到对应的请求
-
(6)点击Headers,找到Request URL即几评论区数据背后的URL
3行代码爬取京东数据
梳理代码流程:
-
(1)引入Python工具包requests
-
(2)使用工具包中的get方法,向服务器发起请求
-
(3)打印输出请求回来的数据(print语法)
借助于chrome浏览器的强大功能找到真实的url请求地址

输入:
import requests
resp = requests.get('https://club.jd.com/comment/productPageComments.action?callback=fetchJSON_comment98&productId=72316371472&score=0&sortType=5&page=0&pageSize=10&isShadowSku=0&fold=1')
print(resp.text)
输出:
fetchJSON_comment98({"productAttr":null,"productCommentSummary":{"skuId":72316371472,"averageScore":5,"defaultGoodCount":0,"defaultGoodCountStr":"0","commentCount":38,"commentCountStr":"30+","goodCount":37,"goodCountStr":"30+","goodRate":0.97,"goodRateShow":97,"generalCount":1,"generalCountStr":"1","generalRate":0.03,"generalRateShow":3,"poorCount":0,"poorCountStr":"0","poorRate":0.0,"poorRateShow":0,"videoCount":0,"videoCountStr":"0","afterCount":0,"afterCountStr":"0","showCount":0,"showCountStr":"0","oneYear":0,"sensitiveBook":0,"fixCount":0,"plusCount":0,"plusCountStr":"0","buyerShow":0,"poorRateStyle":0,"generalRateStyle":5,"goodRateStyle":145,"installRate":0,"productId":72316371472,"score1Count":0,"score2Count":0,"score3Count":1,"score4Count":0,"score5Count":37},"hotCommentTagStatistics":[{"id":"68d9fe2b7dda7fa5","name":"穿行舒适","count":15,"type":4,"canBeFiltered":true,"stand":1,"rid":"68d9fe2b7dda7fa5","ckeKeyWordBury":"eid=100^^tagid=68d9fe2b7dda7fa5^^pid=20008^^sku=72316371472^^sversion=1000^^token=542601cacfb53118"},{"id":"9815fee0e9450c2d","name":"轻便舒适","count":7,"type":4,"canBeFiltered":true,"stand":1,"rid":"9815fee0e9450c2d","ckeKeyWordBury":"eid=100^^tagid=9815fee0e9450c2d^^pid=20008^^sku=72316371472^^sversion=1000^^token=d838dcb7f49e2cf6"},{"id":"ced7252f693db71f","name":"底软舒适","count":7,"type":4,"canBeFiltered":true,"stand":1,"rid":"ced7252f693db71f","ckeKeyWordBury":"eid=100^^tagid=ced7252f693db71f^^pid=20008^^sku=72316371472^^sversion=1000^^token=8154e38864da3dce"},{"id":"85c6824f5e9e52fb","name":"舒适合脚","count":4,"type":4,"canBeFiltered":true,"stand":1,"rid":"85c6824f5e9e52fb","ckeKeyWordBury":"eid=100^^tagid=85c6824f5e9e52fb^^pid=20008^^sku=72316371472^^sversion=1000^^token=db8e999c25217239"},{"id":"6b4b635ef67faaee","name":"简单大方","count":2,"type":4,"canBeFiltered":true,"stand":1,"rid":"6b4b635ef67faaee","ckeKeyWordBury":"eid=100^^tagid=6b4b635ef67faaee^^pid=20008^^sku=72316371472^^sversion=1000^^token=a0003ae85a9d5c72"},{"id":"e6b427362f053488","name":"尺码精准","count":2,"type":4,"canBeFiltered":true,"stand":1,"rid":"e6b427362f053488","ckeKeyWordBury":"eid=100^^tagid=e6b427362f053488^^pid=20008^^sku=72316371472^^sversion=1000^^token=2732eeef4cce5452"},{"id":"3d2d90297babe95a","name":"耐磨防滑","count":2,"type":4,"canBeFiltered":true,"stand":1,"rid":"3d2d90297babe95a","ckeKeyWordBury":"eid=100^^tagid=3d2d90297babe95a^^pid=20008^^sku=72316371472^^sversion=1000^^token=575e98ae3abce984"},{"id":"b6d917307b3bfea8","name":"豪无色差","count":1,"type":4,"canBeFiltered":true,"stand":1,"rid":"b6d917307b3bfea8","ckeKeyWordBury":"eid=100^^tagid=b6d917307b3bfea8^^pid=20008^^sku=72316371472^^sversion=1000^^token=34ba455e7e1c6e04"},{"id":"a100fe5e12c2d634","name":"外形尚可","count":1,"type":4,"canBeFiltered":true,"stand":1,"rid":"a100fe5e12c2d634","ckeKeyWordBury":"eid=100^^tagid=a100fe5e12c2d634^^pid=20008^^sku=72316371472^^sversion=1000^^token=3fa3a827dc8b5520"},{"id":"1a6afff8063c8fcf","name":"样式靓丽","count":1,"type":4,"canBeFiltered":true,"stand":1,"rid":"1a6afff8063c8fcf","ckeKeyWordBury":"eid=100^^tagid=1a6afff8063c8fcf^^pid=20008^^sku=72316371472^^sversion=1000^^token=f4664df8e6a49dfc"},{"id":"f8952a488a11ced2","name":"做工精细","count":1,"type":4,"canBeFiltered":true,"stand":1,"rid":"f8952a488a11ced2","ckeKeyWordBury":"eid=100^^tagid=f8952a488a11ced2^^pid=20008^^sku=72316371472^^sversion=1000^^token=9492fefa4f702e95"}],"jwotestProduct":null,"maxPage":4,"testId":"cmt","score":0,"soType":5,"imageListCount":0,"vTagStatistics":null,"csv":"eid=100^^tagid=ALL^^pid=20008^^sku=14748638360^^sversion=1001^^pageSize=1","comments":[{"id":14503666640,"guid":"0215dabbcb5d633c0fca1c4a7a92209d","content":"网面透气。鞋底柔软。穿起来舒适,不磨脚。穿起来特别好看。价格又合理。客服的服务态度也很不错。非常满意","creationTime":"2020-08-07 09:37:59","isDelete":false,"isTop":false,"userImageUrl":"misc.360buyimg.com/user/myjd-2015/css/i/peisong.jpg","topped":0,"replyCount":0,"score":5,"title":"","usefulVoteCount":0,"userClient":2,"anonymousFlag":1,"plusAvailable":0,"mobileVersion":"","productColor":"白色【七彩】","productSize":"41","textIntegral":20,"status":1,"referenceId":"72316371472","referenceTime":"2020-08-06 10:33:40","nickname":"j***f","replyCount2":0,"userImage":"misc.360buyimg.com/user/myjd-2015/css/i/peisong.jpg","orderId":0,"integral":20,"productSales":"[]","referenceImage":"jfs/t1/130686/1/6200/321936/5f2accedE73e9c6f9/57bdfa3587d8ab5d.jpg","referenceName":"男鞋网面透气椰子鞋运动休闲鞋子男跑步鞋复古老爹鞋透气网面小白鞋网鞋增高大码帆布潮鞋 白色【七彩】 41","firstCategory":11729,"secondCategory":11730,"thirdCategory":6908,"aesPin":null,"days":1,"afterDays":0},{"id":14503635855,"guid":"d3d6f20e1e49f66cc08435d94ac86f9e","content":"穿着轻便舒适,透气性良好。尺寸合适。鞋底厚实耐磨耐穿。外形好看。价格也不贵。物流也很给力,还会再来","creationTime":"2020-08-07 09:28:57","isDelete":false,"isTop":false,"userImageUrl":"misc.360buyimg.com/user/myjd-2015/css/i/peisong.jpg","topped":0,"replyCount":0,"score":5,"title":"","usefulVoteCount":0,"userClient":2,"anonymousFlag":1,"plusAvailable":0,"mobileVersion":"","productColor":"白色【七彩】","productSize":"41","textIntegral":20,"status":1,"referenceId":"72316371472","referenceTime":"2020-08-06 10:25:22","nickname":"j***V","replyCount2":0,"userImage":"misc.360buyimg.com/user/myjd-2015/css/i/peisong.jpg","orderId":0,"integral":20,"productSales":"[]","referenceImage":"jfs/t1/130686/1/6200/321936/5f2accedE73e9c6f9/57bdfa3587d8ab5d.jpg","referenceName":"男鞋网面透气椰子鞋运动休闲鞋子男跑步鞋复古老爹鞋透气网面小白鞋网鞋增高大码帆布潮鞋 白色【七彩】 41","firstCategory":11729,"secondCategory":11730,"thirdCategory":6908,"aesPin":null,"days":1,"afterDays":0},{"id":14503600613,"guid":"503e98a15c019243361982339900af6e","content":"鞋子的外观特别好看。没有色差,关键穿着很舒服,很满意,性价比很高的一款鞋子。特别百搭,推荐你购买。","vcontent":"鞋子的外观特别好看。没有色差,关键穿着很舒服,很满意,性价比很高的一款鞋子。特别百搭,推荐你购买。","creationTime":"2020-08-07 09:18:58","isDelete":false,"isTop":false,"userImageUrl":"misc.360buyimg.com/user/myjd-2015/css/i/peisong.jpg","topped":0,"replyCount":0,"score":5,"title":"","usefulVoteCount":0,"userClient":2,"anonymousFlag":1,"plusAvailable":0,"mobileVersion":"","productColor":"白色【七彩】","productSize":"41","textIntegral":20,"status":1,"referenceId":"72316371472","referenceTime":"2020-08-06 11:09:12","nickname":"j***h","replyCount2":0,"userImage":"misc.360buyimg.com/user/myjd-2015/css/i/peisong.jpg","orderId":0,"integral":20,"productSales":"[]","referenceImage":"jfs/t1/130686/1/6200/321936/5f2accedE73e9c6f9/57bdfa3587d8ab5d.jpg","referenceName":"男鞋网面透气椰子鞋运动休闲鞋子男跑步鞋复古老爹鞋透气网面小白鞋网鞋增高大码帆布潮鞋 白色【七彩】 41","firstCategory":11729,"secondCategory":11730,"thirdCategory":6908,"aesPin":null,"days":1,"afterDays":0},{"id":14503646340,"guid":"ae62e4080fa5250fb6a78d705912877c","content":"鞋面透气。适合夏天穿着。鞋底材质柔软。穿着轻便。走路很舒适。也不臭脚。良心鞋。良心价。值得拥有","creationTime":"2020-08-07 09:33:17","isDelete":false,"isTop":false,"userImageUrl":"misc.360buyimg.com/user/myjd-2015/css/i/peisong.jpg","topped":0,"replyCount":0,"score":5,"title":"","usefulVoteCount":0,"userClient":2,"anonymousFlag":1,"plusAvailable":0,"mobileVersion":"","productColor":"白色【七彩】","productSize":"41","textIntegral":20,"status":1,"referenceId":"72316371472","referenceTime":"2020-08-06 10:29:25","nickname":"j***k","replyCount2":0,"userImage":"misc.360buyimg.com/user/myjd-2015/css/i/peisong.jpg","orderId":0,"integral":20,"productSales":"[]","referenceImage":"jfs/t1/130686/1/6200/321936/5f2accedE73e9c6f9/57bdfa3587d8ab5d.jpg","referenceName":"男鞋网面透气椰子鞋运动休闲鞋子男跑步鞋复古老爹鞋透气网面小白鞋网鞋增高大码帆布潮鞋 白色【七彩】 41","firstCategory":11729,"secondCategory":11730,"thirdCategory":6908,"aesPin":null,"days":1,"afterDays":0},{"id":14503593099,"guid":"7c96e77cd24453da144b41bd3d54fe43","content":"鞋子穿上脚感很好,鞋底柔软。走起路来感觉很舒服,款式很好看,洋气,很喜欢这种质量透气性也好。","vcontent":"鞋子穿上脚感很好,鞋底柔软。走起路来感觉很舒服,款式很好看,洋气,很喜欢这种质量透气性也好。","creationTime":"2020-08-07 09:17:34","isDelete":false,"isTop":false,"userImageUrl":"misc.360buyimg.com/user/myjd-2015/css/i/peisong.jpg","topped":0,"replyCount":0,"score":5,"title":"","usefulVoteCount":0,"userClient":2,"anonymousFlag":1,"plusAvailable":0,"mobileVersion":"","productColor":"白色【七彩】","productSize":"41","textIntegral":0,"status":1,"referenceId":"72316371472","referenceTime":"2020-08-06 11:01:10","nickname":"j***V","replyCount2":0,"userImage":"misc.360buyimg.com/user/myjd-2015/css/i/peisong.jpg","orderId":0,"integral":0,"productSales":"[]","referenceImage":"jfs/t1/130686/1/6200/321936/5f2accedE73e9c6f9/57bdfa3587d8ab5d.jpg","referenceName":"男鞋网面透气椰子鞋运动休闲鞋子男跑步鞋复古老爹鞋透气网面小白鞋网鞋增高大码帆布潮鞋 白色【七彩】 41","firstCategory":11729,"secondCategory":11730,"thirdCategory":6908,"aesPin":null,"days":1,"afterDays":0},{"id":14503583159,"guid":"1eeb25c247a35a4434d25681c82dae2b","content":"看了一下评价,反应都不错,买了双试试,物流很快到了。看到鞋子很好,鞋面柔软,鞋底舒适,弹性很好。","vcontent":"看了一下评价,反应都不错,买了双试试,物流很快到了。看到鞋子很好,鞋面柔软,鞋底舒适,弹性很好。","creationTime":"2020-08-07 09:14:51","isDelete":false,"isTop":false,"userImageUrl":"misc.360buyimg.com/user/myjd-2015/css/i/peisong.jpg","topped":0,"replyCount":0,"score":5,"title":"","usefulVoteCount":0,"userClient":2,"anonymousFlag":1,"plusAvailable":0,"mobileVersion":"","productColor":"白色【七彩】","productSize":"41","textIntegral":20,"status":1,"referenceId":"72316371472","referenceTime":"2020-08-06 10:56:11","nickname":"j***r","replyCount2":0,"userImage":"misc.360buyimg.com/user/myjd-2015/css/i/peisong.jpg","orderId":0,"integral":20,"productSales":"[]","referenceImage":"jfs/t1/130686/1/6200/321936/5f2accedE73e9c6f9/57bdfa3587d8ab5d.jpg","referenceName":"男鞋网面透气椰子鞋运动休闲鞋子男跑步鞋复古老爹鞋透气网面小白鞋网鞋增高大码帆布潮鞋 白色【七彩】 41","firstCategory":11729,"secondCategory":11730,"thirdCategory":6908,"aesPin":null,"days":1,"afterDays":0},{"id":14503840751,"guid":"5d55064f6bdf265a63d003df1012998a","content":"快递收到了。物流很快。实物与图片相符。穿着挺舒适的。透气性良好。鞋底耐磨防滑。物美价廉。很满意的一次购物","creationTime":"2020-08-07 10:26:39","isDelete":false,"isTop":false,"userImageUrl":"misc.360buyimg.com/user/myjd-2015/css/i/peisong.jpg","topped":0,"replyCount":0,"score":5,"title":"","usefulVoteCount":0,"userClient":2,"anonymousFlag":1,"plusAvailable":0,"mobileVersion":"","productColor":"白色【七彩】","productSize":"41","textIntegral":20,"status":1,"referenceId":"72316371472","referenceTime":"2020-08-06 10:52:48","nickname":"j***Q","replyCount2":0,"userImage":"misc.360buyimg.com/user/myjd-2015/css/i/peisong.jpg","orderId":0,"integral":20,"productSales":"[]","referenceImage":"jfs/t1/130686/1/6200/321936/5f2accedE73e9c6f9/57bdfa3587d8ab5d.jpg","referenceName":"男鞋网面透气椰子鞋运动休闲鞋子男跑步鞋复古老爹鞋透气网面小白鞋网鞋增高大码帆布潮鞋 白色【七彩】 41","firstCategory":11729,"secondCategory":11730,"thirdCategory":6908,"aesPin":null,"days":1,"afterDays":0},{"id":14503706295,"guid":"5751e298c4bfa745a1f908daa751c27f","content":"鞋子很快就收到了,迫不及待的穿上,鞋子面料很好,大小合适。质量也不错,穿着蛮舒服的。给个好评","creationTime":"2020-08-07 09:48:30","isDelete":false,"isTop":false,"userImageUrl":"misc.360buyimg.com/user/myjd-2015/css/i/peisong.jpg","topped":0,"replyCount":0,"score":5,"title":"","usefulVoteCount":0,"userClient":2,"anonymousFlag":1,"plusAvailable":0,"mobileVersion":"","productColor":"白色【七彩】","productSize":"41","textIntegral":20,"status":1,"referenceId":"72316371472","referenceTime":"2020-08-06 10:41:19","nickname":"j***s","replyCount2":0,"userImage":"misc.360buyimg.com/user/myjd-2015/css/i/peisong.jpg","orderId":0,"integral":20,"productSales":"[]","referenceImage":"jfs/t1/130686/1/6200/321936/5f2accedE73e9c6f9/57bdfa3587d8ab5d.jpg","referenceName":"男鞋网面透气椰子鞋运动休闲鞋子男跑步鞋复古老爹鞋透气网面小白鞋网鞋增高大码帆布潮鞋 白色【七彩】 41","firstCategory":11729,"secondCategory":11730,"thirdCategory":6908,"aesPin":null,"days":1,"afterDays":0},{"id":14503705910,"guid":"be18d36e8aac5bd52793486d1ce9e5fc","content":"鞋子质量很好 舒服透气 大小尺寸 也刚好 穿上很舒服 透气性很好 价格还亲民","vcontent":"鞋子质量很好 舒服透气 大小尺寸 也刚好 穿上很舒服 透气性很好 价格还亲民","creationTime":"2020-08-07 09:49:09","isDelete":false,"isTop":false,"userImageUrl":"misc.360buyimg.com/user/myjd-2015/css/i/peisong.jpg","topped":0,"replyCount":0,"score":5,"title":"","usefulVoteCount":0,"userClient":2,"anonymousFlag":1,"plusAvailable":0,"mobileVersion":"","productColor":"白色【七彩】","productSize":"41","textIntegral":20,"status":1,"referenceId":"72316371472","referenceTime":"2020-08-06 11:13:21","nickname":"j***n","replyCount2":0,"userImage":"misc.360buyimg.com/user/myjd-2015/css/i/peisong.jpg","orderId":0,"integral":20,"productSales":"[]","referenceImage":"jfs/t1/130686/1/6200/321936/5f2accedE73e9c6f9/57bdfa3587d8ab5d.jpg","referenceName":"男鞋网面透气椰子鞋运动休闲鞋子男跑步鞋复古老爹鞋透气网面小白鞋网鞋增高大码帆布潮鞋 白色【七彩】 41","firstCategory":11729,"secondCategory":11730,"thirdCategory":6908,"aesPin":null,"days":1,"afterDays":0},{"id":14503720311,"guid":"eec0ce32f576c0292b22032afb9e14aa","content":"鞋子很好看。穿起来和看起来非常透气。外观也不错 大小合适。希望店家继续努力","vcontent":"鞋子很好看。穿起来和看起来非常透气。外观也不错 大小合适。希望店家继续努力","creationTime":"2020-08-07 09:52:40","isDelete":false,"isTop":false,"userImageUrl":"misc.360buyimg.com/user/myjd-2015/css/i/peisong.jpg","topped":0,"replyCount":0,"score":5,"title":"","usefulVoteCount":0,"userClient":2,"anonymousFlag":1,"plusAvailable":0,"mobileVersion":"","productColor":"白色【七彩】","productSize":"41","textIntegral":20,"status":1,"referenceId":"72316371472","referenceTime":"2020-08-06 10:32:01","nickname":"j***i","replyCount2":0,"userImage":"misc.360buyimg.com/user/myjd-2015/css/i/peisong.jpg","orderId":0,"integral":20,"productSales":"[]","referenceImage":"jfs/t1/130686/1/6200/321936/5f2accedE73e9c6f9/57bdfa3587d8ab5d.jpg","referenceName":"男鞋网面透气椰子鞋运动休闲鞋子男跑步鞋复古老爹鞋透气网面小白鞋网鞋增高大码帆布潮鞋 白色【七彩】 41","firstCategory":11729,"secondCategory":11730,"thirdCategory":6908,"aesPin":null,"days":1,"afterDays":0}]});