NanmiCoder/MediaCrawler issues and pull requests

#542 - Update core.py，为爬取类型为`detail`和`creator`的任务，添加了和`search`任务一样的，用于转存up主信息的`bilibili_store.update_up_info`的函数调用

Pull Request - State: closed - Opened by 2513502304 4 days ago

#541 - 抖音爬取指定作品的评论遇到了2025-01-18 14:00:38 MediaCrawler ERROR (core.py:150) - [DouYinCrawler.get_aweme_detail] Get aweme detail error: account blocked,

Issue - State: open - Opened by 1301633938 5 days ago
Labels: question, 风控问题不处理

#540 - 请问开启了小红书一级评论抓取后，在终端中又打印出来，但是最终数据保存只有根据关键词抓取的帖子内容该怎么办？

Issue - State: closed - Opened by Zoey-xc 6 days ago
Labels: question

#539 - [问题]

Issue - State: closed - Opened by Zoey-xc 6 days ago
Labels: question

#539 - [问题]

Issue - State: closed - Opened by Zoey-xc 6 days ago
Labels: question

#538 - 为`base_config.py`添加了是否开启按每一天进行爬取的选项，仅支持bilibiili，Update `core.py`和`client.py`以实现该功能

Pull Request - State: closed - Opened by 2513502304 8 days ago - 2 comments

#538 - 为`base_config.py`添加了是否开启按每一天进行爬取的选项，仅支持bilibiili，Update `core.py`和`client.py`以实现该功能

Pull Request - State: closed - Opened by 2513502304 8 days ago - 2 comments

#536 - 将微博评论爬取函数`get_note_all_comments`的`max_id_type`便为可变请求参数

Pull Request - State: closed - Opened by 2513502304 13 days ago - 2 comments

#535 - 由于获取max_id为None时，导致触发'>' not supported between instances of 'NoneType' and 'int'错误的修复意见

Issue - State: open - Opened by 2513502304 13 days ago - 2 comments

#534 - 请问知乎回答现在可以正常采集吗

Issue - State: open - Opened by xiaohanhahaha 15 days ago - 4 comments

#533 - 请问一下爬取评论是按照时间顺序还是热度的顺序爬取的？

Issue - State: open - Opened by Alanhahaer 15 days ago - 4 comments

#532 - 求助，get_a_bogus提示语法错误

Issue - State: open - Opened by Schwimmer 15 days ago - 2 comments
Labels: question

#531 - xhs有办法爬取搜索关键词下，最新的内容吗？

Issue - State: open - Opened by S1E9 17 days ago - 3 comments

#530 - 小红书现在扫码登录失效了，请问这是为什么呢

Issue - State: open - Opened by Tony2048Cao 19 days ago - 2 comments

#529 - 小红书获取主页数据的时候，用户个人简要信息获取不到

Issue - State: open - Opened by GithubLinfuxi 25 days ago - 1 comment

#528 - fix xhs gender

Pull Request - State: closed - Opened by HuiGitH 26 days ago - 1 comment

#527 - ip获取

Issue - State: open - Opened by ChenWenlon 28 days ago
Labels: enhancement

#526 - 爬取得到的 json ，可以给一个参数说明嘛

Issue - State: open - Opened by Dream-ming 28 days ago - 2 comments
Labels: question

#525 - 无论是爬取什么都无法成功

Issue - State: closed - Opened by mytechdream about 1 month ago - 1 comment

#524 - 爬取抖音数据最多爬取25页，超过后无法获取数据

Issue - State: open - Opened by Bugliiiiii about 1 month ago
Labels: 风控问题不处理

#523 - python main.py --help出现错误求解答

Issue - State: open - Opened by SamWang8888 about 1 month ago - 1 comment
Labels: question

#522 - xhs 获取作者信息失败

Issue - State: open - Opened by hezhenfan about 1 month ago
Labels: 风控问题不处理

#521 - 111

Pull Request - State: closed - Opened by cwjcw about 1 month ago

#520 - 快手关键词搜索只能搜到220条

Issue - State: open - Opened by wmm1130 about 1 month ago

#519 - add ignore

Pull Request - State: closed - Opened by ntz857 about 1 month ago - 1 comment

#518 - python main.py --platform xhs --lt qrcode --type search 命令行执行时缺少库 import httpx ModuleNotFoundError: No module named 'httpx'

Issue - State: closed - Opened by dannishushu about 1 month ago - 4 comments

#517 - 知乎爬虫 Request error: {"error":{"code":101,"name":"AuthenticationError","message":"ZERR_NOT_LOGIN"}}

Issue - State: open - Opened by Once2gain about 1 month ago - 4 comments
Labels: bug

#516 - Create one

Pull Request - State: closed - Opened by JQKdiantang about 2 months ago - 1 comment

#515 - 现在有什么办法手动更新IP池吗？

Issue - State: open - Opened by cresstoo about 2 months ago - 1 comment

#514 - 运行到notes_res = await self.xhs_client.get_note_by_keyword的API请求时，RetryError

Issue - State: closed - Opened by dannishushu about 2 months ago - 2 comments

#513 - feat: xhs 非代理模式下增加随机等待间隔, db存储模式下增加存储xsec_token字段

Pull Request - State: closed - Opened by liudongkai about 2 months ago - 1 comment

#512 - 这种方式获取x-s会失效吗

Issue - State: closed - Opened by hezhenfan about 2 months ago - 1 comment

#511 - 请问为什么除了快手外，其他平台都会存储创作者信息

Issue - State: open - Opened by Linxuer about 2 months ago - 1 comment

#510 - 关于小红书的“访问链接异常”

Issue - State: closed - Opened by Mrmao233 about 2 months ago - 1 comment

#509 - feat: xhs comments add xsec_token

Pull Request - State: closed - Opened by leantli about 2 months ago - 1 comment

#508 - 小红书爬取访问链接异常

Issue - State: closed - Opened by AJCKBB6324 about 2 months ago
Labels: 风控问题不处理

#507 - 小红书的xsec_token如何获取

Issue - State: closed - Opened by xzy222 about 2 months ago
Labels: invalid

#506 - fix：微博根据creator爬取note时，爬取评论失败。原因是解析的参数key有误

Pull Request - State: closed - Opened by JianxunRao about 2 months ago - 1 comment

#505 - 小红书详情获取不了评论了，报 “访问链接异常” 了

Issue - State: closed - Opened by comdotwang162 about 2 months ago - 7 comments
Labels: 风控问题不处理

#504 - 爬取指定微博账号的信息出错

Issue - State: closed - Opened by JianxunRao about 2 months ago - 3 comments
Labels: question

#503 - 请问大家快手爬取评论的时候，一直循环同样的日志信息。求助求助

Issue - State: open - Opened by DLUTSSL about 2 months ago - 1 comment
Labels: 待确认

#501 - 怎么选定时间段进行微博数据的爬取

Issue - State: open - Opened by Application28 about 2 months ago - 3 comments
Labels: enhancement

#500 - playwright报错 Cannot read properties of undefined (reading '2')：爬取dy数据，打开自动控制的chrome数秒后自动关闭报错。

Issue - State: closed - Opened by Isaac-Jin about 2 months ago - 1 comment
Labels: question

#499 - Help for config

Issue - State: closed - Opened by ChloeC857 2 months ago - 2 comments

#498 - 快手 creator模式爬一下就会connect timeout

Issue - State: open - Opened by Ja5onYng 2 months ago - 1 comment
Labels: 待确认

#497 - raise ProgramError(value) execjs._exceptions.ProgramError: SyntaxError: 缺少 ')'

Issue - State: closed - Opened by chaseyoe 2 months ago - 1 comment

#497 - raise ProgramError(value) execjs._exceptions.ProgramError: SyntaxError: 缺少 ')'

Issue - State: open - Opened by chaseyoe 2 months ago - 1 comment

#496 - 这个代理ip还能使用吗按照手册中的流程好像没能成功，是否需要其他操作

Issue - State: closed - Opened by song9910moon 2 months ago - 1 comment

#495 - 为什么爬取抖音的评论没有点赞数这一列？小红书评论是有的，抖音能不能也加上？

Issue - State: closed - Opened by tengzhangtz 2 months ago - 1 comment

#494 - 爬取结构后保存为csv了，请问time和last update的时间数据如何换算？

Issue - State: closed - Opened by CaptainChristmas 2 months ago - 1 comment

#493 - 小红书排序排序方式选择

Issue - State: closed - Opened by CaptainChristmas 2 months ago

#492 - WeiBo爬取只能获取部分内容，无法get到全文

Issue - State: open - Opened by VVogelimkafig 2 months ago - 3 comments
Labels: enhancement

#491 - 小红书搜索返回406

Issue - State: closed - Opened by jimmycmh 3 months ago - 2 comments
Labels: 风控问题不处理

#490 - 为什么微博爬取，只能爬取到120页左右大概1200条数据，后面的就爬取不到了，返回空值？

Issue - State: closed - Opened by LoVei-L 3 months ago
Labels: 风控问题不处理

#489 - 抖音数据爬取，账号被风控

Issue - State: closed - Opened by Juice113 3 months ago - 2 comments
Labels: 风控问题不处理

#488 - 访问链接异常

Issue - State: closed - Opened by ganjunjie1995 3 months ago - 4 comments

#487 - 验证码不转跳

Issue - State: open - Opened by MandyHenderson 3 months ago - 17 comments

#486 - 知乎不能爬取某个问题下的所有回答吗？

Issue - State: closed - Opened by bxdxstu 3 months ago - 1 comment

#485 - 小红书不讲武德，兄弟们小心啊

Issue - State: closed - Opened by GakkiNo1 3 months ago - 1 comment

#484 - 微博Cookie只有一周的有效期，有没有什么方法可以自动更新cookie呢

Issue - State: closed - Opened by wei-mei 3 months ago - 1 comment

#483 - dy今日无法根据creator(创作者主页数据)抓取数据，用search(关键词搜索)可以抓取

Issue - State: open - Opened by Dabiaoge111 3 months ago

#482 - 请问在哪里初始化快代理的示例呢？在那个模块哪个文件下

Issue - State: closed - Opened by Dabiaoge111 3 months ago - 1 comment

#481 - 爬取到的视频能存储为mp4格式吗，想把它下载下来，目前来看只有一个链接

Issue - State: open - Opened by song9910moon 3 months ago - 4 comments

#480 - 能支持 reelshort 吗？

Issue - State: closed - Opened by jearton 3 months ago - 1 comment

#480 - 能支持 reelshort 吗？

Issue - State: closed - Opened by jearton 3 months ago - 1 comment

#479 - fix: xhs出现验证码时报错，增加用户手动验证

Pull Request - State: closed - Opened by lyx0727 3 months ago - 6 comments

#478 - 爬取抖音的时候爬一段时间就会出现account blocked

Issue - State: open - Opened by Ja5onYng 3 months ago - 5 comments

#478 - 爬取抖音的时候爬一段时间就会出现account blocked

Issue - State: closed - Opened by Ja5onYng 3 months ago - 5 comments
Labels: 风控问题不处理

#477 - fix: xhs出现验证码时报错，增加用户手动验证

Pull Request - State: closed - Opened by lyx0727 3 months ago - 2 comments

#477 - fix: xhs出现验证码时报错，增加用户手动验证

Pull Request - State: closed - Opened by lyx0727 3 months ago - 2 comments

#476 - 微博根据关键词，开启了评论，但不成功

Issue - State: open - Opened by JonL01 3 months ago - 1 comment
Labels: 待确认

#476 - 微博根据关键词，开启了评论，但不成功

Issue - State: open - Opened by JonL01 3 months ago - 1 comment
Labels: 待确认

#475 - 抖音get_aweme_detail显示request params incrr，但是视频的评论成功抓取了。

Issue - State: open - Opened by kayzh324 3 months ago - 3 comments
Labels: 待确认

#475 - 抖音get_aweme_detail显示request params incrr，但是视频的评论成功抓取了。

Issue - State: open - Opened by kayzh324 3 months ago - 4 comments
Labels: 待确认

#474 - 安装依赖库失败

Issue - State: closed - Opened by fzlgogoup 3 months ago - 3 comments

#473 - 哈咯，之前代码视频和评论是分开爬的吧，如何只能爬取视频，暂时不需要评论

Issue - State: open - Opened by RICKDONG123 3 months ago - 1 comment

#473 - 哈咯，之前代码视频和评论是分开爬的吧，如何只能爬取视频，暂时不需要评论

Issue - State: open - Opened by RICKDONG123 3 months ago - 1 comment

#472 - 小红书一直搜不出来结果乐

Issue - State: open - Opened by zhangliang0825 3 months ago

#471 - 请问爬取微博评论能够按照特定话题来爬取吗

Issue - State: closed - Opened by Huberyky 3 months ago - 2 comments

#471 - 请问爬取微博评论能够按照特定话题来爬取吗

Issue - State: closed - Opened by Huberyky 3 months ago - 2 comments

#470 - 小红书会出现获取失败或者461报错

Issue - State: closed - Opened by MandyHenderson 3 months ago - 6 comments

#470 - 小红书会出现获取失败或者461报错

Issue - State: closed - Opened by MandyHenderson 3 months ago - 6 comments

GitHub / NanmiCoder/MediaCrawler issues and pull requests