我对此很陌生，但很难理解这如何有意义/为什么似乎没有任何效果。

基本上，我想做的就是从 ncaa.com 链接列表（下面的示例）中刮取表格数据（逐个播放）

https://stats.ncaa.org/game/play_by_play/12465

https://stats.ncaa.org/game/play_by_play/12755

https://stats.ncaa.org/game/play_by_play/12640

https://stats.ncaa.org/game/play_by_play/12290

作为上下文，我通过从不同的链接列表（包含每个 NCAA 球队的比赛时间表）中抓取 HREF 标签来获得这些链接。

我已经为此苦苦挣扎了很多，但是在某处有答案...

Inspector 让 Play by Play (table) 数据看起来像是一个 tbody 标签，或者至少我认为？

我已经尝试过像这样简单的脚本（适用于其他网站）

import pandas as pd

df = pd.read_html(
    'https://stats.ncaa.org/game/play_by_play/13592')[0]
print(df)

但它仍然不适用于这个网站......我读了一些关于使用 lxml.parser 而不是 html.parser （如下面的代码）......但也没有工作 - 我认为这是我最好的机会一次从多个链接获取表格：

import requests
from bs4 import BeautifulSoup

profiles = []
urls = [
    'https://stats.ncaa.org/game/play_by_play/12564',
    'https://stats.ncaa.org/game/play_by_play/13592'

]
for url in urls:
    req = requests.get(url)
    soup = BeautifulSoup(req.text, 'lxml.parser')
    for profile in soup.find_all('a'):

        profile = profile.get('tbody')

        profiles.append(profile)

# print(profiles)

for p in profiles:
    print(p)

任何关于这个网站的独特之处/可能是什么问题的想法将不胜感激。

解决方案

该网站将检查请求是来自机器人还是浏览器，因此您需要使用真实的用户代理更新请求的标头。

每页有 8 个表格。下面的代码将遍历您上面提到的每个 url 并打印出所有表格。您可以查看它们，查看您需要哪一个，等等：

import requests
import pandas as pd

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
}

urls = [
    'https://stats.ncaa.org/game/play_by_play/12465',
'https://stats.ncaa.org/game/play_by_play/12755',
'https://stats.ncaa.org/game/play_by_play/12640',
'https://stats.ncaa.org/game/play_by_play/12290',
]
s = requests.Session()
s.headers.update(headers)
for url in urls:
    r = s.get(url)
    dfs = pd.read_html(r.text)
    len(dfs)
    for df in dfs:
        print(df)
        print('___________')

回复：

  0         1         2      3
0    NaN  1st Half  2nd Half  Total
1  UTRGV        23        19     42
2   UTEP        26        34     60
___________
             0                                    1
0   Game Date:                           01/02/2009
1    Location:  El Paso, Texas (Don Haskins Center)
2  Attendance:                                 8413
___________
            0                                          1
0  Officials:  John Higgins, Duke Edsall, Quinton, Reece
___________
          0     1
0  1st Half  1  2
___________
                   0                                       1                2  \
0               Time                                   UTRGV            Score   
1              19:45                                     NaN              0-0   
2              19:45                                     NaN              0-0   
3              19:33                                     NaN              0-0   
4              19:33        Emmanuel Jones Defensive Rebound              0-0   
..               ...                                     ...              ...   
163            00:11                                     NaN            23-25   
164            00:11                                     NaN            23-26   
165            00:00  Emmanuel Jones missed Two Point Jumper            23-26   
166            00:00                                     NaN            23-26   
167  End of 1st Half                         End of 1st Half  End of 1st Half   

                                            3  
0                                        UTEP  
1     Arnett Moultrie missed Two Point Jumper  
2              Julyan Stone Offensive Rebound  
3    Stefon Jackson missed Three Point Jumper  
4                                         NaN  
..                                        ...  
163            Stefon Jackson made Free Throw  
164            Stefon Jackson made Free Throw  
165                                       NaN  
166            Julyan Stone Defensive Rebound  
167                           End of 1st Half  

[168 rows x 4 columns]
___________
          0     1
0  2nd Half  1  2
[..]