Soccerdata - Efficiently scrape soccer data from various sources

Which Python version are you using?

Python 3.8.13

Which version of soccerdata are you using?

1.0.1

What did you do?

fbref = sd.FBref(leagues="NED-Eredivisie", seasons="2021-2022", proxy='tor') team_season_stats = fbref.read_schedule()

What did you expect to see?

Downloaded team stats

What did you see instead?

requests.exceptions.HTTPError: 403
Client Error: Forbidden for url:
https://fbref.com/en/comps/

• Which Python version are you using?
• Which version of soccerdata are you using?

import soccerdata as sd
import sys
print(sd.__version__)
print(sys.version)

0.0.2
3.7.11 (default, Jul 27 2021, 09:42:29) [MSC v.1916 64 bit (AMD64)]

• What did you do?
• What did you expect to see?
• What did you see instead?

I tried to set use_tor=True for downloading events for a match with tor running in the background, but read_events ended with an error indicating that the proxy connection failed.

ws = sd.WhoScored(leagues="ENG-Premier League", seasons="20-21", use_tor=True)
events = ws.read_events(match_id=1485185)

[03/19/22 09:54:01] INFO     Saving cached data to                              [_common.py](file:///C:/Users/antho/anaconda3/envs/soccerdata/lib/site-packages/soccerdata/_common.py):[59](file:///C:/Users/antho/anaconda3/envs/soccerdata/lib/site-packages/soccerdata/_common.py#59)
                             C:\Users\antho\soccerdata\data\WhoScored                        
[03/19/22 09:54:04] INFO     Retrieving game schedule of ENG-Premier League  [whoscored.py](file:///C:/Users/antho/anaconda3/envs/soccerdata/lib/site-packages/soccerdata/whoscored.py):[314](file:///C:/Users/antho/anaconda3/envs/soccerdata/lib/site-packages/soccerdata/whoscored.py#314)
                             - 2021 from the cache                                           
                    INFO     [2/1] Retrieving game with id=1485185           [whoscored.py](file:///C:/Users/antho/anaconda3/envs/soccerdata/lib/site-packages/soccerdata/whoscored.py):[499](file:///C:/Users/antho/anaconda3/envs/soccerdata/lib/site-packages/soccerdata/whoscored.py#499)
                    INFO     Scraping                                        [whoscored.py](file:///C:/Users/antho/anaconda3/envs/soccerdata/lib/site-packages/soccerdata/whoscored.py):[577](file:///C:/Users/antho/anaconda3/envs/soccerdata/lib/site-packages/soccerdata/whoscored.py#577)
                             https://www.whoscored.com/Matches/1485185/Live                  
---------------------------------------------------------------------------
WebDriverException                        Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_27592\4024154899.py in <module>
      1 ws = sd.WhoScored(leagues="ENG-Premier League", seasons="20-21", use_tor=True, path_to_browser="c:/users/antho/downloads/chromedriver.exe")
----> 2 events = ws.read_events(match_id=1485185)

~\anaconda3\envs\soccerdata\lib\site-packages\soccerdata\whoscored.py in read_events(self, match_id, force_cache, live)
    507                 filepath,
    508                 var="requirejs.s.contexts._.config.config.params.args.matchCentreData",
--> 509                 no_cache=live,
    510             )
    511             json_data = json.load(reader)

~\anaconda3\envs\soccerdata\lib\site-packages\soccerdata\whoscored.py in _download_and_save(self, url, filepath, max_age, no_cache, var)
    576         if cache_invalid or filepath is None or not filepath.exists():
    577             logger.info("Scraping %s", url)
--> 578             self.driver.get(url)
    579             time.sleep(5 + random.random() * 5)
    580             if "Incapsula incident ID" in self.driver.page_source:

~\anaconda3\envs\soccerdata\lib\site-packages\undetected_chromedriver\__init__.py in get_wrapped(*args, **kwargs)
    495                     },
    496                 )
--> 497             return orig_get(*args, **kwargs)
    498 
    499         self.get = get_wrapped

~\anaconda3\envs\soccerdata\lib\site-packages\undetected_chromedriver\__init__.py in get(self, url)
    533         if self._get_cdc_props():
    534             self._hook_remove_cdc_props()
--> 535         return super().get(url)
    536 
    537     def add_cdp_listener(self, event_name, callback):

~\anaconda3\envs\soccerdata\lib\site-packages\selenium\webdriver\remote\webdriver.py in get(self, url)
    435         Loads a web page in the current browser session.
    436         """
--> 437         self.execute(Command.GET, {'url': url})
    438 
    439     @property

~\anaconda3\envs\soccerdata\lib\site-packages\selenium\webdriver\remote\webdriver.py in execute(self, driver_command, params)
    423         response = self.command_executor.execute(driver_command, params)
    424         if response:
--> 425             self.error_handler.check_response(response)
    426             response['value'] = self._unwrap_value(
    427                 response.get('value', None))

~\anaconda3\envs\soccerdata\lib\site-packages\selenium\webdriver\remote\errorhandler.py in check_response(self, response)
    245                 alert_text = value['alert'].get('text')
    246             raise exception_class(message, screen, stacktrace, alert_text)  # type: ignore[call-arg]  # mypy is not smart enough here
--> 247         raise exception_class(message, screen, stacktrace)
    248 
    249     def _value_or_default(self, obj: Mapping[_KT, _VT], key: _KT, default: _VT) -> _VT:

WebDriverException: Message: unknown error: net::ERR_PROXY_CONNECTION_FAILED
  (Session info: headless chrome=99.0.4844.74)
Stacktrace:
Backtrace:
	Ordinal0 [0x00509943+2595139]
	Ordinal0 [0x0049C9F1+2148849]
	Ordinal0 [0x00394528+1066280]
	Ordinal0 [0x00390DB4+1052084]
	Ordinal0 [0x003863BD+1008573]
	Ordinal0 [0x00386F7C+1011580]
	Ordinal0 [0x003865CA+1009098]
	Ordinal0 [0x00385BC6+1006534]
	Ordinal0 [0x00384AD0+1002192]
	Ordinal0 [0x00384FAD+1003437]
	Ordinal0 [0x00395C4A+1072202]
	Ordinal0 [0x003EC19D+1425821]
	Ordinal0 [0x003DB9EC+1358316]
	Ordinal0 [0x003EBAF2+1424114]
	Ordinal0 [0x003DB806+1357830]
	Ordinal0 [0x003B6086+1204358]
	Ordinal0 [0x003B6F96+1208214]
	GetHandleVerifier [0x006AB232+1658114]
	GetHandleVerifier [0x0076312C+2411516]
	GetHandleVerifier [0x0059F261+560433]
	GetHandleVerifier [0x0059E366+556598]
	Ordinal0 [0x004A286B+2173035]
	Ordinal0 [0x004A75F8+2192888]
	Ordinal0 [0x004A76E5+2193125]
	Ordinal0 [0x004B11FC+2232828]
	BaseThreadInitThunk [0x76106739+25]
	RtlGetFullPathName_UEx [0x76FF8E7F+1215]
	RtlGetFullPathName_UEx [0x76FF8E4D+1165]

Here's what my terminal looks like with tor running (prior to calling read_events()

[email protected]:/c/Users/antho/soccerdata$ tor

Mar 19 09:53:33.865 [notice] Tor 0.4.2.7 running on Linux with Libevent 2.1.11-stable, OpenSSL 1.1.1f, Zlib 1.2.11, Liblzma 5.2.4, and Libzstd 1.4.4.
Mar 19 09:53:33.865 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
Mar 19 09:53:33.865 [notice] Read configuration file "/etc/tor/torrc".
Mar 19 09:53:33.866 [notice] Opening Socks listener on 127.0.0.1:9050
Mar 19 09:53:33.866 [notice] Opened Socks listener on 127.0.0.1:9050
Mar 19 09:53:33.000 [notice] Parsing GEOIP IPv4 file /usr/share/tor/geoip.
Mar 19 09:53:33.000 [notice] Parsing GEOIP IPv6 file /usr/share/tor/geoip6.
Mar 19 09:53:34.000 [notice] Bootstrapped 0% (starting): Starting
Mar 19 09:53:34.000 [notice] Starting with guard context "default"
Mar 19 09:53:35.000 [notice] Bootstrapped 5% (conn): Connecting to a relay
Mar 19 09:53:35.000 [notice] Bootstrapped 10% (conn_done): Connected to a relay
Mar 19 09:53:35.000 [notice] Bootstrapped 14% (handshake): Handshaking with a relay
Mar 19 09:53:35.000 [notice] Bootstrapped 15% (handshake_done): Handshake with a relay done
Mar 19 09:53:35.000 [notice] Bootstrapped 75% (enough_dirinfo): Loaded enough directory info to build circuits        
Mar 19 09:53:35.000 [notice] Bootstrapped 90% (ap_handshake_done): Handshake finished with a relay to build circuits  
Mar 19 09:53:35.000 [notice] Bootstrapped 95% (circuit_create): Establishing a Tor circuit
Mar 19 09:53:36.000 [notice] Bootstrapped 100% (done): Done

I've opened my browser to the port to verify that something is running, although this is using an HTTP proxy, so the warning here is expected.

Hi @probberechts - this looks like a wonderful set of tools. Can't wait to get stuck deeper into it. Thank you!

Objective: To be able to scrape FBRef stats for historic World Cups (and upcoming 2022 World Cup) from this page

World Cup stats landing page -> https://fbref.com/en/comps/1/World-Cup-Stats Stats page for 2018 World Cup -> https://fbref.com/en/comps/1/2018/2018-FIFA-World-Cup-Stats

1. Adding a new league - Working as expected

In the "Adding additional leagues" (here: https://soccerdata.readthedocs.io/en/latest/usage.html) I successfully added a new league called "INTL-WorldCup"

Content of league_dict.json

{
  "INTL-WorldCup": {
    "FBref": "World-Cup-Stats",
    "season_start": "Aug",
    "season_end": "May"
  }
}

Note: I had to remove a comma from just after the 2nd last curly bracket.

Result: When I sd.FBref.available_leagues() it returns the expected result below

[
  'Big 5 European Leagues Combined',
  'ENG-Premier League',
  'ESP-La Liga',
  'FRA-Ligue 1',
  'GER-Bundesliga',
  'INTL-WorldCup',
  'ITA-Serie A'
]

2. Can I pull back scraped data?

This line ran without error: fbref = sd.FBref(leagues="INTL-WorldCup", seasons=2018)

However, when I ran the 2 lines below

team_season_stats = fbref.read_team_season_stats(stat_type="standard")
team_season_stats.head()

...I got this error below. What am I doing wrong?

ValueError                                Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_11984/128004415.py in <module>
----> 1 team_season_stats = fbref.read_team_season_stats(stat_type="standard")
      2 team_season_stats.head()

soccerdata\fbref.py in read_team_season_stats(self, stat_type, opponent_stats)
    252 
    253         # get league IDs
--> 254         seasons = self.read_seasons()
    255 
    256         # collect teams

soccerdata\fbref.py in read_seasons(self)
    169             seasons.append(df_table)
    170 
--> 171         df = pd.concat(seasons).pipe(standardize_colnames)
    172         # A competition name field is not inlcuded in the Big 5 European Leagues Combined
    173         if "competition_name" in df.columns:

~\Miniconda3\lib\site-packages\pandas\util\_decorators.py in wrapper(*args, **kwargs)
    309                     stacklevel=stacklevel,
    310                 )
--> 311             return func(*args, **kwargs)
    312 
    313         return wrapper

~\Miniconda3\lib\site-packages\pandas\core\reshape\concat.py in concat(objs, axis, join, ignore_index, keys, levels, names, verify_integrity, sort, copy)
    302         verify_integrity=verify_integrity,
    303         copy=copy,
--> 304         sort=sort,
    305     )
    306 

~\Miniconda3\lib\site-packages\pandas\core\reshape\concat.py in __init__(self, objs, axis, join, keys, levels, names, ignore_index, verify_integrity, copy, sort)
    349 
    350         if len(objs) == 0:
--> 351             raise ValueError("No objects to concatenate")
    352 
    353         if keys is None:

ValueError: No objects to concatenate

Hi! First of all, thank you for the code!

I'd like to ask if it would be possible to also get stats from the "Opponent Stats" table, Thanks!

I am having another go at this [previous attempt #69] because you have updated the FBRef class to use the league pages. I have tried this against all stat_types for 2020-2021 and it seems to work

• Amended the FBRef scraper so it uses the Big 5 pages if all five leagues are requested.
• Added some type checks for the stats_type argument.

I am not able to run the tests locally, but I'll try to fix anything that doesn't work after.

(seen on Windows 11)

The Tor port specified in the code for when initializing a scraper with the proxy="tor" option is 9050, whereas the new Tor versions seem to use port 9150.

Something should be done to check whether port 9050 works in the first place and if it doesn't check with 9150.

For other people it doesn't work for right now, you can always do this :

return_proxies = lambda: {
    "http": "socks5://127.0.0.1:9150",
    "https": "socks5://127.0.0.1:9150",
}

ws = sd.WhoScored(leagues="ENG-Premier League", seasons=20-21, proxy=return_proxies)

if you run:

import soccerdata as sd
fbref = sd.FBref(leagues="ENG-Premier League", seasons=2021)
print(fbref.__doc__)

epl_schedule = fbref.read_schedule()

You will get an error

frame.py 3832 _set_item value = self._sanitize_column(value)

frame.py 4535 _sanitize_column com.require_length_match(value, self.index)

common.py 557 require_length_match raise ValueError(

ValueError: Length of values (0) does not match length of index (31)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | Sphinx (source) | ^4.3.2 -> ^5.0.0 | | | | | | sphinx (source) | ==4.5.0 -> ==5.0.2 | | | | |

Release Notes

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about these updates again.

• [ ] If you want to rebase/retry this PR, click this checkbox.

This PR has been generated by Mend Renovate. View repository job log here.

FBRef read_player_season_stats method does not seems to be considering the seasons parameter. No matter what the season is mentioned inside the brackets it just retrevied the recent season player stats. 2021-2022.

Code used is mentioned below

import soccerdata as sd

fbref = sd.FBref(no_cache=False, no_store=False, leagues="ENG-Premier League", seasons='11-12')
pl_player_season_stats = fbref.read_player_season_stats(stat_type='standard')
pl_player_season_stats .head()

Seasons parameter works perfectly with the read_team_season_stats method though.

When using the API to scrape from FBRef I noticed something odd.

The APIs for each category of stats (e.g. shooting, passing) don't seem to be working off the same baseline of minutes played.

For example: take Emiliano Martínez (Argentina). The "90s" value should be consistent across each of these categories - but across the 5 categories below it varies extensively.

fbref.read_player_season_stats(stat_type="standard") = 2.0 fbref.read_player_season_stats(stat_type="shooting") = 2.0 fbref.read_player_season_stats(stat_type="goal_shot_creation") = 3.0 fbref.read_player_season_stats(stat_type="passing") = 3.0 fbref.read_player_season_stats(stat_type="defense") = 5.3

Note: This didn't seem to be an issue earlier in the tournament. Also, I only recently scraped "defense" statistics for the first time, so I wonder if somehow the results of old queries are being cached?

Also, when I navigate to the specific pages in by browser the data looks to be consistent & fully up to-date (i.e. a value of 6.3 for the "90s" attribute for Emiliano Martínez)

https://fbref.com/en/comps/1/passing/World-Cup-Stats

Hello,

I have found a small bug when pulling data from FBRef.com. NaN values appearing in the MP columns in the data for stat_types standard and playing_time for players who have played in the season.

I found this problem after I wrote a function to obtain multiple stat_types for multiple seasons and converted the DataFrames from a multiindex to a standard pandas DataFrame. I found a large quantity of NaNs due to this transformation. 

To troubleshoot, I did a single pull using the .read_player_season_stats(stat_type = 'standard') call on 2 seasons of data (1718 & 1819) and found NaN values in both the MP and Playing Time MP columns. Players who played and did not play had received NaN values in the aforementioned columns.  Under the "Playing Time" section's MP column, I found 890 NaN values and in the standalone 'MP' column, I found 380 NaN values.  I am transitioning from R to Python and have always used the flattened-style DataFrame in the past.

Attached is a csv file containing the aforementioned data.

Call:

fbref_test = sd.FBref(leagues=['ENG-Premier League'], seasons= ['1718', '1819'])

hold = fbref_test.read_player_season_stats(stat_type = 'standard')
hold.head()

I greatly appreciate your assistance. fbref_nan_bug_df.csv

Hello everyone,

If you try to scrap a game from WhoScored using the package, with a 0-0 score, a KeyError occurs. Within the file whoscored.py at line 676, the process try to access the feature 'is_goal', which seems to not exist if the score at the end of the game is 0-0.

Example of game where this issue happen : https://www.whoscored.com/Matches/1640849/Live/England-Premier-League-2022-2023-Newcastle-Leeds

Python code :

import soccerdata as sd ws = sd.WhoScored(leagues="ENG-Premier League", seasons="22-23") events = ws.read_events(match_id=1640849)

Error :

KeyError: ['is_goal'] not in index

A solution could be to check if all the necessary features are indeed part of the dataframe, and if not, add it with np.nan values. The python code could be as follow, before the line 676 :

Python code :

cols = ['event_id', 'expanded_minute', 'is_touch', 'minute', 'outcome_type', 'period', 'qualifiers', 'satisfied_events_types', 'second', 'team_id', 'type', 'x', 'y', 'end_x', 'end_y', 'player_id', 'blocked_x', 'blocked_y', 'goal_mouth_y', 'goal_mouth_z', 'is_shot', 'related_event_id', 'related_player_id', 'is_goal', 'card_type', '$idx', '$len', 'field', 'minute_info', 'satisfiers', 'text', 'game_id', 'player', 'team']

for col in cols: if col not in df.columns: df[col] = np.nan

PS: It exists a similar issue with feature 'card_type' if no card were given during the game.

Thanks, Ben

Hello,

I'm trying to pull the schedule from any league, but it keeps getting an error in the date format. Even when I input the match ID, keeps with problem to read the data because of the date format. How can I solve it? ValueError: time data 'Jumatatu, Des 26 2022 12:30' does not match format '%A, %b %d %Y %H:%M'

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | poetry (source, changelog) | ==1.2.2 -> ==1.3.1 | | | | |

Release Notes

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

• [ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Mend Renovate. View repository job log here.

After updating from version 1.2.0 to 1.3.0 data from the fbref is repeated twice The sample code

fb_data = fbref.read_player_match_stats(
                stat_type='summary', match_id=None, force_cache=False)
            fb_data.to_csv('./summary.csv')

log from version 1.3.0

log from version 1.2.0

In version 1.3.0, the log Retrieving game with id=**** is repeating twice.

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | flake8 (changelog) | ^5.0.4 -> ^6.0.0 | | | | |

Release Notes

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

• [ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Mend Renovate. View repository job log here.

Hi,

This is an amazing package! I think the docs are mostly very clear. However, is it possible to have a more in-depth tutorial on how to add new leagues to FBRef? I'm trying to add the English Championship, which is available on FB Ref, but wasn't able to. I added a league_dict.json file (with the correct config I assume) to the "SOCCERDATA_DIR/config/" file path, but it seems like the code is not picking up on it when I call fbref = sd.FBref(leagues="EFL Championship", seasons=2019). It gave me a ValueError noting "Invalid League". Thank you so much!