Pandas Datareader with Morningstar

I'm web scraping with pandas and using morningstar's API. Every now and then I scrape a website with a ticker that morningstar struggles with and crashes python.

I've included relevant lines below and it almost always work and I don't think my code is causing the errors but I can't get around pandas freezing up on me. Date is a variable here that is in the correct format for DataReader, it doesn't raise problems.

import pandas as pd
import pandas_datareader.data as web

df = web.DataReader(ticker, "morningstar", date)


If a ticker is getting stuck, after a couple minutes pandas will print (on it's own, not my print statement) the phrase "adding (insert ticker here) to retry list". Shortly after the message pops up I get a pop up window saying "Python quit unexpectedly".

I've tried to wrap the datareader in a while loop with a timer to stop it and move on to the next ticker but it didn't work. I'm assuming this is because of control flow.
I've also tried to use the interruptingcow module in hopes it could help but got the same result as a regular timer.

Is there a way to get around this? It crashes my script every time.
Some tickers I've come across that are a pain are 'BBXTB', 'CRXPF', 'IMMVD', and 'FFRMF'

1 Answer
1

I still do not know exactly why the tickers would fail and am investigating. however, I was able to allow the failure to not crash the scripts by directly modifying pandas_datareadermstardaily.py.

pandas_datareadermstardaily.py

My version is 0.6.0. The instructions are for this version only.

0.6.0

lines 134 - 138 looks like this:

if len(failed) > 0 and self.retry_count > 0:
# TODO: This appears to do nothing since
# TODO: successful symbols are not added to
self._dl_mult_symbols(symbols=failed)
self.retry_count -= 1


the counter is in the wrong place and needs to be swapped with the above line, as such:

if len(failed) > 0 and self.retry_count > 0:
# TODO: This appears to do nothing since
# TODO: successful symbols are not added to
self.retry_count -= 1
self._dl_mult_symbols(symbols=failed)


this should allow the failed call to exit gracefully without causing a stack overflow.

incidentally, https://github.com/pydata/pandas-datareader/blob/master/pandas_datareader/mstar/daily.py shows that @dtemkin has directly fixed the current build of pandas_datareader so this may be released/resolved on next release.

You can find this file by locating where python stores its libraries. I installed anaconda through Visual Studio 2017, and the location of the file is at: C:Program Files (x86)Microsoft Visual StudioSharedAnaconda3_64libsite-packagespandas_datareadermstardaily.py

C:Program Files (x86)Microsoft Visual StudioSharedAnaconda3_64libsite-packagespandas_datareadermstardaily.py

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.