Gtjkyu

XPATH syntax validator in Python I develop a crawler with many actions. Many xpaths are involving and for this reason I use a json file for storing. Then crawler start running I would like to make a basic syntax check (before xpath usage) on xpaths and raise error for invalid xpaths. for example: xpath1 = '//*[@id="react-root"]/section' xpath2 = '//*[[@id="react-root"]/section' xpath3 = '//*[@id="react-root"]section' from these xpaths only xpath1 is valid is there any module or regex which does this kind of validation? 1 Answer 1 You can compile the xpath strings with lxml.etree.XPath which will raise an exception if the syntax is incorrect: lxml.etree.XPath >>> import lxml.etree >>> lxml.etree.XPath('//*[@id="react-root"]/section') //*[@id="react-root"]/section >>> lxml.etree.XPath('//*...

Pandas Datareader with Morningstar I'm web scraping with pandas and using morningstar's API. Every now and then I scrape a website with a ticker that morningstar struggles with and crashes python. I've included relevant lines below and it almost always work and I don't think my code is causing the errors but I can't get around pandas freezing up on me. Date is a variable here that is in the correct format for DataReader, it doesn't raise problems. import pandas as pd import pandas_datareader.data as web df = web.DataReader(ticker, "morningstar", date) If a ticker is getting stuck, after a couple minutes pandas will print (on it's own, not my print statement) the phrase "adding (insert ticker here) to retry list". Shortly after the message pops up I get a pop up window saying "Python quit unexpectedly". I've tried to wrap the datareader in a while loop with a timer to stop it and move on to the next ticker but it didn't w...

Web scrapping of masked URL using VBA I want to scrape some stock data from a website https://dps.psx.com.pk/ using VBA in Excel, but the problem is the URL of this website does not change. When I click on market summary as highlighted in the below image that will return the whole market summary, I just need to scrape data in Excel using VBA as highlighted in the following image: Did you look at the requests being made when you search for "EFOODS"? – antfuentes87 Jul 1 at 16:30 This is basic web scraping. You need to inspect the source code elements and like @antfuentes87 says , follow the requests. It sounds more like you need to purchase a third party tool to help you. It's also called "scrape" and "scraping" – dbmitch Jul 1 at 16:44 ...