The post on the testing of google search script I created last week describe the limitations of the script to scrape the required information. The search phrase is “best hotels to stay in Tokyo”. My objective is to find suitable and popular hotels to stay in Tokyo and within the budget limit.
The other limitation is that the script can only take in one input or key phrase at one go. This is not very useful. Users would tend to search a variation of the key phrases to get the desirable results. I done some modifications to the script so it can take in either a key phrase (str) or a list of key phrases (list) so it can search all the key phrases at one go.
The script will now iterate the search phrases. Below is the summarized flow:
- For each key phrase in key phrase list, generate the associated google search url, append all url to list.
- For the list of google search url, Scrapy will scrape the individual url for the google results links. Append all links to a output file. There is one drawback. The links for the first key phrases will be displayed first followed by the 2nd key phrase.
- For each of the links, Scrapy will scrape the content namely the title, meta description and for now, if specified, all the text within the <p> tag.
- The resulting file will be very big depending on the size of the search results.
The format of the output is still not to satisfaction. Also printing all the <p> tag does not accomplished much in summarizing what I need.
The next step, hopefully, can utilize some of the NLTK and summarize tools to help filter the results.
The current script is in Git Hub.
Simply Python 