36 comments

1. I am trying to work on these scripts and using ubuntu 15.04. These scripts requried many modules. Is there list of modules and place to get them. Please suggest

   Reply

   1. Hi Zealny, thanks for the feedback. For each of the sub module, I usually include the modules and links required at each individual posts. You can also install them using pip. However, I will try to write a separate post on the list of modules required as I know it can be confusing.

      As for now, the key modules (link found in posts) are as below:
      1. Pandas (data frame and data tables)
      2. Pattern (web related)
      3. simplejson (json handling)

      There are some additional modules that are used for more specific modules such as
      1. scipy (regression and other scientific function)
      2. matplotlib (plotting)
      3. difflib (string comparison)
      4. pypushbullet (notification)

      Below are two modules that can be found in my github (spidezad):
      1. Dict_create_fr_text (get dict fr text)
      2. xls_table_extract_module (get table fr excel)

      Note that the main module “yahoo_finance_data_extract” require Excel (Windows) to extract certain paramters. This can be disabled if not running on window system. You can email me if you need help on this. Thanks.

      Reply

      1. Hello Kok Hua,

         Thank you for helping me.
         I installed all modules except difflib. I could not find it online.

         I am having problem as follows
         >python Stock_Combine_info_gathering.py
         Unable to use the GUI function.
         Traceback (most recent call last):
         File “Stock_Combine_info_gathering.py”, line 40, in
         from SGX_stock_announcement_extract import SGXDataExtract
         File “SGX_stock_announcement_extract.py”, line 49, in
         from xls_table_extract_module import XlsExtractor
         ImportError: No module named xls_table_extract_module

         Help me.

      2. Hi zealny, not sure if difflib is available in standard python 2.7 library, Can you try to just import it? If not, you may need to use pip install.
         xls_table_extract_module can be found in Github as followed: https://github.com/spidezad/excel_table_extract. You also need pyExcel which is available in Github as well.

      3. Hi Kok Hua,

         How do I disable it so that I can use this in OSX?

         Thank you.
         Jessie

      4. Hi Jessie,

         You can comment out the module (xls extractor). The module mainly for extracting list of stock symbols. You can replace the lines where it call the module function with appropriate list.

         You can refer to my Reply to Alberto on Mar 5, 2016 for more details. Just scrolled further down to it.

         Hope it helps.

2. Hello Kok Hua,

   Thank you for help.

   I found documents for difflib module but not the source code
   https://docs.python.org/2/library/difflib.html#a-command-line-interface-to-difflib

   pyExcel.py need win32com.client.dynamic module. I think I cant make it work on linux systems.

   Reply

   1. Hi Zealny, I think you can straight away import the difflib library, think it is already available in the std library.

      As for the pyExcel, it is just provide a more convenient way for me to set the parameters that I need for the data retrieval. You can bypass it by setting it directly in yahoo_finance_data_extract module as below;
      self.enable_form_properties_fr_exceltable = 0 # set to zero
      self.cur_quotes_property_str = ‘nsl1opvkj’ #default list of properties to copy. can set the properties here.

      The list of properties can be found in the below url link:
      https://code.google.com/p/yahoo-finance-managed/wiki/enumQuoteProperty

      Hope that helps

      Reply

3. I have been trying to do something like this. Wow, there are so much details in all your posts. Have to say Thanks very much!

   Reply

   1. Thank you for your comments. Glad it is helpful to you 🙂

      Reply

4. Hello Kok Hua,

   Thanks for the great work, this code is amazing!

   I’m currently trying to run the combined run script Stock_Combine_info_gathering.py, however I’m getting the following error:

   File “Stock_Combine_info_gathering.py”, line 40, in
   from SGX_stock_announcement_extract import SGXDataExtract
   File “/Users/Alberto/Desktop/yahoo_finance_data_extract-master/SGX_stock_announcement_extract.py”, line 49, in
   from xls_table_extract_module import XlsExtractor
   File “/Users/Alberto/Desktop/yahoo_finance_data_extract-master/xls_table_extract_module.py”, line 32, in
   from pyExcel import UseExcel
   File “/Users/Alberto/Desktop/yahoo_finance_data_extract-master/pyExcel.py”, line 126, in
   import win32com.client.dynamic
   ImportError: No module named win32com.client.dynamic

   I’m using a Mac… I read in the comments section that you have a way to avoid this error when you are not using windows. Can you please help me?

   Thanks in advance!

   Reply

   1. Hi Alberto,

      Thanks for your comments and feedback. I mainly use the xlsExtractor to retrieve certain settings such as company list or parameter list which can be easily replaced by setting it to be a list. You can remove all instance of xlsExtractor and replace them by a list of the required parameters. For example,

      Under SGX_stock_announcement_extract.py (Line 118):
      ## target stocks for announcements — using excel query
      xls_set_class = XlsExtractor(fname = r’C:\data\stockselection_for_sgx.xls’, sheetname= ‘stockselection’,
      param_start_key = ‘stock//’, param_end_key = ‘stock_end//’,
      header_key = ‘header#2//’, col_len = 2)
      xls_set_class.open_excel_and_process_block_data()
      self.announce_watchlist = xls_set_class.data_label_list #also get the company name

      You can comment all and replace the self.announce_watchlist with a list of stock symbol, replace the self.companyname_watchlist with list of corresponding company name.

      self.announce_watchlist = [‘O5RU’, ‘A17U’, ‘B20’]
      self.companyname_watchlist = [‘AIMSAMP Cap Reit’, ‘Ascendas Reit’, ‘Biosensors’]

      Hope that helps.

      Reply

5. I’m getting an error and having a lot of trouble figuring out the problem:

   Traceback (most recent call last):
   File “Stock_Combine_info_gathering.py”, line 34, in
   from Basic_data_filter import InfoBasicFilter
   File “/home/corncob/Projects/Stocks/yahoo_finance_data_extract-master/Basic_data_filter.py”, line 46, in
   from DictParser.Dict_create_fr_text import DictParser
   ImportError: No module named DictParser.Dict_create_fr_text

   I installed the DictParser module using “python setup.py install” and checked the dist-package directory, all of the files seem to be present. Any help you can offer is greatly appreciated.

   Some amplifying information:
   Currently using Ubuntu 14.04, Python 2.7.6

   Reply

   1. Hi Jake, are you able to find this module DictParser folder in the python site-packages directory? If yes, you might be missing an __init__.py. Can you try create an empty file and rename is as __init__.py? See if it works.

      Reply

6. Thanks Kok Hua!

   Reply

   1. You welcome.

      Reply

7. Hi, first of all thanks for this great work !
   I am new to python and trying to run your scripts.
   Which version of python are you using for the scripts ?
   I have different problem when running using different version of python.
   Thanks

   Reply

   1. Hi Wai Kun, thank you for your good feedback. I am using python 2.7. Hope that helps.

      Reply

      1. I am now having problem on the pyExcel part.
         I installed the pyexcel using pip, and the pyexcel from your github too.

         however I got this error :
         File “c:\yahoo_finance\SGX_stock_announcement_extract.py”, line 49, in
         from xls_table_extract_module import XlsExtractor
         File “C:\Python27\lib\site-packages\xls_table_extract_module.py”, line 36, in
         from pyET_tools.pyExcel import UseExcel
         ImportError: No module named pyET_tools.pyExcel

         Can you help?

      2. Hi Wai Kun, can you change the line to: from pyExcel import UseExcel and see if it works? Thanks.

      3. Hi Kok Hua, manage to make it pass the last error.
         Now I stuck at the stockselection_for_sgx.xls file.
         Can you share the format of this xls file, so that i can create my own list.
         Thanks

      4. Hi Wai Kun,

         I have added the file to the git hub. https://github.com/spidezad/yahoo_finance_data_extract. Hope it helps.

8. Hi, great work on this code. I am getting an error “ImportError: cannot import name pyExcel” but I have already installed pyExcel (from Git). Also, there are some references to pyET_tools which does not exist. Can you help?

   Reply

   1. Hi, are you able to find this module PyExcel folder in the python site-packages directory? If yes, you might be missing an __init__.py. Can you try create an empty file and rename is as __init__.py? See if it works.

      Reply

9. Hey kok Hua, I am getting an issue where there is no module named DictParser. I read a previous comment someone else was struggling with it. I downloaded dict Parser module and ran the setup.py install. All ran correctly but it still wont import. After checking out the site-packages folder i have : DictParser-1.0-py2.7.egg-info(file not a folder), Dict_create_fr_text.py all inside just site-packages , there is no DictParser Folder. so it doesnt look right at all. How do i fix this?

   Reply

   1. Hi geoff, sorry to hear about the issue. Are you able to find this module DictParser folder in the python site-packages directory? If yes, you might be missing an __init__.py. Can you try create an empty file and rename is as __init__.py? See if it works. If there is no DictParser Folder, please create one in site-packages and copy both the __init__.py and the Dict_create_fr_text.py to the folder. Hope that helps. Please let me know if that is not working for you.Thanks

      Reply

10. Hello, is there a get started instruction on how to use it?

    thanks

    Reply

    1. as i keep getting” name ‘YFinanceDataExtr’ is not defined” error even though I already imported

       Reply

       1. Hi IKEL, regarding your question on the get started instruction,unfortunately I do not have an instruction on hand. I will try to write one in near future. For now, perhaps you can start with the Stock_Combine_info_gathering.py which will run the scripts. There is an option to run different parts of the module in line 85.
          partial_run = [‘a2′,’b’,’c_pre’,’c’,’d_pre’,’e’,’f’, ‘g’]#e is storing data

          For a simple running version, you can try the other script which is the simpler to use
          https://github.com/spidezad/google_screener_data_extract.

          Hope that helps

    2. Hi IKEL, regarding the error on the ‘YFinanceDataExtr not defined, can you check the yahoo_finance_data_extract.py file is in the same directory as the script (Stock_Combine_info_gathering.py)you running? It should work if they are all under same directory.

       Let me know if you still have problems running.

       Reply

    3. I have some “get blown out” instruction…
       Try Running It!
       I was not able to run it either.
       No DictParser & other stuff.

       self.FeelingStupid = True

       Reply

       1. Hi Avraham, most of the yahoo stock API no longer working so you might have trouble running this. Let me see if I can come up with a new post for a working version.

11. Hello,

    First of all, great job!

    Secondly, I’m having the following error while compiling it in anaconda navigator environment:

    File “Stock_Combine_info_gathering.py”, line 82
    print time.ctime()
    ^
    SyntaxError: invalid syntax

    Reply

    1. Hi Slasanto, thank you for your compliment. 🙂 For the anaconda environment, are you using Python 3.x? If yes, then the scripts will not work as they are based on python 2.x.

       Reply

       1. It worked. Thank you. Now I’m having troubles with the DictParser. I installed but I can’t see any package named “DictParser” or something similar. What I do see are the following files:
          “DictParser-1.0-py2.7.egg-info
          Dict_create_fr_text.py
          Dict_create_fr_text.pyc”

          All three are in the anaconda site-packages, more detailed:
          /Users/user-name/anaconda/envs/py27/lib/python2.7/site-packages

       2. Hi Slasanto, are you able to find this module DictParser folder in the python site-packages directory? If yes, you might be missing an __init__.py. Can you try create an empty file and rename is as __init__.py? See if it works.

Leave a Reply Cancel reply

Enter your comment here...

Fill in your details below or click an icon to log in:

Email (required) (Address never made public)

Name (required)

Website

You are commenting using your WordPress.com account. ( Log Out /  Change )

You are commenting using your Facebook account. ( Log Out /  Change )

Cancel

Connecting to %s

Notify me of new comments via email.

Notify me of new posts via email.

Δ