# Rapid input data from list of files to SQLite DB

Suppose you wish to input all the data from a series of CSV files residing in a particular folder to a SQLite database, the following tools and and commands will make it a relatively fast and painless.

Each of the CSV file are assumed to be in the format where the first line is the headers and the subsequent lines are series of data. The headers need not be the same for each CSV file.

The following python modules are required.

1. CSV module
2. Pandas (0.15)
3. SQLite3

The following lines of commands will input execute the following actions.

1. Use list comprehension to get the list of file path from a particular folder.
2. Convert the data from each CSV file into dict format using CSV DictReader and iterate over all the files, joining them to a list.
3. Convert the list of dicts to a single Pandas Dataframe.
4. Create a connection to a particular SQLite Database.
5. Use Pandas to_sql() function to pass all the data to SQLite database.
6. The code is displayed below.
<pre>import os
import csv
import pandas
import sqlite3 as lite

path = r'C:\folderpath\to\list\of\files'
## Get the full path of all the csv files.
full_path_list = [os.path.join(path,f) for\
f in os.listdir(path) if os.path.isfile(os.path.join(path,f)) ]

## Convert all data to list of dicts.
full_data_list =  [n for f in full_path_list for n in csv.DictReader(open(f,'r'))]

## SQL database name and initialize the sql connection.
db_filename = r'c:\data\sbrtemp3.db'
con = lite.connect(db_filename)

## Convert to dataframe and write to sql database.
pandas.DataFrame(full_data_list).to_sql('test', con, flavor='sqlite',
schema=None, if_exists='replace', index=True,
index_label=None, chunksize=None, dtype=None)

## Close the SQL connection
con.close()


The Pandas to_sql() function is able to handle issues of duplicates and can be called multiple times if users required to add additional data. In addition, converting to a dataframe from a list of dicts also allows the headers to be different for different CSV files. Besides saving to database, user can also choose to consolidate to a single csv file by using Pandas to_csv() function.

# Extracting stocks info from yahoo finance using python (Updates)

Have made several updates to the script from previous posting. Firstly is the capability to sweep through all the stocks symbol for a .csv file. The list of stocks symbol is easily generated using the extract all symbol script describe in the previous post. Reading all the symbols from the CSV can be done using python Pandas as shown below.

        data_ext = YFinanceDataExtr()
## read  data from .csv file -- full list of stocks
csv_fname = r'C:\pythonuserfiles\yahoo_finance_data_extract\stocklist.csv'
# convert from pandas dataframe object to list
stock_list = list(stock_list['SYMBOL'])
#stock_list = ['S58.SI','S68.SI']
data_ext.get_cur_quotes_fr_list(stock_list)


The second improvement is instead of keying all the individual properties that need to be extracted (as illustrated below), the list of properties can be read from a xls table using the xls_table_extract_module described in the following post.

original method to set the property in the url

    def form_cur_quotes_property_url_str(self):
""" To form the properties/parameters of the data to be received for current quotes
To eventually utilize the get_table_fr_xls.
Current use default parameters.
name(n0), symbol(s), the latest value(l1), open(o) and the close value of the last trading day(p)
volumn (v), year high (k), year low(j)

Further info can be found at : https://code.google.com/p/yahoo-finance-managed/wiki/enumQuoteProperty
"""
start_str = '&f='
target_properties = 'nsl1opvkj'
self.cur_quotes_property_portion_url =  start_str + target_properties


New method: xls table format. (the xls illustrated here is the simplified version). The full property xls is in Github.

The data can be retrieved easily using the xls_table_extract_module hence easily forming the properties str by concat the tag together. The information required can be customized to the order based on the order of xls and the information required can be turned on and off using the comment tag ‘#’.  Note  some of the properties retrieved might not be in format that easy to parse and might result in extra column upon downloading. The portion of script to handle this is as described below.

    def form_cur_quotes_property_url_str_fr_excel(self):
""" Required xls_table_extract_module.
Get all the properties from excel table.
Properties can be selected by comment out those properties not required.

"""
from xls_table_extract_module import XlsExtractor
self.xls_property_data = XlsExtractor(fname = self.properties_excel_table, sheetname= 'Sheet1',
param_start_key = 'stock_property//', param_end_key = 'stock_property_end//',
header_key = '', col_len = 2)

self.xls_property_data.open_excel_and_process_block_data()

self.cur_quotes_parm_headers = [n.encode() for n in self.xls_property_data.data_label_list]

## form the url str
start_str = '&f='
target_properties = ''.join([n[0].encode().strip() for n in self.xls_property_data.data_value_list])
self.cur_quotes_property_portion_url =  start_str + target_properties


The last update enable the script to handle more than one url query (each query can handle up to 50 stocks). This enable the full sweep of all the stocks listed in the stocklist and downloaded it to single results file. A sweep of around 1000 stocks symbol take less than 3 mins (it also depends on the internet connection).

The updated script can be found at GitHub.

# Extracting stocks info from yahoo finance using python

There are many ways to extract stocks information using python. A simple way to get the current stocks data can be achieved by using python Pandas. The data retrieved however are limited.

The method I use below are based on downloading the various data .csv file, a service provided by the Yahoo Finance. The method to construct the various url to download the .csv information are described in great details from the Yahoo Finance API.

The current script created can only retrieved the most current data statistics for the various stocks. First, it will construct the URL based on user stocks input and the parameters required. It then makes use of the PATTERN module to read the url and download the information to local drive. Next, it will call the pandas function to read the .csv file and convert it to data frame for further analysis.

Sample output of the script is as shown below.


data_ext = YFinanceDataExtr()

## Specify the stocks to be retrieved. Each url constuct max up to 50 stocks.
data_ext.target_stocks = ['S58.SI','S68.SI'] #special character need to be converted

## Get the url str
data_ext.form_url_str()
print data_ext.cur_quotes_full_url

## Stored the data as pandas.Dataframe.
data_ext.get_cur_quotes()
print data_ext.cur_quotes_df
## >>>   NAME  SYMBOL  LATEST_PRICE  OPEN  CLOSE      VOL  YEAR_HIGH  YEAR_LOW
## >>> 0  SATS  S58.SI          2.99  3.00   3.00  1815000       3.53      2.93
## >>> 1   SGX  S68.SI          7.18  7.19   7.18  1397000       7.63      6.66



To specify the parameters to be output, it can be changed in the following method of the script. In future, this will be refined to be more user friendly.


def form_cur_quotes_property_url_str(self):
""" To form the properties/parameters of the data to be received for current quotes
To eventually utilize the get_table_fr_xls.
Current use default parameters.
name(n0), symbol(s), the latest value(l1), open(o) and the close value of the last trading day(p)
volumn (v), year high (k), year low(j)
Further info can be found at : https://code.google.com/p/yahoo-finance-managed/wiki/enumQuoteProperty
"""
start_str = '&f='
target_properties = 'nsl1opvkj'
self.cur_quotes_property_portion_url =  start_str + target_properties