Team minutes Data Feed - nordquip/sousms GitHub Wiki

Goals for 121119:

  • Make parseFeedEnterData.php use the presence of a file to start and stop. - DONE file is called feedControl
  • Shaun - Edit enterTickerDataForSymbol -- add handler or try catch that will ignore duplicate (key) insert
  • Call Jeff's php database connection code

Goals for 121112:

Bret not present

  • Bret - Create bash script feedControl that starts and stops (with a different switch) parseFeedInsertData.php - DONE(Shaun)
  • Shaun - Create bash script feedCron to: -- DONE -- all done within the script
    1. start feed at the next appropriate time - DONE -- sets env variable
    2. enter a stop feed + a start feed for appropriate times -DONE -- sets env. variable
  • Make parseFeedInsertData.php stop if stopFeed file is present in the same directory as the script. - NOT DONE -- used environment variable instead.
  • Research how vol, BestBP, BestBQ, BestAP, BestAQ change after close, after after hours, before pre hours, before open. - NOT DONE

stretch goals:

  • make feedCron have ability for build team to call it to stop it immediately then start it again - DONE
  • Make stored procedure insert only if lastTrade and/or vol has changed. (Consider making different primary key for feed table?) - NOT DONE seems like a database related problem, new primary key?

Goals for 121105: (Edited 11/02/2012 by Shaun)

both present

  • Remove 'select * from feed' from enterTickDataForSymbol - DONE

  • make parseFeed.php use CURL to call the NASDAQ feed (test with php command line tool on server) - NOT DONE - Turns out we can pass a url address to simplexml_load_file() and it will return the feed.

  • make parseFeed.php parse the 10 symbols - DONE

  • make parseFeed.php cycle through 4 sets of 10 symbols - DONE

  • make parseFeed.php call stored procedure once for each symbol - DONE

  • make testFile.php that sleeps when trading day not in session (evenings and weekends). - NOT DONE - attempted, but times out when started by apache. Doesn't time out when started from command line.

  • Must contain a loop forever that the sleep prevents from running, then starts again. - DONE- Added sleep(1) between each loop of 40 to not overwhelm the server and get us banned.

  • Integrate this code into nordquip/sousms/src/DataFeed/parseFeed.php - DONE- Replaced parseFeed.php with parseFeedInsertData.php

Stretch goals:

  • Test how vol, bAP, bAQ, bBP, bBQ change when no trades have happened
  • Make stored procedure insert record only if last trade has 'changed' -- see notes below.

Goals for 121029:

Notes:

  • Would like to insert only if some portion of the record other than (date, time, ms) changes.
    I think vol && (bestAskPrice || bestAskQty || bestBidPrice || bestBidQty) will tell us whether a trade has happened.

Repository

Implemetation

Shaun:

  • rename interface.sql to enterTickerDataForSymbol.sql -- DONE

  • put enterTickerDataForSymbol.sql in shared directory -- DONE

  • add MYSQL types for each field to enterTickDataForSymbol() -- DONE

  • make enterTickDataForSymbol body print "insert data into database" -- NOT DONE: have an insert command

  • create parseFeed.php (from feedParseTest.php)

  • make parseFeed.php use CURL to call the database (test with php command line tool) -- NOT DONE

  • make parseFeed.php call stored procedure -- NOT DONE

  • Each person install XAMPP stack -- DONE both

  • Brett - move Shaun's changes up to nordquip/sousms -- DONE: Shaun

Notes:

  • Still need to choose the 40 stocks
  • can call php from command line to test the parsing page -- good find, Shaun
  • Feed should stop capturing when market is closed

Goals for 121022:

Design goal:

Implementation goal:

  • Create php program that will:
  1. call feed for 10 symbols - DONE (Shaun)
  2. read XML of the 10 symbols returned -- DONE (Shaun)
  3. call enterTickData() interface for each of the 10 symbols ('echo out' this call until we have the stored procedure that will implement it -- next week) -- DONE (Shaun)

Stretch goal:

  • 'Loop for 4 sets of 10 stocks each' -- DONE
  • Write storedProcedure enterTickData() -- N/A

Sample Feed data:

https://basicapp.nasdaqomx.com/BasicDataXML/getBasicData?symbolsCsvList=INTC,GOOG,MSFT,GE,ADM,GM,APPL,FB,DIS,ALU
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><BasicData><dataItems><bestAskPrice>21.77</bestAskPrice><bestAskQty>300</bestAskQty><bestBidPrice>21.73</bestBidPrice><bestBidQty>1700</bestBidQty><close>21.72</close><high>21.9</high><lastSale>21.74</lastSale><low>21.48</low><netChg>0.2597</netChg><open>21.51</open><pcl>21.4803</pcl><vol>49494727</vol><date>20121015</date><pctChg>1.209</pctChg><class>Q</class><symbol>INTC</symbol><time>17:22:16.155</time><tradingState>T</tradingState><tradingStateReason></tradingStateReason></dataItems><dataItems><bestAskPrice>740</bestAskPrice><bestAskQty>15</bestAskQty><bestBidPrice>739.5</bestBidPrice><bestBidQty>25</bestBidQty><close>740.98</close><high>748</high><lastSale>740.48</lastSale><low>730.701</low><netChg>-4.27</netChg><open>741.31</open><pcl>744.75</pcl><vol>3017768</vol><date>20121015</date><pctChg>-0.5733</pctChg><class>Q</class><symbol>GOOG</symbol><time>17:22:16.212</time><tradingState>T</tradingState><tradingStateReason></tradingStateReason></dataItems><dataItems><bestAskPrice>29.52</bestAskPrice><bestAskQty>150</bestAskQty><bestBidPrice>29.48</bestBidPrice><bestBidQty>440</bestBidQty><close>29.51</close><high>29.72</high><lastSale>29.48</lastSale><low>29.25</low><netChg>0.28</netChg><open>29.37</open><pcl>29.2</pcl><vol>42417172</vol><date>20121015</date><pctChg>0.9589</pctChg><class>Q</class><symbol>MSFT</symbol><time>17:22:16.260</time><tradingState>T</tradingState><tradingStateReason></tradingStateReason></dataItems><dataItems><bestAskPrice>22.69</bestAskPrice><bestAskQty>300</bestAskQty><bestBidPrice>22.65</bestBidPrice><bestBidQty>1000</bestBidQty><close>22.64</close><high>22.7</high><lastSale>22.63</lastSale><low>22.49</low><netChg>0.15</netChg><open></open><pcl>22.48</pcl><vol>29509927</vol><date>20121015</date><pctChg>0.6673</pctChg><class>N</class><symbol>GE</symbol><time>17:22:16.317</time><tradingState>T</tradingState><tradingStateReason></tradingStateReason></dataItems><dataItems><bestAskPrice>28.33</bestAskPrice><bestAskQty>300</bestAskQty><bestBidPrice>27.9</bestBidPrice><bestBidQty>50</bestBidQty><close>28.1</close><high>28.22</high><lastSale>28.0501</lastSale><low>27.98</low><netChg>0.0001</netChg><open>28.11</open><pcl>28.05</pcl><vol>4583457</vol><date>20121015</date><pctChg>0.0004</pctChg><class>N</class><symbol>ADM</symbol><time>17:22:16.366</time><tradingState>T</tradingState><tradingStateReason></tradingStateReason></dataItems><dataItems><bestAskPrice>29.11</bestAskPrice><bestAskQty>20</bestAskQty><bestBidPrice>24.3</bestBidPrice><bestBidQty>800</bestBidQty><close>24.465</close><high>24.58</high><lastSale>24.47</lastSale><low>24.33</low><netChg>0.03</netChg><open>24.51</open><pcl>24.44</pcl><vol>5019413</vol><date>20121015</date><pctChg>0.1227</pctChg><class>N</class><symbol>GM</symbol><time>17:22:16.422</time><tradingState>T</tradingState><tradingStateReason></tradingStateReason></dataItems><dataItems><bestAskPrice>0</bestAskPrice><bestAskQty>0</bestAskQty><bestBidPrice>0</bestBidPrice><bestBidQty>0</bestBidQty><close></close><high>0</high><lastSale>0.03</lastSale><low>0</low><netChg>0</netChg><open></open><pcl>0.03</pcl><vol>0</vol><date>20121015</date><pctChg>0</pctChg><class></class><symbol>APPL</symbol><time>17:22:16.471</time><tradingState></tradingState><tradingStateReason></tradingStateReason></dataItems><dataItems><bestAskPrice>19.55</bestAskPrice><bestAskQty>500</bestAskQty><bestBidPrice>19.52</bestBidPrice><bestBidQty>449</bestBidQty><close>19.51</close><high>19.88</high><lastSale>19.52</lastSale><low>19.49</low><netChg>0.0019</netChg><open>19.68</open><pcl>19.5181</pcl><vol>20154159</vol><date>20121015</date><pctChg>0.0097</pctChg><class>Q</class><symbol>FB</symbol><time>17:22:16.528</time><tradingState>T</tradingState><tradingStateReason></tradingStateReason></dataItems><dataItems><bestAskPrice>51.25</bestAskPrice><bestAskQty>100</bestAskQty><bestBidPrice>50.32</bestBidPrice><bestBidQty>100</bestBidQty><close>50.78</close><high>50.81</high><lastSale>50.7701</lastSale><low>50.355</low><netChg>0.1801</netChg><open></open><pcl>50.59</pcl><vol>6871233</vol><date>20121015</date><pctChg>0.356</pctChg><class>N</class><symbol>DIS</symbol><time>17:22:16.576</time><tradingState>T</tradingState><tradingStateReason></tradingStateReason></dataItems><dataItems><bestAskPrice>0.965</bestAskPrice><bestAskQty>150</bestAskQty><bestBidPrice>0</bestBidPrice><bestBidQty>0</bestBidQty><close>0.9647</close><high>0.9839</high><lastSale>0.96</lastSale><low>0.9464</low><netChg>-0.04</netChg><open>0.96</open><pcl>1</pcl><vol>94387147</vol><date>20121015</date><pctChg>-4</pctChg><class>N</class><symbol>ALU</symbol><time>17:22:16.633</time><tradingState>T</tradingState><tradingStateReason></tradingStateReason></dataItems></BasicData>

##Goals for 121017:

Design goals:

  • Create a specification for an interface enterTickData() for putting data into the database from the feed. -- NOT DONE: close made a use-case in doc/UCDataFeed.md
  • Create java program that will read and print this XML data -- NOT DONE: close - Brett wrote XMLParseTest, which parses data for 'books' not 'stocks'

Implementation goals:

##Goals for 121010: ###Infrastructure goals:

  • Create a GitHub fork for the Data feed team -- DONE: both of you have created a fork. You need to choose one of them as the Data Feed team fork.
  • Bret: Whoever owns the fork you choose needs to make the other a collaborator on this fork -- DONE: Brett owns fork, and Shaun collaborates
  • Both: Clone the team's chosen fork as a repository on your local system - DONE

###Research goals:

  • Review the output format shown in http://www.nasdaqtrader.com/content/technicalsupport/specifications/dataproducts/BASIC_Web_Service_1.0.pdf and make a list of terms you don't understand. Put this list in a file named notes.md in the doc directory of your fork. DONE: moved it into the doc directory during the meeting
  • Search for the best way you can find to parse XML. DONE: Shaun found java and liked it
  • Add a section to notes.md with a short discussion of the alternatives you found and a recommendation on the technology we should use to parse the feed. -- DONE
  • All: Commit your changes on your local repository then sync these changes to the data feed fork -- DONE

###Notes: Division of labor:
We need to decide who is going to do what for the feed.

⚠️ **GitHub.com Fallback** ⚠️