A one-click tool to populate/update a local version of the Prosper DB's for the purpose of debug.

Features

• Automatically calls and processes eve-central dump files
• Writes a "all the data" SQL table (rawdata)
• Calls NORMALIZER, which does a second-pass processing on the raw data
• Builds Prosper-like DB locally
• Queries and processes zKillBoard data.
• Uses blueprints.sql to process a range of build data for items that can be player made

#DB_builder.py

A top-level wrapper that executes each of the steps for building/processing/writing database tables. Intended to be run as a single command-line function to populate data. This builds both the initial database and acts as updater to keep database up-to-date. This is designed to be run at any time to refresh the existing data.

1. Init globals/functions from config.ini
2. Query/process eve-central raw files into rawdata table. First-pass boils down "wisker plot" entries for each day/system/item/buy_or_sell combination available. Max, min, avg, stdev, top-5%, bottom-5%, volume
3. Process rawdata into prosper table (cloud replica db). Limit systems to top 100 by market capitalization, fill in NULL values for missing day data, build "gobal" values for region/universe.
4. Build kills from zKillboard. Process all kills to build lists of every ship/item destroyed and build totals for global/high-security/low-security/null-space/wormhole-space.
5. Build builder using blueprint data, for all items that can be manufactured by players. Exclude "special" BPs. Write out now/1d/7d/14d/30d lagging values. For T2, have permutations decryptors enable. MIGHT include meta-item use as well

##config.ini keys

#eve_central.py Includes all the methods/classes for handling eve-central data. Processes data in a 3-step process

1. Fetch dump.gz file from eve-central
2. FIRST-PASS: Process CSV and pair-down duplicate entries
3. SECOND-PASS: sort entries into region/system/item/buy_or_sell and build list of tuples (price, quantity)
4. THIRD-PASS: iterate over structure and generate max/min/avg/stdev/vol/top-5%/bottom-5% entries for SQL
5. Write out to rawdata SQL

##config.ini keys

#normalizer.py After rawdata is populated, it's possible to further refine the data in a second pass. Normalizer first builds a filter list by figuring out top-systems by total market capitalization for each day. The reasoning being that only a small subset systems is actually worth tracking, rather than all 5,000+ systems. Then, normalizer writes out global-global and region-global values for all tracked items. Finally, to have a full dataset, normalizer fills in missing values (not sure how yet).

1. Build list of valid systems by market capitalization, using daily data. Store "rank" data
2. For each day's valid list, build global-global (universe) and region-global values
3. Transfer item data for valid systems and fill-in-the-blanks. (Need to find regression method)

##config.ini keys

#toaster.py kills table is independent of price data. Toaster builds a daily kill report for every destroyed ship and item then sorts them into a range of bins. Using the API from zKillBoard, toaster iterates over each ship type for the missing time frame, and totals up hulls and destroyed equipment. Kill data is sorted into global and "zone" bins, resulting in a report of total quantity destroyed and where it was destroyed (HS/LS/null/WH). Also, like prosper's buy_or_sell key, kills has a PVE_or_PVP key, that also includes a "combined" value. Framework for Toaster.

A high-quality record of destroyed items helps build a picture of the lifecycle of an item. By combining both market-side analytics from prosper and builder with the end-of-life data from kills, it should be much easier to track trends as they occur. This beats playing catch-up using only market data, or heavy manual human work of trying to guess what volumes items are being utilized at. This DB should make Prosper unique.

1. itterating over a list of ships (ships.json), pull down .gzip JSON of kill data
2. process each kill object for destroyed data only
3. sort into appropriate location/type bins and update global counts

##config.ini keys

#builder.py Uses EVE static data dump data to calculate build prices for player manufacturable items. Needs special switches (or understanding of special cases) for each class of producible item. May need to split builder into sub tables for each class of manufacturing.