Future Iterations - stlrda/211Dashboard-Workflows GitHub Wiki

What's Needed?

We are extremely happy with the progress we've made thus far, however, there is always room for improvement! In this section, I'll outline some important "features" that I believe are necessary to fully realize the potential of this project.

First, all data collection needs to be automated. Currently, there's 4 data sources that aren't truly automated, meaning they require some human intervention to actually get the data into the database. The main concern is with the U.S. Census data. The collection of Census data is a tedious and potentially error-prone process, so accessing this data from an Airflow Operator will be very important. To accomplish this, we are really at the mercy of the U.S. Census Bureau, specifically we need them to continue to build out their API. While the bureau has an API for developers to access, it does not include all the tables that we are concerned with. It'll be a good idea to monitor the Census API development, hopefully they'll have what we need sooner rather than later. Also, having the census data automated means that the areas of interest table can be automatically created/updated as well (besides editing the areas_of_interest.json file in this repo's resources/ directory).

Similarly, there's actually already an API to access the HUD Crosswalk files (shoot! we missed this one...). So this feature can be added whenever someone has the time to commit. As for the funding data, the RDA will have to figure out a way to have that CSV file web-accessible (or continue to just load any updates directly to the project's S3 bucket). Overall, these 4 data do not change frequently (every 1-2 years on average); thus, automating these data loads is not super critical. However, adding this automation shouldn't take much effort (unless the Census Bureau slows things down!) and will totally be worth the reward.

That's all I have for project improvements at the moment. Stay tuned.

Endless Possibilities

Seriously though, the possibilities are endless! We really hope to see this project continue to expand and incorporate more data. Hopefully we've inspired some companies/organizations to share additional data, so that we may best assess and serve the needs of our local communities.