Home - jmadison222/knowledge GitHub Wiki

Data Engineering is Fixing Messy Data - Most data is a mess. Then you have to fix it. That’s my definition of data engineering—​Fixing Messy Data. The result is data that is both ready for business use and able to be composed with other data for larger and larger business solutions—​sometimes called "tidy data" or "composable data" or what I like to call "lego brick data" since when it’s done right you can click the data sets together like the famous toy bricks.

Data Quality Issue Resolution Options - When a batch of data is being processed and a data error occurs, there are only four courses of action. They all have down sides, so the question is not whether you like or dislike a particular course of action, but do you dislike it better than the other three alternatives.

Linux as IDE - Know that Linux—​the entire system—​functions as an IDE. Consider using Linux as the single most powerful IDE available. Plan for the learning curve as Linux-as-IDE is also the hardest IDE paradigm, to be sure. Embrace the "and"; use both graphical IDEs and Linux-as-IDE, each for their respective strengths.

Knowledge Management Principles - We live in a knowledge economy, yet most people and organizations are terrible at knowledge management (KM). Here are the top themes I’ve learned from decades of KM work in the technology domain in the financial services sector.

Outside First Forecasting - Estimate from the top down. Look at the big picture first, generalizing it to your current situation. Then use the specifics to adapt your generalization as needed.

Reusable Component Success Factors - Reusable components are units of code that follow the write-once-use-many rule. The value is clear to see, but making it happen is very difficult. The following aspirations increase the odds of successfully building reusable components.

Social Facilitation Effect - Keep your work under the radar as long as possible to maximize success. Assume that once your work goes public, managing the publicity will take more time than doing the work; plan accordingly.

Leaders Versus Managers - Not all managers can lead. Some individual contributors can lead well. It is useful to separate the two notions so they can be measured and utilized independently. The best bifurcation I’ve found is from this Harvard Business Review article.

Strategy Versus Tactics - Strategy and tactics are both valuable, but the line between the two can be fuzzy. This article from Harvard Business Review was about CRM tools but provided a great contrast between strategy and tactics. Forget the CRM tools. Consider the contrast of strategy and tactics, as provided in a table from the article.

Apollo 12 - Thirty-six seconds after liftoff, the Apollo 12 spacecraft was struck by lightning. The resulting power surge caused instrument malfunction, communication system failure, and telemetry data distortion. Mission control engineers on the ground scrambled to find solutions. The flight director prepared to abort the mission.

Health Program - I lost 50 pounds in 5 months using the techniques listed here. I had spent 5+ years eating poorly and not exercising. Then a coworker started a weight loss competition at work. I worked hard and did well in the competition, but far more important was what I learned about health and fitness.