Video Series Summary - VTAstrobotics/Documentation GitHub Wiki

Contents

Prerequisites

To understand the content on this page, you should have read the SE Overview Wiki.

Why? When I first watched the videos, a lot of it didn't stick. More bluntly—I thought they were stupid. When I revisited them after doing SE for a year, I was like wow these literally told me exactly how to avoid a zillion issues we ran into, why didn't I give them credit before?

I think I needed a primer to understand why this stuff is useful and some context as to what everything is. Hopefully, the SE Overview Wiki accomplishes that for you. If not—if you find yourself thinking that a particular video was a waste of time—you can try reaching out to the leadership team. If you're on the leadership team, try asking your team lead. If you're the team lead, I would recommend bringing it up with your faculty advisor. Hopefully, this can help you get meaning from them. And maybe then you can improve our SE documentation for the next person :)

FYI

Here is the video series (backup copy) that these notes are based on.

I've watched the entire series a few times, so I can promise you that a lot of it feels repetitive, boring, obvious, or useless. I don't think he's a strong presenter. Still, I feel that there is a lot of value in each one, and believe Mark to be a very good SE.

Systems Engineering and the Project Life Cycle and Systems Design vs Engineering Design were selected as the most critical for all team members by a previous team lead, Thomas Giattino.

Note: This page is a summary of the video series. It is not a place to inject personal opinion, though anecdotes to guide understanding are used.

Systems Engineering and the Project Life Cycle

Why listen to this series?

The series was made in conjunction with the Systems Engineering judges at NASA. The presentation was made by Mark Powell, who is a longtime (and quite successful) systems engineer and professor of SE. He is still working with NASA on the SE part of the competition as of 2024.

Types of engineers

Traditional disciplines

Mechanical, electrical, aerospace, civil, chemical, software, etc.

  • All solve problems within their domain (i.e. mechanical engineers solve mechanical problems).
  • Taught to get to a point solution very quickly (and are quite good at this)
    • If you don't know what point solution means, please Google it, ask ChatGPT, whatever. This goes for any term, really, assuming you want to learn the material.

Specialists

Reliability, availability, maintainability, sustainability, survivability, serviceability, manufacturability, transportability, safety, logistics, etc.

Note: These are often referred to as the "ilities"

This is usually learned in graduate school, and most of the ilities are closely interrelated. Almost all manage uncertainty with probability and statistics.

Like traditional engineers, specialists get to a point solution quickly.

Systems engineers

Systems engineers break down complex problems into a set of simpler, single-domain problems.

They purposefully avoid point solutions to allow the expert assigned to the subproblem freedom in using their best judgment. For example, SEs would break down the complex task of building a lunar construction robot into a set of mechanical, electrical, and software problems for traditional engineers and specialists to solve.

Once the subproblems have all been solved, the SEs take back over to test that the system meets all of its requirements.

A real experience I had of needing SE: We had a robot that needed a certain component to move between two set points. As a software engineer, I said we could easily do that with code. Just get some encoder on there and we'll use pid—it'll be no problem. A mechanical engineer said what? we don't need code, this is simple. We can just use a hard stop on each side. Well, which solution would you choose? The job of the systems engineer is to be objective and find which discipline should solve which problems, so we don't get into arguments like this one.

Note: He presents a great illustration of this problem at around 15:00.

Important SE concepts

SEs are "wholistic," they're always looking at the big picture or the whole. The concepts are fractal (they're the same no matter how micro- or macroscopic) and holographic (they're the same no matter where they're applied)

  • The process is iterative and repetitive
  • Same process at every level of the system hierarchy (I recommend writing this down to come back to if you haven't yet learned what a system hierarchy is) The development is top-down with bottoms-up integration
  • This just means SEs start by breaking down the big picture and then we integrate once they have all the little pieces SEs are disciplined in doing things most people don't like to do. Specifically, high-quality documentation is a must. SEs are sometimes the bearer of bad news. They are honest when someone's great idea won't work.

The Project Life Cycle

I strongly recommend looking at his V-shaped chart of the Project Life Cycle (PLC) at around 18:00.

In general, the project begins at a very high level, and as time goes on (as you travel down the V), the level of detail increases as the project is broken into increasingly smaller pieces. At PDR, the SEs hand off driving control of the project to the Traditional disciplines, but remain working in a supporting role, until the full system has been integrated. Then, SEs reassume control of the project to validate the resulting system.

The level of detail changes in a discrete, not continuous, fashion. This is in conjunction with control gates (reviews). Control gates are critical for ensuring the project does not progress with inherent, high-level flaws. It is the opportunity to bring in new eyes/perspectives to spot anything the team may have missed. Control gates serve to validate that the solution is solving the correct problem, and assures consistency, completeness, and coherency. For example, control gates help teams spot conflicting requirements. This gives the team the approval they need to proceed.

The left half of the V is the Decomposition and Definition (or planning) phase. The bottom portion of the V—separated by PDR—is where we design, build, and verify each piece. The right half of the V is the Integration and Verification (or execution) phase.

The Central Elements of Project Management

Summary

The triad of PM

Schedule, budget, and technical performance form the triad of project management. These three are all extremely interrelated.

A baseline is the formal description of the problem space, solution space, and any parts the team has built—established by all of the stakeholders (this includes external reviewers). The baseline gets more detailed at each control gate.

How to manage the triad

First, set target values at milestones (start, SRR, PDR, CDR, etc.). If we're on target, great. If we're below the target, we take corrective action. If we're above the target, we replan to take advantage.

Remember, these are all heavily dependent on one another. If one, say, cost falls behind, this will affect both others (TPMs and schedule). Thus, it would be foolish to think of one without the other two.

Schedule

Some milestones are:

  • The start of the project
    • When you begin planning the project
  • The end of the project
    • Disposal
  • NASA-imposed deadlines
    • Think PMP, POL, SEP, STEM, etc.
  • Reviews/control gates
  • Major decision points
  • Each new baseline

The schedule is fundamentally a time ordering of these milestones—paired with objectives for each milestone.

The schedule evolves over time. It should get also more detailed over time.

  • These changes usually affect cost and technical performance (remember the triad!)
  • Evolves with the baseline. Schedule is part of the baseline.

Review of the schedule should be less frequently than every day but more commonly than just at control gates.

Budget

Must estimate the total project cost at the very start to effectively schedule and plan TPMs.

The budget evolves over time. It grows in detail!! Are you getting it?? Everything follows the same trajectory!

Guess when you review your cost budget? As needed!! Definitely at the reviews, and sometimes more, but not too often.

Again, this is part of the baseline.

Technical performance

Usually, some technical performance requirements are set by the competition. For example, the maximum altitude of a <=100 kg rocket. These lead to Technical Performance Measurement (TPM) targets, which may be a maximum or minimum or may be a parameter to maximize or minimize.

Note: The ilities are included in TPMs.

These evolve over time. They grow in detail. They are reviewed at least at every control gate. TPMs are part of your baseline.

Common sense for PM

You can manage yourself to death. You can't be perfect—rather, perfection includes small oversights. People can use their judgment. The key is keeping these oversights small.

  • You never have enough time
  • You never have enough money
  • You never have enough technical ability

To do this successfully, document why you are excluding something. This helps ensure it was a conscious decision and you/others remember why the choice was made.

This means you have—at any time—your past and current plans for schedule, cost budget, and technical performance; your system hierarchy; actual performance thus far; and what happened to cause any changes. These are critical to a successful project.

More detail does not make planning high-quality; informed decisions do.

How to Use Requirements

A requirement is a documented decision as a contractually binding statement. It documents the decisions we make, the problem space, and the solution space.

Requirements are traceable to a set of sources (answers "Where did that requirement come from?"). Requirements contain the justification for the decision. After all, how can you know whether to trust a requirement if you don't know why it was written? It may have been a mistake or no longer relevant. Requirements consist of evidence and reasoning.

Why we use requirements

We use requirements to communicate the desired outcome of a project. In our case, we receive our requirements in the form of a guidebook from NASA, which describes the problem space and constraints on our system (the solution space).

Note: NASA's guidebook does not deliver requirements in the preferred form (proper "shall" language) on purpose—in the real world, it would be extremely rare to get proper requirements from your customer. It is then the team's job to convert the guidebook into usable requirements.

"The primary reason why we use requirements in engineering is because people respond well to contracts and contract language."

Where to get requirements

We never make up requirements. They always come from somewhere (a decision made by someone) and are justified (why that decision was made).

  • They can come from contracts, memos, meeting minutes, competition rubrics, plans (i.e. PMP), analyses, etc.
  • When they come from outside, they may not be in proper requirements language. So we rewrite this information to the proper, usable form.

When engineers make decisions, they add requirements to the system. Thus, they need to write them down to document their decision and justification.

What makes a good requirement

A good requirement:

  • can stand alone
  • can be easily verifiable
    • quantifiable requirements make this easy
    • addressing only one thing at a time helps
  • is not subject to judgment
  • is only included if it is necessary

Requirement writing requirements

Requirements shall use the word shall.

  • Shall is the universal binding contract word. Shall is a binding word in U.S. law.
  • Suboptimal alternatives:
    • Will (means futurity; not binding)
    • Must (never use)
    • Should (never use)

Requirements shall be written using simple language. Requirements shall be written using concise language. Requirements shall never use descriptive adjectives or adverbs (words ending in "ly"). Requirements shall not use compound statements.

  • Compound requirements can be confusing
  • Compound requirements can also be difficult to verify.

Requirements shall never use pronouns. Requirements shall not use indefinites such as "etc.," "including but not limited to," and "support."

  • Once again, this makes a requirement hard to verify. (How can you verify that a system has met all of the criteria when you don't know all of the criteria?)

Requirements shall be written in active voice.

Example derivation of requirements

He uses the example of a spacecraft's power requirements.

One of the system's mission requirements is to provide coverage of the Atlantic and Pacific oceans between 10° and 60° N. latitude.

From an orbit coverage analysis, we can write a requirement that the payload shall operate for 4 hours per day.

From a spacecraft conceptual design and payload analysis, we can then write a requirement that the system shall provide 1000W average power with a 2000W peak for 4 hours per day.

From a subsystem conceptual design study and equipment survey, we can write some requirements stating that the solar arrays shall provide 250W per panel, batteries shall provide 60 Ah storage with 50% depth-of-discharge, and the charge controller shall allow the battery to achieve 95% charge.

To be clear, this is not written in proper requirement format because he uses this as an exercise to trace the derivation of each successive requirement from its predecessors and analyses.

How to Conduct a Review

Why reviews are useful

We conduct reviews to take a moment to ensure that we've addressed everything (all of our requirements) at our current baseline before we move on and decompose a new level of baseline. These also allow external reviewers ("alien eyes") to assure validity, consistency, completeness, and coherency.

Good reviews can help you catch things that you missed early. This makes it easier and cheaper to fix by minimizing rework and redesign.

Before the review

Ensure the products are ready to be reviewed.

  • This means our requirements and justifications (including traceability) are completed
  • The team believes that they are valid, consistent, complete, and coherent

Assemble a team of reviewers.

  • The project's team can review other subteams' work
  • Customer (faculty advisor can represent the customer)
  • Qualified outsiders (faculty members, graduate students, industry members, etc.)

Create a presentation.

  • Describe the project and the competition
  • Describe the current baseline
  • Describe your current solution
  • Explain that you are looking for help with validity, consistency, completeness, and coherency
  • Explain that if they find something wrong, they should suggest a solution as if it were theirs

At the review

Present to the reviewers.

After the review

Send the reviewers the presentation and give them a time limit (3–7 days) for their comments.

Collect and analyze all comments.

  • Decide what to do about each comment
    • If you like their fix, great, approve it
    • If you don't like their suggestion, modify or disapprove the comment
  • Implement decisions in your documents

Get a consensus with your reviewers for approval.

  • At a second meeting with all of the reviewers, walk through every disapproved or modified suggestion

Establish the new baseline.

  • Get approval from your faculty advisor and continue to the next baseline

Systems Design vs Engineering Design

This is a focus on the differences between the traditional disciplines and systems engineering. It will give an overview of systems engineering, which we will go more in-depth in the remaining videos in this series.

Classic engineering design process (waterfall)

First, engineers will analyze the problem space.

Then, they can design a solution. Often, there is feedback due to incomplete problem analysis which will update the design. The design process is iterative until convergence to a solution.

Next, engineers will implement the solution, which again may provide feedback to both the problem analysis and the design process.

Finally, engineers move on to verify their solution. Of course, this can provide feedback on all three previous steps.

This process works great for most traditional engineering disciplines (some exceptions being software and aerospace).

Systems design

Systems design is done functionally. The use of black boxes is key. First, we begin with our system requirements. Then, we can create a functional hierarchy, which describes all of the functions that must be accomplished to fulfill our system requirements. For example, we may have a "navigation" black box. The function is navigation. We do not put the tool or way to get there, such as GPS, SLAM, etc.

From that functional hierarchy, we can derive a functional architecture, which is a bit more low-level and groups functions together. This allows us to create our physical architecture (this is where we can say, OK let's use GPS).

Several traditional and specialty disciplines are involved simultaneously. Additionally, there are many different views of the problem and solution spaces involved simultaneously. This is why systems design tends to be more of a complex process.

Functional decomposition

Again, functional means we describe what is to be done, not what does it. Navigation, not GPS.

Each function has at least one requirement and each requirement has at least one function in the decomposition. Additionally, the decomposition has an input and output for each function.

Each function has identified all of its internal and external interfaces.

Systems design tools

Each diagram shows a different view of the functional and physical architecture. Together, they must be valid, consistent, complete, and coherent.

Functional hierarchy diagrams These provide a hierarchical view of the functions in the functional architecture. Related functions are visibly connected, which helps us track relations.

Functional block diagrams These are very similar, but only one level of the hierarchy, and show the connections in much more detail.

Functional flow diagrams This is a flowchart of functions. Each function may be broken into a few sub-functional flow charts (as you go down the system hierarchy).

Data flow diagrams Shows how data flows between functions. Every function has a data input and data output.

N^2 charts These show interfaces. Functions are on the diagonal, with inputs and outputs shown on the vertical and horizontal, respectively. Once again, these can be broken down into sub-functions which would have their own N^2 chart! Are you getting how these are all diagrams of the same thing that just highlight different aspects?

Timelines Shows how different functions are used over time.

ConOps Once there exists a physical architecture, it describes how the parts will be operated to accomplish the mission. This can be easily accomplished with an annotated flow chart. Of course, this should be done at each baseline, which increases in detail with the baseline.

The Central Role of the Systems Hierarchy

Everything SEs do is while looking at the hierarchy. When working top-down, it takes a lot of discipline to focus on the current level and to not decompose prematurely. The hardest part is refraining from decomposing into more than 9 subentities, but this allows us to manage our project to achieve a valid, consistent, complete, and coherent system.

What systems hierarchies look like

The typical hierarchy begins at the top at the System level, followed by its elements, then the elements' subsystems, the subsystems' assemblies, subassemblies, components, and finally parts.

  • A part is the lowest level, such as a bolt.
  • A component is an identifiable group of parts, such as the earpiece in a headset.
  • A subassembly is an integrated set of components that comprise an assembly, such as the entire headset.
  • An assembly is an integrated set of subassemblies and/or components, such as a pilot's console.
  • A subsystem is an integrated set of assemblies, subassemblies, and parts that performs a clearly separated function, involving similar technical skills, such as an aircraft's onboard communication system.
  • An element is a major product, service, or facility of the system, such as an aircraft.
  • A system is an integrated set of elements, segments, and/or subsystems that accomplish a defined objective, such as air transportation.

How systems hierarchies develop

We always start at the top—at the system level.

Each level of the hierarchy should be decomposed into no more than 9 entities. For our competition, 5 may be a better number. You may be able to identify more, but the point isn't to identify as many little units as possible. That being said, this rule of thumb does not apply to components and parts.

The system level is reviewed at a System Requirements Review (SRR). The element level is reviewed at a System Design Review (SDR). The subsystem level is reviewed at a second SDR, or Preliminary Design Review (PDR). The assembly level is reviewed at a PDR. Again, this is where the SEs hand off control of the project to the traditional disciplines. The component and part levels are reviewed at a Critical Design Review (CDR).

Rules of thumb when developing a hierarchy

The most difficult challenge for SEs is to never decompose an entity to more than 9 subentities between control gates. The second most difficult challenge is to never decompose beyond one level.

This is due, in part, to the number of interfaces. The maximum number of interfaces between 9 entities is 36 (8+7+6+...+1) per type (i.e. mechanical, electrical, software, thermal, data, etc.). As you can imagine, the complexity between 9 interfaces is nearly unmanageable.

Using the hierarchy

When top-down, the SE process is used to decompose each level with a control gate before proceeding to the next baseline.

When bottoms up, the SE process is used to verify each level before proceeding to integrate the entities up the hierarchy. There are again control gates here (integration reviews).

The Riskiest Part of Systems Design

Most of the risk in any project is held with its interfaces and integration.

Why interfaces and integration carry so much risk

Even a simple system hierarchy with a conservative (5–9) entity decomposition rule can create a staggering number of interfaces. One neglected interface can ruin the entire system.

Discovering a neglected interface during integration can cause major schedule slips and cost increases.

How SE reduces risk

He gives the example of the Destiny shuttle having the wrong connector on one of its interfaces. If it had gone to space with the wrong connector, it may have never worked. Luckily, one of their SEs carefully verified every interface and discovered this issue in the design phase.

How N^2 charts reduce risk

Typically, the functions or physical architecture pieces are placed on the diagonal. It tends to work best when placed in order of functional flow.

The key is that N^2 charts are a view of the interfaces of the functional and physical architectures. They show all internal interfaces as well as external interfaces.

This can show how functions should be grouped as well as control loops. For this reason, N^2 charts are the first place to look for Functional/Physical Architecture Synthesis.

He gives an example using a shopping trip beginning at 12:50. I think that it's very helpful to watch him walk through this simple example.

Concept of Operations

The Concept of Operations (ConOps) describes how users will operate a system. Do not confuse this with how the system operates. This is very common, but if you think of how the system operates before you design the system, you ruin the point of systems engineering. You end up doing a whole bunch of SE work only to go with the system you designed in your head, which can often be a point solution.

It is a critical view used in System Design. Think about it: if you don't consider how it will be used, it may be that a system is developed that cannot be used! Like all other views, this is done for each baseline. I hope you've come to expect that by now. You may even have one for each entity of that level.

At the competition, this can function as a script for how to successfully operate the system.

Note: This is very close to the specialty "Human Factors."

Uses for a ConOps

Often used in functional/physical architecture synthesis. Ask: "What human interactions are necessary for each function?"

ConOps can help identify the ease of use regarding an emergency stop button, for example.

ConOps can also help identify the requirements for training. In our case, how difficult is the system to operate? The easier, the more optimal the system's performance.

How to develop a ConOps

You can create a table. Begin with the functional flow for the current level in the system hierarchy. Each entity in the functional flow is a column. Then, divide the human interfaces into segments, for example, the robot transporters, mission control, and the robot itself. These will form your rows.

Fill out the table, piece by piece, and describe how humans from a particular segment will interface with that entity of the system.

Once you have a completed table, audit your N^2 chart to confirm that all human interactions have been captured.

Then, capture the key driving requirements for operations, human factors, training, etc.

Alternative methods

Instead of the table, you can simply annotate your functional/physical flow diagram with the human interactions.

Alternatively, you can annotate a timeline analysis chart. This is a chart of, for us, the competition run that over time that would show inspection, loading the robot, and unloading, as well as everything in between.

Using Trade Studies to Make Systems Decisions

A trade study is a process that helps us make functional and physical architecture decisions. They are also great at presenting decisions.

Trade studies can be derived from the first principles in the decision and multi-attribute utility theories.

Note: While useful, trade studies are incredibly easy to bias to favor a point solution. A proper trade study is difficult to pull off, but extremely beneficial if done well.

Even a perfectly performed trade study may result in the decision-maker selecting a lower-scoring alternative due to some inherent limitations.

How to conduct a trade study

Order matters.

Identify system decision

First, identify the objective of the decision, or what you are trying to accomplish. These come from the key driving requirements.

Develop evaluation criteria

Second, create criteria on which to evaluate the potential solutions. Think of what criteria would distinguish one alternative from another.

These are usually based on the key driving requirements. If possible, quantify them. Consider thresholds (must-haves) and balancing criteria (attributes to optimize).

These criteria need to be independent, or the scoring will double-count certain aspects of the systems that will load the study.

Develop weights for the criteria

Third, weight the criteria to reflect the relative importance of each one. It is easiest if weights add up to 1 or 100 or something, but not required.

The weights reflect the values of the decision maker, not the trade developer or the scorers.

Order matters. If this is done after developing suitable alternatives, a loaded trade study will be the result.

Develop suitable alternatives

Fourth, come up with suitable alternatives to achieve your objectives. This is a brainstorming activity where you consider point solutions.

  • We did some kind of like 3-5-5 method I can't remember the name but it was everyone presented 3 ideas in 5 minutes in groups of 5 or something like that. Then, the best ideas were selected as suitable alternatives to be scored.

Alternatives are distinct from one another.

Conduct this activity with different people than those who created the evaluation criteria and weights. In fact, these people should not even know the evaluation criteria or their weights! This helps to keep the trade study honest.

Score alternatives

Fifth, score the alternatives.

The scoring may be done by different people. For example, a mechanical engineer should not score software complexity.

The scorers do not have access to the weights of the evaluation criteria. They get the range that the value can be in for that criteria, but they do not know the weights. Again, this helps to keep the trade study honest.

Determine the winner

Sixth, capture the winner as a key driving requirement for your system.

Multiply the weights by the scores and sum for each alternative. Then, rank.

Note: Any zero score for a threshold evaluation criteria eliminates that alternative.

Note: It is good to compute the maximum score for the trade study to compare the alternatives with.

Perform a sensitivity study of the criteria and weights. For example, if you remove an evaluation criterion, and the ranks are all the same, you may want to eliminate that criterion. Or, if you remove one criterion and the highest-ranked alternative becomes the lowest-ranked alternative, you may want to do some more research.

Conduct an adverse consequences analysis for the top scorers. If there is something bad that could happen with an alternative, you may want to eliminate that alternative.

BE CAREFUL! You may revise the trade study, but you must remain an honest broker. The eventual winner may not have been the top scorer, but then the burden of proving an honest trade study lies with the decision maker.

Results of a trade study

Now that the trade study has concluded, it becomes a part of your baseline, as it is the justification for your decision. That decision becomes a key driving requirement—which is traced to the trade study. He shows a presentation of a trade study at around 12:50.

Add the rationale if the top scorer was not the winner.

In the end, your decision is justified, defensible, and documented.

If there is a problem with the winner

Suppose the project begins with the winner of the trade study, but at some point, a problem is discovered. At that point, you can go back to your trade study and analyze the second-place alternative to move forward. You do not have to conduct a new trade study.

Next, go back to the system hierarchy and locate the key driving requirement that was traced from the original trade study result. All entities below this entity are potentially affected, but no others. Therefore, you can focus on fixing that branch of the system hierarchy.

Integrating Reliability, Safety, and Other Specialties

The specialties support the development of the project throughout the entire life cycle.

Why the specialties are important

Often, a specialty requirement can be the primary key driving requirement. For example:

  • If a defense system does not have an extremely high probability of availability, it may be useless in a moment of need.
  • If a commercial product does not have cost-effective manufacturability, it won't be able to make the corporation a profit.
  • If NASA's missions doesn't have an extremely high probability of safety, the crew could be lost and the mission could fail, wasting billions of dollars.

If an ility is a key driving requirement, then, of course, it will drive the design of the system. So that's why they're important, Jack.

Defining the specialties

Reliability is the probability that the system will operate as intended for the specified service lifetime.

Availability is the probability that the system will be ready to perform its intended function.

Maintainability is the probability that a failed system can be repaired and returned to service within a specified period of time.

Logistics is the probability that a part needed to repair a failed item can be obtained within a specified period of time.

Safety is the probability that the system will cause no harm or injury within a specified period of time.

Human factors is the probability that a human will be able to operate the system as intended.

Transportability is the probability that an item can be restored to operate as intended after transport.

How to use information from the ilities

The ilities are probabilistic at their core. This establishes the maximum acceptable risk. Most of them are highly interrelated.

For example, if the system ends up with 95% reliability, the maximum acceptable risk of failure (during a specified period of time) is 5%. Notice that availability cannot exceed 95%, because a broken system is not available. Similarly, a system in repair is not available, so availability is also driven by maintainability and logistics.

Therefore, it is important to set up relationships between them to ensure that each ility is satisfied at every baseline.

How to do the ilities

State ility requirements using a probability over a period of time. For example: the system shall have 95% reliability over 100 hours of operation.

Even one performance requirement may have many associated ility requirements. Just like performance requirements, ility requirements are traceable. Verifying ility requirements is very difficult, but a good starting place is:

  • First, identify the important ilities for the project
    • Ensure the whole team is aware of these and their meanings
  • Consider these as key parameters for TPMs and as evaluation criteria for trade studies
  • Address these at every milestone
    • Use a checklist to ensure they have been considered with all work up through the present milestone
  • It should be someone's job to address these ilities at every step of the system design process
  • Use your common sense

In my opinion, his advice is for us to use our best judgment. I think we could try to self-study the specialties or find a graduate student studying them.

Verification

Verification is how to be sure that you got what you needed.

If you don't verify that your system completes its objectives within its constraints, how can you know that you've met the objectives of the competition until you go to the competition? If you don't verify its size, for example, it may be outside of the maximum bounding box, and too late to fix it, disqualifying the team.

Why we verify

To reduce the risk of failure.

If we wrote perfect requirements that others understood exactly as we mean them, there would be no risk. But we are humans. Therefore, we verify to reduce the risk of imperfect requirements or misunderstandings.

Lacking just one key verification requirement, systems have failed, costs have blown up, companies have died, etc.

Why verification is hard

Verification can be successful when the key driving requirement is not satisfied. For example, a verification requirement by the U.S. government for dog food was that it must have a certain amount of protein, but their test did not distinguish between digestible and non-digestible proteins, so a foreign company essentially killed a bunch of dogs because of the poor verification requirement. Therefore, it is important to write high-quality verification requirements.

Verification can fail when the key driving requirement is satisfied.

Verification can consume over 95% of the development budget for major projects. For example, if we had to build a realistic lunar simulated environment, that would cost a whole lot more than our robot. We can't afford to do that, but you see how it can be very expensive to verify properly.

Testing and verification can take much longer than system development. Therefore, allocate time in your schedule to verify. More time to verify = more risk mitigated.

How to verify

Include what constitutes success in the verification requirements. For example, if you are verifying your system is beneath a mass threshold via inspection on a scale, success is that number reading less than X kg.

Verify all of your interfaces. For example, if two things are connected by a plug and receptacle, verify by inspection that they are of the same type.

Consider your ConOps for your verification context. For example, verify that a human can press your emergency stop button.

Note: He recommends writing the verification requirement simultaneously with its corresponding requirement. These may iterate.

Verification methods

You can use a trade study to determine which method to use. You can verify if you:

  • Inspect
    • Look at it
    • Use a simple measurement such as reading a scale.
  • Test
    • Run an experiment, gather data, and analyze the results
  • Demonstrate
    • Run the system to see if it works, usually at extreme conditions or for extreme durations.
  • Analyze
    • Work out the math
    • Use a simulation

There is a great chart at 23:40, but basically expect to use mostly analysis and demonstrations at SRR, a little more testing and inspection at SDR, a close to even split at PDR, and testing and inspection at CDR. Notice that you plan these as you go down the system hierarchy, but you implement them in reverse. Remember the "V" chart!

What to verify

Focus on the key driving requirements, but a verification requirement should be written for each requirement.

Know that it is unlikely to verify all requirements at all levels of the system hierarchy. Therefore, use common sense to select a balance of time, money, and risk by not verifying certain requirements. Document this decision with its justification.