Evaluation comments from professionals

Instructor's Reflection on Feedback

A foundational concept of STAMP is building effective feedback to inform decision-making. Feedback is just as important in education as it is in engineering and operations. As an instructor, feedback from participants is essential to keep improving the course and making sure the education is high-quality. I often use evaluation forms to collect this valuable input after class. Below are comments from participants in previous evaluations that may offer insights into the experience from their perspective.

Comments from professionals:

What did you think of the class?

I was afraid I wouldn't be able to follow the class due to the fact that I had no experience in safety. However, and surprisingly, the class explained the method in a very intuitive way and the instructor was very clear while explaining. From my point of view, this class is an A+.

I've spent my career in aviation safety. This class provided a deeper insight into hazard, safety and risk than what I was used to. It's a revolutionary understanding of these concepts for aviation. 

This is one of the few virtual courses of several that I have taken that has been fully engaging with knowledgeable and efficient instruction, good hands-on interactive group exercises and plenty of material for further study. The customer I support is DoD and the more STPA's value and results are properly communicated to this particular organization, the more they will see value in implementing this method within their analyses, as well as getting their folks fully trained up and attending STPA courses. I had a different perspective of what STPA was even from reviewing the handbook and seeing outcomes of DoD-completed STPA results, and that makes me think it may not be properly implemented in some cases, and I hope to help change that perception by being able to communicate the things I learned in this class and share my work as I apply this method to my own analyses for my job. Thank you so much for a very valuable class. 

The new perspective this method provides is useful from a regulatory perspective even if industry uses older methods because it gives us a different tool to test their design analysis and make certification more complete.

Where were you when I had to take all the boring safety classes before? This is the class I should have taken. The way the case studies are explained make it impossible to forget. Well done!

I was a skeptic at the start about this class being beneficial. I've been doing safety for a long time and I have applied ARP4761 in such a way that functional failures are well captured before consolidating failure conditions. What I believed was that STPA might just capture some common modes and risks associated with external environments earlier than we do today. I wasn’t fully aware of the SOTIF problem before this class, and that was my first AHA moment. Safety engineers should approach this class with an open mind and be willing to discover gaps in their own thinking, even if you have a lot of experience like me. The format of the training opens minds through real world applications. I thought the early STPA papers were unfortunately (1) aggressively written against traditional techniques and (2) “academic”. This training corrected my prejudiced bias. Thank you, I would have dismissed something good.

The case studies brought the concepts to life. Seeing examples from different industries opened my eyes to these patterns and gave me a deeper understanding that I hadn’t noticed before.

I really like this class, it is exposing some holes in my thinking. This is the best class I have taken.

As someone with a lot of safety experience, I just wanted to get into STPA and honestly wasn’t excited about going through the "foundations" part. Now that I've been through it, I can appreciate why the class started with general safety foundations. It doesn’t just go over the usual principles—it actually picks them apart, pointing out the practical "gotchas" that often get missed and showing how things can backfire. It was more helpful than I expected, and helped me understand where a new approach like STPA fits into our engineering processes.

Learning to using the right syntax in the group exercises really helped. It took away the awkwardness we usually have on deciding how to convey our thoughts and reach a consensus about safety concerns. It helped speed the processes of pulling together and documenting our thoughts about potential risks, and making requirements to solve them.

Loved how the class structured the teaching with hands-on sessions.

I really like the work we did around the STPA control structure. It helped me understand what I was doing right and wrong.

I loved the interaction in the practice sessions. It helped to identify if we were on the right track as a group instead of only identifying things in a narrow context. The practice sessions are really what helped set up the system thinking we need to have in each step.

I started this class as a test/system engineer focusing only on failure conditions. This class taught me how to zoom out and look at it from an interaction and operational point of view. The instructor is very knowledgeable and taught well.

Super awesome class. Every day I had new mind-blowing insights. Great class with pertinent examples!

This class greatly exceeded expectations. I can immediately begin applying this into open-ended and missing processes in my work. Really great class!

[On the first day of class] I'd recommend continued caution on comparing STPA vs. PRA. An experienced HRA/PRA analyst can do pretty much anything that's needed.

[On the last day of class] STPA could be an important & useful complement to PRA. Also, STPA is the only tool that could identify many of the automation/operation control problems in the industry.

Applying STPA to nuclear systems provides new insights beyond what our current processes find--I agree. (unanimous agreement from all NRC participants in the post-class survey)

Dr. Thomas is a very good speaker. I figured the class would be effective but boring. It has turned out to be effective and engaging and I think this is mainly due to Dr. Thomas's presentation style about the subject matter.

I really enjoyed the examples, especially when coupled with diagrams. Reviewing other folks activities together was also immensely valuable, especially for postmortems I'm familiar with and / or systems similar to the ones I handle.

The best part of the class was the mindset we learned to look at problems from an interactions point of view and not just component point of view. Another valuable lesson was how to design systems around interactions to prevent dysfunctional behaviors.

It's so much easier to learn STPA this way. Learning on your own is hard!

The examples have all been fascinating and have helped build a very solid foundational understanding of STAMP. I'm already starting to look at all sorts of systems and possible failure/error modes differently!

The best part of the class was the detailed explanations of how each part of the analysis translates to examples. Not just walking through the examples, but looking at the blank slate and how to develop each piece from a point of reference.

I like that John spent ~2 sessions explaining the fundamentals and using real-life examples. It was an excellent way to instill knowledge-based change of our behavior.

I love the examples. I’ve been waiting for someone to teach me a good way to analyze failures since I joined Google. After years as a site reliability engineer, I do feel some level of intuition but seeing it all explained and formulated in words and concepts and theory has just been such a mind blowing experience.

For me, having already attempted to apply STPA on Google systems earlier this year, the class was very useful for me to understand how to think using STAMP and the foundations that were covered early on in the class. The real world examples are also helpful; it's easy to see why STPA is being adopted by software companies.

This is a fun class, with great examples from other industries that I know I'll quote. Great pointers to terms and research that is immediately relevant to the job.

The best part was the numerous case studies and examples, which helped tremendously. Looking at different control loops and so many of them just registered well in my brain.

This is seriously fantastic. I feel like I still need to work on how to apply this in some ways, but I see the amazing potential.

Working through the examples was great. The quick iteration between practice and feedback helps solidify the knowledge.

Really liked working in small groups that get shuffled for each exercise, it's my favorite format of the various ways we've worked in class.

Seeing the real examples was brilliant. Trying to work in the group to build a system is really helpful. Also: Dr. Thomas is so fantastically encouraging- I really appreciate that.

Thank you so much for setting this up! I expect this will prove to be my top training at Google impact-wise (even after nine years here with lots of training and workshops).

The best parts of the class were the real life examples and clear understanding of how systems were designed, unexpected or undesirable interactions, and how the ultimate loss/hazard was realized.

The human vs. autonomous/ADAS statistics were eye opening. Enjoyed learning how easily automating human tasks can make things worse if you're not careful, and how safety mechanisms can (and sometimes have) made things more hazardous, it was powerful.

The instructor is both passionate and knowledgeable. It really keeps the class interesting and it's hard not to stay engaged.

The whole class was excellent. I enjoy the instructor's passion! As a former college professor and adult trainer, I appreciate how you have crafted the training: weave deep, powerful, personally meaningful lessons into a gradual transition from high-level, simple explanations and exercises to increasing detail and complexity. If it were all put on the table in one theoretical chunk, it would be indigestible for nearly all except those with cast iron stomachs and steel trap minds. I'm enjoying both the content and the craft!

The examples are really driving home the point. They are relevant and the findings are really interesting, even to those of us with lot of experience in safety.

Thank you for broadening my world and giving me a powerful set of new tools. And a very fun and engaging time, to boot.

Great class, one of the best and most practical new tools I have learned in a long time. From prototyping demos for one off projects or full production, this class provided a usable method even in the face of ambiguity and lack of design to find those issues that can cause large delays in programs.

Amazing class John. I have been working as Functional Safety professionals for the past 10 years and your class has changed my point of view towards autonomous vehicle systems.

There are three trainings in my life that have been pivotal. This is one of them. Thank you for an incredible class!

The class was unexpectedly entertaining and highly informative. Very good energy on the part of John. His enthusiasm really keeps you engaged and interested.

I've been in industry for one month and I feel very comfortable with everything we've done in class. The lessons are both intuitive and surprising at the same time.

I really enjoy the practical applications and seeing how we think and sometimes omit poorly engineered things.

This class had the perfect balance of "theory" and application with exercises. Getting to do hands-on work kept me entertained and ensured that I was understanding things. The presenter was also great and had high energy.

The most useful part for me was the comparison of results from different analysis techniques on a real control system. I was unconvinced at the start of the class, but walking through the real examples and results really helped me see the limitations of our current approach.

I really liked how we did the step by step walk through applying STPA. Honestly I was a little skeptical at first but now I see the impact this process will have in our analysis.

This course was an AHA moment for me. Seeing the benefits of the new method beyond our usual process has really changed my perspective.

I loved the mix between the more directed examples where the answer is provided with an explanation versus the more open-ended examples where we have to come up with answers. The class has helped us build confidence and muscle memory on these challenging applications.

The most useful part of the class to me was when you went over the "bad" examples of each method and the things to avoid. It reinforced the learning on what we should be expecting in every step.

I appreciated how the sessions alternated from (over)confidence boosting, then a pullback down to earth where we struggled a bit and learned a lot, then another confidence boost, and then a reality check comparing our work to other practitioners in the field. It helped us learn a ton of material in a short time without getting frustrated. At the end of the class we landed at a just-comfortable point, ready for next steps. Nicely calibrated!

Great class! Very convincing. Great presentations skills, kept the audience interested and invested throughout.

Just wow, fantastic!

The class was really interesting with the case studies from real world. I was surprised that so many famous accidents have been linked to system hazards. It helped me realize that my understanding of these accidents was superficial and I wasn't aware or downplayed the causes that are most relevant to my work. This class was a wake up call to understand how some of our common assumptions have led to accidents in the past, and what to do about it.

The examples and practice sessions were beneficial to learning the process. Being able to tie in the process to industry was extremely helpful in understanding the significance of the class lessons.

The instructor's passion was very useful and it provokes thought on how others can implement this.

This is a great course, and these methods should be widely taught as part of any systems engineering education.

Overall, I enjoyed the class and learned a lot. I think the results speak for themselves.

Really great presentations. The enthusiasm was a highlight and helped me stay engaged. The real-life examples are always the best part, and your slides were full of them. I appreciate the checking in and the attention to making sure we fully understood each concept before moving on.

What groups in your organization would benefit most from this class?

We have a fairly diverse group of job functions represented, so I don't say this lightly. But this should be spread wider throughout our entire product development engineering teams.

Design engineering teams

Our complete product development organization would benefit from this training in addition to the R&Q organization.

Leadership!

Reliability, Management. Anyone trying to make software people do FMEA's.

People in upper management pushing for L5 AV's [level 5 autonomous vehicles] in 3 years.

This class should be given to all technical leaders, especially in the SW and Algorithms groups.

All IT teams

Vehicle Engineering

This class should be expanded to invite more of our team members from each group.

All of Engineering.

The majority of our product development and R&Q organizations would benefit. Could see some benefits for customer service, warranty, and other teams like that potentially as well.

All! Seriously!

To start with, executives. Then technical teams and middle management involved in developing products. This would include Reliability, Manufacturing, and Manufacturing Quality.

Really important are the "skunkworks" type engineers who develop custom solutions for customers: often developed without much consideration of safety, their products often slip into full-scale production without program-level engineering checks and balances. Those products are notorious for surprise costs.

Inviting chief engineers and directors would help buy-in. I'd like everyone on my team to hear some of this course too.

Senior management should attend a presentation summarizing this course.

Software group

Any group that is providing a product or service.

Chassis team would find benefit with this program as this has useful implementations outside software as well.

Product Development, Quality, Reliability

This is useful for requirements discovery in general, not just safety/cybersecurity. All requirements engineers should have training in STPA.

The Human Factors and HMI groups

Design engineering teams

All of them

Body Engineering - as a Body Engineering group there would be some good benefits to try to identify requirements that we often miss or overlook.

All of them if you want a true safety culture

Design Engineering

Management-level leaders attending this training would streamline the process before major milestones in our production.

We would get good ROI on Advanced Reliability Engineering, Manufacturing Engineering, and Quality Engineering attending this course.

It would be good for our Program Management organization to support refinements of the New Product Development process.

All of our air vehicle IPTs should have representatives taking this course.

Security Auditing

Incident Management

Software Design Teams

Most of the Systems Engineers should be taking this

Test Pilots

Anyone who is involved in Autonomy

Management

The Tech Fellows should be introduced to this

Any/all engineering groups

Flight Control System group

Avionics IPTs

Infosec and CyberSecurity groups

Quality Assurance group

Middle management, to understand that we need multidisciplinary teams and resources available to make sure these new findings don't get lost

Additional members of cyber/safety teams and also some key members of different systems - so they can understand the questions we need answered from their team

Clearly, our decisions makers at a program level need to take this class.

The System Engineering and the Safety groups should take this class together

Weapon systems + subsystems engineers

Other software design groups, including strictly security-focused products

The Nuclear Surety group

Formal methods group

Project Managers

Junior and senior software developers

Software Testers

Emergency planning team

Program engineering

Operators and operations groups

Site engineering

Other design teams