The Hidden Risks of Engineering Without Proper Checking

I’ve written quite a lot about problems with the use of AI chatbots, modelling and simulation software by chemical engineers. I have also written about the use of questionable advice and spreadsheets from random people on social media. I see now that this is actually an aspect of another problem I have written about a few times. It’s a validation/verification problem. It’s all to do with failure to check sources, inputs and outputs.

I get to check a lot of process and hydraulic designs, mostly in situations where designs didn’t do what they were supposed to. I have also had jobs recently focusing on design processes which were thought to be viable until I started looking at them. I have seen very few cases where the correct level of checking was applied in recent years.

The problem appears to be systematic: there aren’t enough people who actually know how to design things to carry out the checking, and new designers are not being taught how to do it. Neither do they seem keen to learn.

Doesn’t everyone know that LLMs give you an instant answer in a way that no one really understands, written in such a way as to sound plausible? The LLMs tell you to treat their output with great caution in real engineering applications: “Triple-check every number. Consult standards (e.g., ASME, IEEE, ISO), use certified tools, or peer review with professionals.” But what percentage of users actually do that? If you were to do this level of checking then you’d have been quicker to just design it yourself in the first place, but of course you may not know how, and your boss might not be willing to give you the time or resources to learn.

Doesn’t everyone know that modelling and simulation software output is at best as good as the input data, the process model you set up, and whether you chose the right thermodynamic model? That getting your model to converge is not the end of the design process? The advice on checking for LLMs applies also to simulation and modelling output. I don’t see much evidence that it is commonly followed.

Doesn’t everyone know that the internet is full of liars, charlatans and incompetent self-publicists? Are people unaware of the risks to IT systems of downloading excel spreadsheets offered by random strangers on the internet? Don’t they care that such spreadsheets are impossible to validate because they lack referencing to sources, annotation and documentation? That there is no way of assessing that their authors are even competent? I could go on, but the answer to all of my questions about this is clearly no. Those who offer such spreadsheets on Linkedin are rapidly swamped by eager public requests for a copy.

These tools should only be used by people who do not need them. If you cannot design without them, you should not be allowed to design, because you have no way of checking the output for sense. All design output is wrong to some extent. Spotting the places where this is a real problem is what engineers are for.

Part of the problem is that the educational system does not teach this, and the advent of LLMs has worsened this. Far too many students simply get ChatGPT to write their homework, and too many lecturers cannot be bothered even to resist.

A lot of people find checking boring, but checking IS design, in the same way that proofreading is what being an author is mostly about. LinkedIn is also full of self-published literature, the quality of which gives you an idea of how trustworthy those spreadsheets are likely to be.

I have already been involved in expert witness cases where over-reliance on modelling and simulation software has led to bad outcomes, with engineers being judged incompetent. I have seen unchecked calculations used to for construction, and signoffs by more senior engineers on calculations which have clearly not been subjected to the most cursory examination.

The problem here isn’t these new tools. The problem is that “a fool with a tool is still a fool”, (which the internet will tell you should be attributed to the software engineer, Grady Booch, or possibly R. Buckminster Fuller, Abraham Verghese etc. Someone should check. I can’t be bothered).

The biggest problem is posed by LLMs. Unless you are going to get your design spreadsheet or model from some random guy on the internet, even bad design was effortful before LLMs. Now you can generate something which looks pretty good in no time at all. Skip the necessary checking time, and you can spend the rest of the day scrolling your feeds. It’s not like your boss can be bothered to check. Checking is so boring. Also they probably don’t know how to either!