Computer Systems will Fail

A lot of people are putting their trust in the new AI systems. In reality, all that we can be confident in is that eventually these systems will fail. The failure point is not in the systems themselves, but in the people. Users will always push systems past their safety points. People will always misuse tools. Human beings will make systems fail.

At a manufacturing plant, some material got jammed and a machine stopped working. A young guy hops up on the conveyor belt, climbs up and starts unjamming the material. This is a total violation of safety rules and if something had started up again, he would have been killed. But this is typical of how human beings operate.

People push themselves and their machines past the limits. Folks drive on bald tires. Truckers drive past the time limits. People drive cars with broken brakes, front windshields totally covered in ice or broken out, or any number of other things that make the car unsafe. People run through red lights with the confidence that others will yield the right of way.

No matter how good a computer system is, humans will use that computer system in ways that it was not intended for.

People will use software in ways that the designers didn't expect. Using systems in these ways can cause problems when the software is upgraded. When Microsoft releases a new version of their software, they run into situations where people have used bugs as features. Those people then report problems with the new version as the "feature" / bug had been fixed. How they used the system does not work anymore.

Many computer CPU chips have been misused. Every computer CPU chip has instructions that were not "finished", documented, or even intended. On many a CPU, people explored those undocumented instructions, found out how they worked on a certain version of the CPU, wrote programs around those undocumented instructions, and then had problems on the next version of the CPU when that "instruction" didn't work any more or worked in a different way.

A few years ago, Microsoft attempted to release an AI chat bot. It took only a few hours before people had fed it enough harmful material that it was generating similar harmful material for more innocent requests.

An AI system intended to help diagnose hospital patients had not been trained on enough minority patient data and was giving incorrect suggestions when faced with minority patients.

So, we can expect the computer system to fail even when all the hardware is working properly. It will fail because human beings will push the hardware past safe operating parameters, push the software with data that it wasn't designed for, and use the system in places and situations that it wasn't intended for.

Principles over personalities

A number of organizations operate on personalities. Each organization certainly has a style which some will call a personality. But the issue is when personalities determine direction for an organization instead of asking what is the right thing. Personalities are more likely to cause problems than solve the real issues.

The personalities at the top of an organization are able to set the tone for the organization as they select which principles the organization with operate under. But organizations get into real trouble when they operate to please a personality instead of operating under principles.

Personality driven organizations often get filled with "yes men" who protect that personality from reality. There is often a lack of trust. Issues rarely make it up to the top level. Nobody calls out the personality for the mistakes that they are making. No matter how good or how smart a person is, everybody makes mistakes. The more a person believes that they are right, the more likely they are to be making mistakes that nobody confronts them on.

Well run businesses operate on principles. These principles can be things like "take care of the employees and they will take care of the customers." Because the organization has principles, the top people can be confronted on their mistakes, their ignorance of situations, and when they have ignored a principle. By relying on principles, the organization is able to have higher levels of trust, more open communications, and is more likely to be able to react to changes in the marketplace or the business environment. All those mean that the business is more likely to be profitable.

Risky World

There are representatives going around from a fake country called the "United States of Kailasa." They have spoken at UN events. They spend five days in the City of Newark as guests of the city. They went down to Paraguay and conned the Agriculture Ministry into signing a memorandum of understanding. The country has issued coins and visas. The leader is wanted on criminal charges.