Underwriters’ Laboratories has just released its draft of UL 4600 — the first comprehensive safety standard for autonomous products. The new standard offers a guide to “build the safety case” for your AV.
The hype cycle of autonomous vehicles (AVs) might have already passed the peak of inflated expectations. The next step for AV vendors is to level with the public, to acknowledge that there's a critical question that still lacks an answer: How safe is safe enough?
Not coincidentally, Underwriters’ Laboratories (UL) has just released its draft of UL 4600 — the first comprehensive safety standard for autonomous products.
UL 4600 isn’t your grandfather’s rulebook. Anyone looking for technical specifications to quickly build a safe AV, will be barking up the wrong tree. UL 4600 isn’t that.
This standard lists no specifications, stipulates no hardware or software (no mandates for types of sensors, SoCs or embedded software), and provides no prescribed guidance as to the proper development process. Instead, UL 4600 offers a guide to “build the safety case” for your AV.
In short, other existing safety standards prescribe “how to do safety” by following step 1, step 2 and step 3. UL 4600, in contrast, is about “how you’ve done it [safety] enough,” explained Phil Koopman, CTO of Edge Case Research, and a professor at Carnegie Mellon University.
Asked about the basic design principles of the UL standard, Koopman, one of UL 4600’s authors, told us, “If you can't say what it means to be safe, and you can't explain why you think the system is actually safe, then probably your system is not safe.” A safety case is “an important piece of designing safe systems,” he noted.
Recommended
UL Takes Autonomy Standards Plunge
Why another safety standard is needed
As Koopman noted, “Self-driving cars will change how we'll have to do safety compared to human-driven vehicles in some fundamental ways.” He stressed, “It should be no surprise if our approach to safety standards changes as well. But it's important not to forget the hard-won lessons and engineering approaches we already use.”
UL 4600 makes it very clear that it’s not the only safety standard AV designers need. “You also need good engineering methods such as those discussed in other standards [including IEC 61508, ISO 26262 and ISO/PAS 21448 (SOTIF)],” said Koopman. “I expect developers will use both conventional standards such as ISO 26262 together with UL 4600.”
UL 4600 markedly differs from other safety standards in that it focuses on full autonomy, without human assistance.
For example, existing safety standards are designed for vehicles that ultimately have a human driver responsible for safe operation. In contrast, UL 4600 deals with full autonomy head-on. The draft standard explains, “complete removal of humans from performing aspects (including supervision) of autonomous item operation brings with it numerous additional concerns.” UL 4600 addresses these “additional concerns.”
Safety Standard Landscape (Source: Phil Koopman, Edge Case Research)
Making a safety case
The 295-page draft document departs from others by emphasizing the engineering rigor required to build a safety case for autonomous “items,” because autonomous systems will absolutely not be limited to AVs.
“The safety case,” the draft UL 4600 standard stipulates, “includes a structured set of goals, argument, and evidence supporting the proposition that the item is acceptably safe for deployment. In support of that goal, UL 4600 assessments emphasize ensuring that the safety case is reasonably complete and well formed.”
UL 4600 covers a variety of categories that affect the safety of autonomous systems. They include interaction with humans, autonomy functions and support to software and the system engineering process, life cycle concerns and maintenance of autonomous items. In each category, UL 4600 lists the safety arguments that must be made — either as mandatory, required, highly recommended and recommended. UL 4600 also shares at each level “examples” of safety hazards, risks and scenarios that must be considered before designers make a safety case for their systems.
Did You Think of That?
When fully autonomous vehicles strike out on their own on public roads, it’s not hard to imagine the million-odd things that could go wrong, or the million details that hadn’t previously been thought of by system engineers. Each represents an unintended consequence that could derail the very foundations of the AV safety.
As described in the UL 4600 draft:
An autonomous item’s behavior might be different than (but compatible with) human operator behaviors and might need to deal with situations a human operator would not normally experience. Moreover, there might be an expectation that an autonomous item successfully handles a wide variety of exceptional or unusual situations beyond the normal expectations of human operator proficiency.
…When autonomous vehicles are driving with no humans running interference for them, all bets are off.
The examples shared in UL 4600 pose the question that Koopman typically asks AV designers, “Did you think of that?”
Put simply, these examples can offer developers “ten things [actually a lot more] you need to worry about “when you design the safety of your AV.”
Koopman said that many of the examples (already in UL 4600) derive from real-world stories. He said, “they are not only mine, but also Uma Ferrell and Frank Fratrik.” Ferrell, currently a system engineer at Mitre Corp., is a former consultant DER (Designated Engineering Representative) for the Federal Aviation Administration (FAA).
Koopman said, “Uma has extensive experience in aircraft certification, and Frank (a lead engineer at Edge Case Research) has expertise in military system safety assessment.”
By no means does anyone believe the draft UL 4600 covers every contingency. Koopman expects more examples and more bins via review. “After the first version is released, we'll get more,” especially once it is applied to systems, he said. “The standard will be part of UL's ‘continuous maintenance’ program, which means we'll have the opportunity to add those new bins and new examples during periodic updates. Also, developers can add bins on their own to their local safety case without waiting for the standard to be updated.”
UL 4600 Explained (Source: Phil Koopman)
Nine months in making
The first iteration of UL 4600 draft — a 10-page outline — was written by Koopman in mid-December last year, then marked up by Ferrell and Fratrik with a lot of red ink.
The first fully-formed draft went in May to standard technical panel (STP) with 35 voting members. The panel included such chip vendors as Renesas, Intel and Infineon, and commercial AV users and developers. Among them are Uber, Nio, Bosch, Argo AI and Aurora. Both the U.S. Department of Transportation (DoT) and Pennsylvania DoT are sending representatives are also part of the STP. The Panel also includes three insurance companies: AXA, Liberty Mutual and Munich Re America.
In June, the STP met for the first time in person for review and discussion of the initial draft.
The UL 4600 draft standard in its current form, released Wednesday to stakeholders (who don’t have voting rights but have comment rights), is “version four,” the standard that has survived after discussion, modification and articulation through three separate comment periods. The UL 4600 group has already received close to 1,000 comments.
UL is moving fast. Deborah Prince, UL’s program manager, told EE Times , that the team expects UL 4600 to go to ballot in December 2019. By the first quarter of 2020 it will become the ANSI standard. “Once the documents and the technology become stable,” she said, “I don’t see why this won’t become an ISO standard.” After all, “safety must be global,” she added.
With “a built-in” feedback loop installed inside UL 4600, how often is UL planning to update the UL 4600?
Prince said, “Given that a lot of new technologies are inside the system, we would let people use the new standard for about a year first and then have the STP re-evaluate it…to see if we missed something big.”
Recommended
You Say Your AV Is Safe? Show Me
Highlights of UL 4600
In Koopman’s mind, several principles make UL 4600 particularly important for ensuring safety in autonomous systems.
First, UL uses “a safety case as the overarching approach,” he noted. “Other standards have also moved in this direction recently, such as the way FDA developed safety standards for infusion pumps.”
Second, UL 4600 uses “feedback loops at all levels,” he noted. These apply to “design improvement [and] safety case improvement” while collecting field feedback, thus constantly improving the standard.
Third, the standard is “designed for objective, repeatable assessment.”
Fourth, UL 4600 is a “repository for community knowledge, collecting #DidYouThinkofThat lists without having to share raw system data,” noted Koopman.
Fifth, the standard also offers “specific coverage of machine learning.”
Finally, UL 4600 provides “compatibility with functional and other accepted safety standards,” he concluded.
Safety of autonomous systems
Many observers, both experts and “outsiders,” worry about the safety of fast-emerging autonomous systems. The industry’s desire to get UL 4600 done is so strong that many members within the STP have been “really engaged in the discussions” and “surprisingly constructive and collaborative,” said UL’s Prince.
As Koopman summed it all up, “The regulator isn’t going to make their system safe. They [AV companies] have to own it, and there is no way around it.”
UL 4600’s registered stakeholders can submit comments to the draft standard until November 1st . Asked about those who have not yet registered but want to become stakeholders now, Prince said, “E-mail me.”
Deborah Prince can be reached at:
— Junko Yoshida, Global Co-Editor-In-Chief, AspenCore Media, Chief International Correspondent, EE Times