**Recent Clients**

BayesiaLab Seminar in New York City

# The Delphi Method Revisited:

Knowledge Elicitation & Reasoning with Bayesian Networks

New York University Kimmel Center

60 Washington Square South, Classroom 803, New York, NY 10012

In this half-day workshop, we demonstrate how we can—in the absence of data—elicit human knowledge for developing a high-dimensional computational model of a problem domain. We build on knowledge elicitation techniques that were first utilized in RAND Corporation's Delphi Method in the 1950s and 60s. Furthermore, we employ Bayesian networks as the modeling framework for representing the elicited knowledge.

## The Seminar at a Glance

### Motivation

- Substantive question: how big is the electric vehicle market in 2025?
- Methodological question: how to reason formally without data?

### Knowledge Representation

- An overview of analytic modeling (theory vs. data; association vs. causation).
- Introducing the Bayesian network paradigm as a modeling and reasoning framework.
- Introducing the BayesiaLab software platform.

### Knowledge Elicitation

- History: RAND Corporation's Delphi Method from the 1950s.
- Reinventing Delphi: the new web-based Bayesia Expert Knowledge Elicitation Environment (BEKEE).
- The BEKEE workflow: brainstorm in a group, elicit knowledge individually, provide feedback, repeat.

### Case Study

- Review of context and current EV technology background.
- Brainstorming session to create a qualitative structure of problem domain.
- Knowledge elicitation exercise to obtain assumptions about problem domain in 2025.
- Simulation of the (elicited) Bayesian network model of the domain for policy analysis.
- Distribution of final model to external stakeholders via the BayesiaLab WebSimulator.

## The Program in Detail

On the conceptual basis of both the Delphi Method and Bayesian networks, we introduce a new web-based tool, the **Bayesia Expert Knowledge Elicitation Environment (BEKEE)**, for eliciting knowledge from experts, in our case, the seminar participants. As our case study topic, we attempt to assess the future market potential for electric vehicles. Given that little in-market data exists for electric vehicles and that their technology is rapidly evolving, we have to rely exclusively on our assumptions regarding future conditions and capabilities.

We shall see that systematically eliciting and encoding numerous pieces of speculative knowledge into a Bayesian network can produce a remarkably useful model of the domain. In a way, we leverage the "wisdom of crowds" to quantify many hypothesized interactions within our high-dimensional problem domain. As a result, the participants' collective belief is represented in a Bayesian network, which provides us with a common framework for reasoning about the problem domain. Thus, we can infer numerous quantities of interest, such as a volume forecast or the effect of incentives.

Importantly, the probabilistic nature of Bayesian networks allows us to preserve the uncertainty that exists in the participants' beliefs. Any inferred quantify will naturally reflect these uncertainties and provide a sense of the collective confidence in the obtained estimates.

## Knowledge Elicitation Exercise: The Potential of Electric Vehicles

Back in 2011, President Obama predicted that by 2015 there would 1 million electric vehicles on the road in the U.S. As it turns out, less than 30% of that target was achieved by 2015. Today, the number of electric vehicles being sold remains tiny considering the size of the overall vehicle market. It is easy to come up with a long list of reasons for why sales are so low: limited range, inadequate charging infrastructure, long charging time, high battery cost, relatively low fuel prices, etc.

### What's the True Potential?

More important than looking for what currently constricts electric vehicle sales is determining what would happen under counterfactual conditions: what if the electric vehicle range were better, what if there were more charging points, what if batteries were cheaper, what if fuel prices were higher? Such counterfactual conditions might help us understand the ultimate volume potential of electric vehicles. That is the quantity we clearly wish to know for making sound decisions about long-term investments in infrastructure and product development.

### No Data = No Model = No Science?

Of course, if we had a mathematical model describing all the dynamics in this problem domain, we could compute various scenarios and come up with a volume forecast. However, we don't have any data from which we could generate such a model. Currently, none of the available electric vehicles are close to the technical specifications that we may wish to consider as realistic scenarios for the future.

Without data, and without a mathematical model generated from such data, are we, then, stuck with mere speculation? Is it even possible to come up with reasonably objective volume estimate?

In this day and age of "Big Data," we may be led to believe that facts can only be established from data, especially in the context of a scientific inquiry. This is a misconception. Even without data, humans do often possess

### The Wisdom of Crowds

Although there may not be a single expert who can fully comprehend the entire domain, there may be numerous individuals, or stakeholders, who are more or less knowledgeable about different parts of the puzzle. It is our objective to break down the overall problem domain into numerous simpler questions, which are perhaps more easily "knowable," at least to some.

So, we are not looking for a single authoritative opinion. Rather, we are looking to collect and consolidate the full spectrum of thought, including causal relationships, from a number of stakeholders. This is where the "wisdom of crowds" comes into play. We want stakeholders to provide their individual and independent assessments of different elements and relationships within the problem domain.

### Knowledge Elicitation with the Bayesia Expert Knowledge Elicitation Environment

While the objective of collecting multiple opinions is straightforward, there are many technical and practical challenges in terms of implementing such a process. In this seminar, we propose employing the **Bayesia Expert Knowledge Elicitation Environment (BEKEE)** for this purpose. More specifically, we plan to utilize BEKEE for collecting the opinions of our seminar participants about the forces that affect the electric vehicle market, such as the issues mentioned above. We certainly don't expect to have any bona fide experts in this field. Rather, we anticipate a fairly diverse range of opinions given the presumably wide range of backgrounds of the participants.

### Spreadsheets are Deterministic, Opinions are Probabilistic

One may be tempted to think that the knowledge collected via BEKEE can be assembled in a spreadsheet so that various conditions can be simulated. Unfortunately, spreadsheets are of little use in this context. A multitude of opinions cannot be neatly translated into spreadsheet formulas.

Quite naturally, combining multiple assessments across numerous dimensions produces uncertainty, which cannot be directly encoded in a spreadsheet. A Bayesian network, on the other hand, can explicitly capture the uncertainty generated from the diversity of opinions. Using the BEKEE platform, we can directly encode the elicited knowledge, along with all the associated uncertainty, into a Bayesian network using BayesiaLab software platform.

### Reasoning with Bayesian Networks and BayesiaLab

On the basis of the newly-generated Bayesian network, which represents the collective knowledge of the seminar participants, we can use the simulation tools in BayesiaLab to reason probabilistically about the implications of various hypothetical conditions. Thus, the collective knowledge will produce a volume estimate, i.e. the principal quantity of interest in this seminar.

Furthermore, we can use the same Bayesian network model to reason about the causal effects of hypothetical policy mandates and other government interventions beyond the natural equilibrium state of the electric vehicle market.

### Seminar Certificate

Participants who complete this seminar program will automatically receive a BayesiaLab certificate after the event. All BayesiaLab badges and digital credits are managed via Credly, which allows you to accept your credits and share them via social media platforms.

Please remember that Bayesian network skills are in high demand these days, so don't delay in claiming and posting your credits!

### Event Format

Throughout the seminar, we alternate slide presentations and practical software demonstrations. We encourage a lively dialogue throughout the seminar, so there is plenty of opportunity for Q&A. Also, we'll have two coffee breaks for networking and offline questions.

## What is BEKEE?

Everybody is talking about "Big Data" and all the opportunities that are associated with it. Very often, though, we hear almost as much about the challenges that come with this flood of data. Where to store it, how to analyze it, how to explain it, the list goes on and on. We think this is a very nice problem to have. Much more serious problems exist on the opposite end of the spectrum, where there is not enough data. Unfortunately, all the advanced knowledge discovery algorithms fail in the absence of data.

In over ten years of continuous development, and in increasingly sophisticated ways, BayesiaLab has permitted deriving knowledge from data through its machine learning algorithms, very much in the spirit of understanding "Big Data." However, BayesiaLab has maintained an equal focus on managing knowledge that exists beyond measurable and countable data points, such as the knowledge contained in the human mind. BayesiaLab's graphical user interface has made it highly intuitive for individual subject matter experts to encode their own domain understanding into a Bayesian network, thus capturing what they explicitly or implicitly know. What is especially important, one can very easily and formally capture causal directions in a Bayesian network graph, which is something that few other frameworks can do.

However, when it comes to consolidating the collective knowledge from a group of experts, rather than from an individual, the process is not as straightforward any longer. Traditionally, one would perhaps bring the experts together in a brainstorming session and let them form a common understanding. Subsequently, such a consensus could be encoded manually. However, brainstorming sessions are prone to introducing a wide range of biases, which can be disastrously counterproductive in studying complex domains.

Bayesia Expert Knowledge Elicitation Environment, or BEKEE for short, is a new web application that is designed to minimize detrimental group biases. The central idea is not to coerce consensus, but rather to elicit everyone's individual views regarding the domain under study. To ensure the independent elicitation of probabilities, BEKEE queries stakeholders individually via an interactive questionnaire linked to the core BayesiaLab application. Retrieving expert views in such a fashion generates many "parallel universes" in terms of domain understanding. These different perspectives can be formally compared by the facilitator and potentially returned to the group for a formal debate in the case of seriously conflicting assessments.

In most cases, this is an iterative process and, even if stakeholder opinions do not converge, BayesiaLab will compile all views and produce a unifying Bayesian network. This graph is now the mathematically correct summary of all the available expert opinions. As such, it can be utilized as a formal representation of the underlying domain. Most importantly, this graph is not merely a qualitative illustration. Rather, a Bayesian network is a fully computable model of the domain, which immediately facilitates the simulation of what-if scenarios.

In fact, we can evaluate this Bayesian network model the same way as a statistical model estimated from "Big Data." One might still prefer a data-based model if data were indeed available, but in the absence thereof, the formally-encoded collective expert knowledge best represents what is known at the time.

## The Motivation for BEKEE

### Complexity & Cognitive Challenges

It is presumably fair to state that reasoning in complex environments creates cognitive challenges for humans. Adding uncertainty to our observations of the problem domain, or even considering uncertainty regarding the structure of the domain itself, makes matters worse. When uncertainty blurs so many premises, it can be particularly difficult to find a common reasoning framework for a group of stakeholders.

### No Data, No AI?

It is presumably fair to say that Artificial Intelligence is nowadays perceived to be associated with Big Data. We, however, plan to employ AI at the opposite end of the spectrum, where we have no data. More specifically, we propose using Bayesian networks as a form of Artificial Intelligence that makes formal reasoning and optimization possible under such conditions.

### No Data, No Analytics?

If we had hard observations from our domain in the form of data, it would be quite natural to build a traditional analytic model for decision support. However, the real world often yields only fragmented data or no data at all. It is not uncommon that we merely have the opinions of individuals who are more or less familiar with the problem domain.

### To an Analyst With Excel, Every Problem Looks Like Arithmetic.

In the business world, it is typical to use spreadsheets to model the relationships between variables in a problem domain. Also, in the absence of hard observations, it is reasonable that experts provide assumptions instead of data. Any such expert knowledge is typically encoded in the form of single-point estimates and formulas. However, using of single values and formulas instantly oversimplifies the problem domain: firstly, the variables, and the relationships between them, become deterministic; secondly, the left-hand side versus right-hand side nature of formulas restricts inference to only one direction.

### Taking No Chances!

Given that cells and formulas in spreadsheets are deterministic and only work with single-point values, they are well suited for encoding “hard” logic, but not at all for “soft” probabilistic knowledge that includes uncertainty. As a result, any uncertainty has to be addressed with workarounds, often in the form of trying out multiple scenarios or by working with simulation add-ons.

### It Is a One-Way Street!

The lack of

Assuming that A1 is the cause, and B1 is the effect, we can indeed use a spreadsheet for inference in the causal direction, i.e. perform a simulation. However, even if we were certain about the causal direction between them, unidirectionality would remain a concern. For instance, if we were only able to observe the effect B1, we could not infer the cause A1, i.e. we could not perform a diagnosis from effect to cause. The one-way nature of spreadsheet computations prevents this.

### Bayesian Networks to the Rescue!

Bayesian networks are probabilistic by default and handle uncertainty “natively.” A Bayesian network model can work directly with probabilistic inputs, probabilistic relationships, and deliver correctly computed probabilistic outputs. Also, whereas traditional models and spreadsheets are of the form y=f(x), Bayesian networks do not have to distinguish between independent and dependent variables. Rather, a Bayesian network represents the entire joint probability distribution of the system under study. This representation facilitates

### Probabilistic Reasoning

On the basis of this newly-generated Bayesian network, we can reason probabilistically about the implications of actual observations or hypothetical scenarios. Of particular interest are diagnostic inference tasks, i.e. reasoning from observed effects back to the not-directly-observable cause. Our example illustrates that reasoning correctly about the given question can become intractable—both cognitively and mathematically—unless we employ Bayesian networks as a reasoning framework. The example also proves how valuable even vague and diverse opinions can become if we systematically elicit them from stakeholders and encode them in a Bayesian network.

### Optimization Under Uncertainty

In addition to ad hoc inference, we can use the Bayesian network in conjunction with BayesiaLab's search algorithms to identify the optimal course of action for achieving the desired outcome, given current conditions and uncertainties and while also taking into account any costs and utilities.

### Value of Information

Finally, the Bayesian network allows us to quantify the value of information and measure the consistency (or conflict) of different pieces of actual or hypothetical evidence. Most importantly, the network can identify those pieces of yet-to-be-observed evidence that would reduce our uncertainty the most or would have the greatest impact on determining the optimal course of action.

## Frequently Asked Questions

### What does it cost to participate?

Registration for this event is free of charge.

### Do I need to bring a computer?

Yes, you should bring a laptop or tablet so you can participate in the knowledge elicitation exercise. However, this is not at all mandatory.

### What educational background is required to understand the seminar program?

If you are proficient in college-level math and core statistical techniques, such as specifying and estimating multivariate regressions, you will find our seminar program useful.

### How should I prepare for the seminar?

We recommend that you read the first three chapters of our book on Bayesian networks. You can download it for free via this link: bayesia.com/book. Also, Chapter 4 in the book will give you a good preview of the case study to be presented in the seminar.

### Will the event be broadcast live?

No, you will need to be physically present to participate in the event. Also, the event will not be recorded.

### Will you serve lunch at the event?

No, we will not serve a meal at the seminar. Please have lunch on your own beforehand or feel free to bring your lunch.