Causal Transformers: Automated Causal Detection

The GPT in ChatGPT stands for Generative Pre-trained Transformer. This technology allows machines to learn from unlabeled text [1]. Estimand’s CR-2 model uses rule-based learning to detect causal relationships — a form of labeling. However, we are working to apply similar transformer methods to automate causal discovery. This is an account of our current research.

Deep learning has instead given us machines with truly impressive abilities but no intelligence. The difference is profound and lies in the absence of a model of reality. — Judea Pearl

Why it matters: When machines can accurately detect causal relationships without human algorithms, they move closer to understanding the world [2]. This prerequisite model of reality leads to machine intelligence — our ultimate goal.

What’s a transformer: GPTs are artificial neural networks pre-trained on large data sets of unlabelled text to generate human-like content. The transformer changes an input sequence into an output using context [3].

For example, consider this prompt: “What day is today?” The transformer model uses an internal mathematical representation that identifies the relevancy and association between the words day and today. It uses that knowledge to generate the output: “Today is Sunday.”

Before transformers: The first attempts at machine learning involved natural language processing (NLP), where computers use probability to determine the next word in a sequence [3]…