Psychometric inventories provide a means to analyze the expression of a variety of human traits. When applied on specific sets of people, they may show shared characteristics that could contribute to their behavior. Common traits can be significant for criminal individuals, as they could be used to discern early warning signs or identifiers. For example, certain profiles of the Dark Triad psychometric test which is composed of psychopathy, machiavellianism, and narcissism, and self-control, have been related to antisocial and criminal externalizing outcomes. In this thesis, we explore the idea of applying psychometrics to generative LLMs that impersonate well-known criminals and dictators to circumvent problems of limited/restricted access to actual, and often historic, people.
Thesis Type |
|
Student |
Sena Raina |
Status |
Running |
Presentation room |
Seminar room I5 6202 |
Supervisor(s) |
Stefan Decker |
Advisor(s) |
Maximilian Kißgen |
Contact |
kissgen@dbis.rwth-aachen.de |
This is a cooperation between the chair and GESIS, the other supervisor will be Claudia Wagner
In this thesis we explore the idea of applying psychometrics on generative LLMs that impersonate well-known criminals and dictators. As this method is an approximation, biases of different models regarding certain features may occur, e.g. possibly more humane personas than real life suggests, and creating personas in some models might not be fully possible due to implemented guardrails or little available information about individuals.
Goals & Objectives:
- Compiling a list of selected historical figures for analysis. Additionally, a control group list of clearly separate “safe” individuals, for which no guardrails/behavior changes should be in place by the model.
- Setting up instances of open-source models and querying instances of commercial generative LLMs.
- Developing prompts that guide the models to respond as if they were the historical figures and evaluating the degree of variation in personas based on different prompts.
- Evaluating the LLM personas for character traits and features using psychometric inventories.
- Determining potential biases/hindrances, e.g. “western lense”, within the models that may influence their responses (e.g. guardrails are built in for some dictators and murders and criminals but not for all) and the generated personas.
Challenges:
It must be assessed whether creating personas is even possible or limited for all selected models/criminals. Additionally, stability of personas could vary for different prompts and even for the same prompt different responses may be generated. Multiple runs are therefore needed and results for inventories have to be averaged to create a general profile for a persona.
Related Literature:
AI Psychometrics: Assessing the psychological profiles of large language models through psychometric inventories
Personality Traits in Large Language Models
2305.18189.pdf (arxiv.org)
- Knowledge of Machine Learning Concepts: A general grasp of ML and LLM concepts
- Programming Skills: Python, R or similar languages to deal with ML applications and REST APIs
Nice to have:
- Experience in prompt engineering
- Experience in psychometrics and similar psychological inventories