QA-GNN [1] and GreaseLM [2] combine textual context with knowledge graph evidence for QA, while DRAGON [3] pretrains joint language–graph representations. ConceptNet [5] provides structured commonsense knowledge, and choice-based decision interfaces such as Jev [4] motivate candidate-defined answer spaces. Yet joint representation pretraining does not directly establish a reusable candidate-decision function. In particular, it remains unclear how KG relations can provide downstream-task-agnostic decision supervision, whether one model can generalize to previously unseen QA tasks and varying numbers of candidates, and whether its decision probabilities remain reliable. This thesis studies ConceptNet-specific pretraining without target-QA-task supervision, using shared candidate scoring to support transferable decision making. Research Questions RQ1 — Knowledge-Driven Decision Learning.How can ConceptNet relations and local structures support task-agnostic decision-pretraining objectives that teach a model to evaluate candidate answers using knowledge evidence? Relative to language-only training and generic self-supervised language–graph pretraining, what produces the observed gains? RQ2 — Transfer across QA Tasks. Can one pretrained decision model be applied to different commonsense QA tasks without using target-task labels? How is transfer affected by question format, knowledge coverage, and reasoning requirements, and what improvements arise from limited labeled adaptation? RQ3 — Generalization across Candidate Sets.</strong> Can the model preserve meaningful semantic decisions when the number of options, distractor difficulty, or answer wording changes? How can architectural support for variable-sized option sets be distinguished from empirical generalization to previously unseen candidate sets? RQ4 — Reliability of Decision Probabilities. Do predicted probabilities reflect actual reliability when answer-option sets and supporting knowledge change? Can task-agnostic training and calibration mitigate overconfidence during cross-task deployment?
Thesis Type |
|
Student |
Yilin Huang |
Status |
Running |
Presentation room |
Seminar room I5 - 6202 |
Supervisor(s) |
Stefan Decker |
Advisor(s) |
Yixin Peng |
Contact |
peng@dbis.rwth-aachen.de |
Tasks
a) Literature Review and Baseline Setup
- Review knowledge-enhanced QA, joint language–graph pretraining, and candidate-scoring decision models.
- Establish ConceptNet retrieval and reproducible language-only, language–graph, and DRAGON-style baselines [1–5].
b) ConceptNet-Based Decision Pretraining Data
- Construct task-agnostic decision examples from ConceptNet triples and paths, including variable-size option sets and challenging distractors.
- Create leakage-aware training and validation splits that account for ambiguous or missing KG relations.
c) Unified Language–Graph Decision Model
- Develop a language–graph model with shared candidate scoring that supports varying numbers of answer options.
- Combine representation-learning and decision-learning objectives, and compare their separate and joint effects.
d) Cross-Task Transfer and Generalization
- Evaluate a single pretrained checkpoint on CommonsenseQA, OpenBookQA, PIQA, and SocialIQA without target-task labels, alongside labeled-adaptation baselines [6–9].
- Test candidate-count and distractor variations and assess accuracy, label efficiency, and probability calibration.
e) Ablation, Robustness, and Failure Analysis
- Ablate graph structure and training objectives to isolate the contributions of KG evidence and decision learning.
- Analyze robustness to missing or noisy knowledge, candidate variation, and overconfident predictions.
References
[1] Yasunaga, M., et al.: QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering. In: NAACL-HLT, pp. 535–546 (2021). https://doi.org/10.18653/v1/2021.naacl-main.45
[2] Zhang, X., et al.: GreaseLM: Graph REASoning Enhanced Language Models for Question Answering. In: ICLR (2022). https://openreview.net/forum?id=41e9o6cQPj
[3] Yasunaga, M., et al.: Deep Bidirectional Language-Knowledge Graph Pretraining. In: Advances in Neural Information Processing Systems 35 (NeurIPS 2022). NeurIPS proceedings
[4] TypeSafe Jev Model Guide: Choice: Pick One. Independent documentation guide (accessed 24 October 2026). https://typesafe-jev.com/en/capabilities/choice/
[5] Speer, R., Chin, J., Havasi, C.: ConceptNet 5.5: An Open Multilingual Graph of General Knowledge. In: AAAI 31 (2017). https://doi.org/10.1609/aaai.v31i1.11164
[6] Talmor, A., et al.: CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. In: NAACL-HLT, pp. 4149–4158 (2019). https://doi.org/10.18653/v1/N19-1421
[7] Mihaylov, T., et al.: Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In: EMNLP, pp. 2381–2391 (2018). https://doi.org/10.18653/v1/D18-1260
[8] Bisk, Y., et al.: PIQA: Reasoning about Physical Commonsense in Natural Language. In: AAAI 34, pp. 7432–7439 (2020). https://doi.org/10.1609/aaai.v34i05.6239
[9] Sap, M., et al.: Social IQa: Commonsense Reasoning about Social Interactions. In: EMNLP-IJCNLP, pp. 4463–4473 (2019). https://doi.org/10.18653/v1/D19-1454
- Solid foundation in Machine Learning / Deep Learning and graph representation learning.
- Good programming skills in Python.
- Experience with PyTorch and graph-learning frameworks such as PyTorch Geometric or DGL.
- Familiarity with Transformers, attention mechanisms, and self-supervised learning.
- Basic knowledge of Knowledge Graphs, heterogeneous graphs, and temporal graph learning is beneficial.
- Experience with NLP models or text embeddings is helpful for representing textual attributes of academic entities.