Knowledge-augmented question answering (QA) combines language understanding with relational evidence from knowledge graphs (KGs). QA-GNN [1] and GreaseLM [2] model language–graph interactions, while DRAGON [3] learns joint representations through masked language modeling and KG link prediction. However, retrieved evidence can exceed practical text and graph budgets. Fixed-size inputs and sampling or truncation may discard useful relational paths, retain redundant evidence, and overlook the cost of retrieval and selection. It also remains unclear whether budget-aware evidence selection should be applied only downstream or incorporated into joint pretraining. This thesis investigates structure-preserving selection during pretraining and its impact on the end-to-end accuracy–cost trade-off. Research Questions RQ1 — Evidence Selection under Budget Constraints. How can relevance, redundancy, complementarity, and structural integrity be jointly considered when selecting knowledge within text and graph budgets? Which selection principles retain decision-relevant evidence more effectively than random sampling, sequential truncation, or relevance-based ranking? RQ2 — Selection and Joint Pretraining. Does incorporating budget-aware selection into language–graph pretraining outperform compressing inputs only at the downstream stage? How do self-supervised objectives, cross-modal interaction, and training-budget settings influence the outcome? RQ3 — Accuracy and End-to-End Cost. After accounting for retrieval, evidence selection, and model computation, can the approach attain higher accuracy at a fixed resource budget or comparable performance at lower cost? What complementary or substitutive effects arise between text and graph budgets? RQ4 — Effective Conditions and Failure Modes. How do the benefits vary with question type, context length, knowledge coverage, and local graph structure? When does compression eliminate irrelevant information, and when does it remove essential evidence?
Thesis Type |
|
Student |
Xianhong Bao |
Status |
Running |
Presentation room |
Seminar room I5 - 6202 |
Supervisor(s) |
Stefan Decker |
Advisor(s) |
Yixin Peng |
Contact |
peng@dbis.rwth-aachen.de |
Tasks
a) Literature Review and Baseline Setup
- Review knowledge-augmented QA, joint language–graph pretraining, and resource-constrained knowledge selection.
- Reproduce DRAGON-based and language-only baselines on CommonsenseQA and OpenBookQA using ConceptNet [3–6].
b) Budget-Aware Evidence Selection
- Define text-token, graph-node, and graph-edge budgets and implement a selection method that accounts for relevance, redundancy, and relational structure.
- Compare the method with random selection, truncation, and relevance-based ranking under matched budgets.
c) Budget-Aware Joint Pretraining
- Integrate evidence selection into DRAGON-style language–graph pretraining and compare pretraining-time versus downstream-only selection.
- Evaluate the effects of pretraining objectives and text–graph budget settings while controlling for model capacity.
d) Accuracy–Cost Evaluation
- Evaluate QA accuracy and end-to-end costs, including retrieval, selection, memory usage, and inference latency, across representative budgets.
- Compare accuracy–cost trade-offs on CommonsenseQA and OpenBookQA, with optional extension to further QA tasks.
e) Ablation and Failure Analysis
- Ablate selection components and pretraining objectives, and analyze retained evidence, structural integrity, and failure cases.
- Report robustness across relevant question and graph characteristics and multiple random seeds.
References
[1] Yasunaga, M., et al.: QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering. In: NAACL-HLT, pp. 535–546 (2021). https://doi.org/10.18653/v1/2021.naacl-main.45
[2] Zhang, X., et al.: GreaseLM: Graph REASoning Enhanced Language Models for Question Answering. In: ICLR (2022). https://openreview.net/forum?id=41e9o6cQPj
[3] Yasunaga, M., et al.: Deep Bidirectional Language-Knowledge Graph Pretraining. In: Advances in Neural Information Processing Systems 35 (NeurIPS 2022). NeurIPS proceedings
[4] Speer, R., Chin, J., Havasi, C.: ConceptNet 5.5: An Open Multilingual Graph of General Knowledge. In: AAAI 31 (2017). https://doi.org/10.1609/aaai.v31i1.11164
[5] Talmor, A., et al.: CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. In: NAACL-HLT, pp. 4149–4158 (2019). https://doi.org/10.18653/v1/N19-1421
[6] Mihaylov, T., et al.: Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In: EMNLP, pp. 2381–2391 (2018). https://doi.org/10.18653/v1/D18-1260
- Solid foundation in Machine Learning / Deep Learning and graph representation learning.
- Good programming skills in Python.
- Experience with PyTorch and graph-learning frameworks such as PyTorch Geometric or DGL.
- Familiarity with Transformers, attention mechanisms, and self-supervised learning.
- Basic knowledge of Knowledge Graphs, heterogeneous graphs, and temporal graph learning is beneficial.
- Experience with NLP models or text embeddings is helpful for representing textual attributes of academic entities.