DBIS

Background

Academic Knowledge Graphs are naturally heterogeneous and temporal. They contain multiple node types, such as papers, authors, venues, institutions, and fields of study, together with typed relations including authorship, citation, publication venue, field assignment, and affiliation. Their structure also evolves over time as new publications, citations, collaborations, affiliations, and research topics emerge [1,9,10]. Consequently, representation learning for AKGs must account simultaneously for schema heterogeneity, temporal evolution, textual attributes, and transfer across tasks or graph sources.

The Heterogeneous Graph Transformer (HGT) provides an important architectural basis for this setting through node- and edge-type-dependent attention, heterogeneous graph sampling, and relative temporal encoding [1]. Beyond task-specific heterogeneous graph learning, recent work on Graph Foundation Models explores whether transferable representations can generalize across graph domains and downstream tasks. GFT, for example, explicitly studies cross-task and cross-domain transfer through a transferable graph vocabulary [2]. These developments motivate investigating whether similar foundation-model principles can be specialized to temporal, text-attributed Academic Knowledge Graphs.

Graph self-supervised learning provides several complementary pre-training paradigms relevant to this goal. Node- and graph-level pre-training objectives have demonstrated that carefully designed self-supervision can improve downstream transfer [3]. GPT-GNN jointly models node attributes and graph structure through generative pre-training and has been evaluated on billion-scale academic graphs [4]. GraphMAE and GraphMAE2 instead use masked feature reconstruction and latent representation prediction to learn transferable graph representations [5,6]. For text-attributed heterogeneous graphs, topology-aware language-model pre-training provides a mechanism for jointly capturing textual and graph-structural information [7]. In parallel, ULTRA demonstrates that relation-level representations can support inductive transfer across previously unseen knowledge graphs [8]. Together, these directions provide the methodological basis for investigating a unified temporal heterogeneous pre-training framework for Academic Knowledge Graphs.

Tasks

a) Baseline reproduction and data/schema setup

• Construct or select a unified Academic Knowledge Graph representation from the available data, including relevant entity types such as papers, authors, venues, fields of study, institutions, and keywords.
• Represent typed relations such as authorship, citation, publication venue, paper–field assignment, and affiliation, and associate temporal information such as publication or citation time whenever available.
• Prepare textual attributes, including paper titles, abstracts, venue names, field labels, and keyword descriptions, using language-model encoders or pre-computed text representations where appropriate.
• Reproduce and analyze representative heterogeneous graph and graph pre-training baselines, including HGT, GPT-GNN, GraphMAE, and GraphMAE2 [1,4–6].
• Include a text-attributed heterogeneous graph pre-training baseline where feasible [7].

b) Analyze pre-training and adaptation paradigms

• Study type-specific projections, relation-aware attention, heterogeneous mini-batch sampling, and temporal encoding for scalable modeling of Academic Knowledge Graphs [1].
• Analyze generative and masked graph pre-training objectives, including attribute reconstruction, edge generation, masked feature reconstruction, and latent representation prediction [3–6].
• Investigate how textual information can be combined with heterogeneous graph topology, comparing text-embedding initialization with topology-aware text pre-training where feasible [7].
• Examine which components of these paradigms are transferable across graph schemas, tasks, and temporal settings, drawing on graph-foundation-model and knowledge-graph transfer principles [2,8].

c) Develop a temporal heterogeneous graph pre-training framework

• Design a unified model that combines heterogeneous graph structure, relation types, textual node attributes, and temporal context.
• Adapt suitable self-supervised objectives from generative graph pre-training, masked graph modeling, and text-attributed graph learning to the temporal heterogeneous setting [3–7].
• Investigate strategies for producing representations that can be reused across different downstream prediction tasks without training an entirely separate representation model for each task.
• Where applicable, study mechanisms that improve transfer across related Academic Knowledge Graph schemas or graph sources [2,8].

d) Train, evaluate, and analyze transfer behavior

• Pre-train the proposed model on selected Academic Knowledge Graph data and fine-tune or probe the learned representations on downstream tasks.
• Evaluate tasks such as paper field classification, venue prediction, paper–paper citation prediction, author–paper relation prediction, and temporal future-link prediction, depending on the labels and temporal information available in the selected data.
• Compare the pre-trained model with training-from-scratch variants and representative task-specific or pre-trained graph baselines.
• Evaluate cross-task transfer by adapting the same pre-trained representation model to different downstream tasks.
• Where compatible graph schemas and data are available, evaluate cross-dataset transfer among arXiv-, DBLP-, MAG-, or OAG-derived graphs.
• Use task-appropriate metrics, such as Accuracy and Macro/Micro-F1 for classification and MRR and Hits@K for link prediction; temporal evaluation should respect chronological data splits where applicable.
• Analyze robustness, transfer behavior, and computational efficiency, including relevant model size, memory, training, fine-tuning, and inference characteristics.

References

[1] Hu, Z., Dong, Y., Wang, K., Sun, Y.: Heterogeneous Graph Transformer. In: Proceedings of The Web Conference 2020, pp. 2704–2710. ACM, New York (2020). https://doi.org/10.1145/3366423.3380027

[2] Wang, Z., Zhang, Z., Chawla, N.V., Zhang, C., Ye, Y.: GFT: Graph Foundation Model with Transferable Tree Vocabulary. In: Advances in Neural Information Processing Systems 37 (NeurIPS 2024) (2024). https://doi.org/10.52202/079017-3412

[3] Hu, W., Liu, B., Gomes, J., Zitnik, M., Liang, P., Pande, V., Leskovec, J.: Strategies for Pre-training Graph Neural Networks. In: International Conference on Learning Representations (ICLR 2020) (2020). https://openreview.net/forum?id=HJlWWJSFDH

[4] Hu, Z., Dong, Y., Wang, K., Chang, K.-W., Sun, Y.: GPT-GNN: Generative Pre-Training of Graph Neural Networks. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1857–1867. ACM, New York (2020). https://doi.org/10.1145/3394486.3403237

[5] Hou, Z., Liu, X., Cen, Y., Dong, Y., Yang, H., Wang, C., Tang, J.: GraphMAE: Self-Supervised Masked Graph Autoencoders. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 594–604. ACM, New York (2022). https://doi.org/10.1145/3534678.3539321

[6] Hou, Z., He, Y., Cen, Y., Liu, X., Dong, Y., Kharlamov, E., Tang, J.: GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner. In: Proceedings of the ACM Web Conference 2023, pp. 737–746. ACM, New York (2023). https://doi.org/10.1145/3543507.3583379

[7] Zou, T., Yu, L., Huang, Y., Sun, L., Du, B.: Pretraining Language Models with Text-Attributed Heterogeneous Graphs. In: Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10316–10333. Association for Computational Linguistics, Singapore (2023). https://doi.org/10.18653/v1/2023.findings-emnlp.692

[8] Galkin, M., Yuan, X., Mostafa, H., Tang, J., Zhu, Z.: Towards Foundation Models for Knowledge Graph Reasoning. In: International Conference on Learning Representations (ICLR 2024) (2024). ICLR 2024 Proceedings

[9] Hu, W., Fey, M., Zitnik, M., Dong, Y., Ren, H., Liu, B., Catasta, M., Leskovec, J.: Open Graph Benchmark: Datasets for Machine Learning on Graphs. In: Advances in Neural Information Processing Systems 33 (NeurIPS 2020) (2020). NeurIPS 2020 Proceedings

[10] Zhang, F., Liu, X., Tang, J., Dong, Y., Yao, P., Zhang, J., Gu, X., Wang, Y., Shao, B., Li, R., Wang, K.: OAG: Toward Linking Large-scale Heterogeneous Entity Graphs. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 2585–2595. ACM, New York (2019). https://doi.org/10.1145/3292500.3330785