Uncertainty-Aware Reinforcement Learning for Efficient Adaptive Educational Assessment
DOI:
https://doi.org/10.54097/xt7mc543Keywords:
Computerized Adaptive Testing, Reinforcement Learning, Uncertainty Calibration, Assumed-Density Filtering, Optimal Stopping, Reward Shaping, Knowledge TracingAbstract
Adaptive assessment is efficient only if the system knows how uncertain it is about the learner, because that uncertainty decides which question is worth asking and when to stop. Reinforcement-learning (RL) assessment agents often derive it from Monte Carlo (MC) dropout over a neural knowledge encoder and reward its reduction. We study uncertainty-aware RL for efficient assessment by combining a calibration-aware Bayesian knowledge-state filter (assumed-density filtering on a probit item-response model with knowledge-component deviations and item-calibration noise), a target-aware information criterion with a closed-form expected covariance contraction, and a PPO-trained termination policy with an uncertainty-based state, a per-question cost, a policy-invariant shaping reward and a cross-fitted training environment. On ASSISTments 2009 and Junyi Academy, the selector significantly outperforms random selection and, from ten questions on, a re-implemented HARLA-ED operational agent under both a Bayesian and the agent's own neural estimator (NLL 0.552 vs. 0.558 after 20 questions on ASSISTments) and is statistically tied with mutual-information selection. In our re-implementation, the agent’s MC-dropout uncertainty grows as responses accumulate (0.10 to 0.18), its uncertainty-reduction reward did not improve validation likelihood in any run, and stopping on it needs 1.7–2.5 times as many questions as a fixed-length test of equal accuracy. The learned termination policy matches fixed-length efficiency (5.1–5.4 vs. 5 questions) without threshold tuning, and a value-of-information analysis explains why larger learner-specific savings are hard to obtain in replay protocols.
Downloads
References
[1] van der Linden, W. J., & Glas, C. A. W. (Eds.). (2010). Elements of adaptive testing. Springer. https://doi.org/10.1007/978-0-387-85461-8.
[2] Wang, J., Ge, Y., Liu, J., & Jiang, B. (2024). Early warning of liquidity stress in U.S. banks with seasonally aligned frailty hazards and delayed-feedback conformal alarms. Journal of Trends in Finance and Economics, 1(1), 85–100.
[3] Zhuang, Y., Liu, Q., Huang, Z., Li, Z., Shen, S., & Ma, H. (2022). Fully adaptive framework: Neural computerized adaptive testing for online education. Proceedings of the AAAI Conference on Artificial Intelligence, 36(4), 4734–4742. https://doi.org/10.1609/aaai.v36i4.20399.
[4] Wang, H., Long, T., Yin, L., Zhang, W., Xia, W., Hong, Q., Xia, D., Tang, R., & Yu, Y. (2023). GMOCAT: A graph-enhanced multi-objective method for computerized adaptive testing. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining.
[5] Liu, Q. (2024). Survey of computerized adaptive testing: A machine learning perspective. arXiv:2404.00712.
[6] Doroudi, S., Aleven, V., & Brunskill, E. (2019). Where’s the reward? A review of reinforcement learning for instructional sequencing. International Journal of Artificial Intelligence in Education, 29(4), 568–620. https://doi.org/10.1007/s40593-019-00187-x.
[7] Gal, Y., & Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning (pp. 1050–1059). PMLR.
[8] Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (pp. 1321–1330). PMLR.
[9] Bock, R. D., & Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6(4), 431–444. https://doi.org/10.1177/014662168200600405.
[10] Choi, S. W., Grady, M. W., & Dodd, B. G. (2011). A new stopping rule for computerized adaptive testing. Educational and Psychological Measurement, 71(1), 37–53.
[11] Ng, A. Y., Harada, D., & Russell, S. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the 16th International Conference on Machine Learning (pp. 278–287).
[12] Chang, H.-H., & Ying, Z. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20(3), 213–229. https://doi.org/10.1177/014662169602000303.
[13] van der Linden, W. J. (1998). Bayesian item selection criteria for adaptive testing. Psychometrika, 63(2), 201–216.
[14] Segall, D. O. (1996). Multidimensional adaptive testing. Psychometrika, 61(2), 331–354.
[15] van der Linden, W. J., & Glas, C. A. W. (2000). Capitalization on item calibration error in adaptive testing. Applied Measurement in Education, 13(1), 35–53.
[16] Bi, H., Ma, H., Huang, Z., Yin, Y., Liu, Q., Chen, E., Su, Y., & Wang, S. (2020). Quality meets diversity: A model-agnostic framework for computerized adaptive testing. Proceedings of the IEEE International Conference on Data Mining, 42–51.
[17] Piech, C., Bassen, J., Huang, J., Ganguli, S., Sahami, M., Guibas, L. J., & Sohl-Dickstein, J. (2015). Deep knowledge tracing. Advances in Neural Information Processing Systems, 28, 505–513.
[18] Wang, F., Liu, Q., Chen, E., Huang, Z., Chen, Y., Yin, Y., Huang, Z., & Wang, S. (2020). Neural cognitive diagnosis for intelligent education systems. Proceedings of the AAAI Conference on Artificial Intelligence, 34(4), 6153–6161. https://doi.org/10.1609/aaai.v34i04.6080.
[19] Zhou, F. (2026). Moment-constrained side-force allocation for wind-assisted merchant ships: Hydrodynamic performance of retractable anti-leeway fins and flap rudders. Innovation and Technology Studies, 3(1), 27–41.
[20] Bickford Smith, F., Kirsch, A., Farquhar, S., Gal, Y., Foster, A., & Rainforth, T. (2023). Prediction-oriented Bayesian active learning. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics (pp. 7331–7348). PMLR.
[21] Minka, T. P. (2001). Expectation propagation for approximate Bayesian inference. In Proceedings of the 17th Conference on Uncertainty in Artificial Intelligence (pp. 362–369).
[22] Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46(4), 443–459. https://doi.org/10.1007/BF02293801.
[23] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv:1707.06347.
[24] Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015). High-dimensional continuous control using generalized advantage estimation. arXiv:1506.02438.
[25] Fang, Q., & Liu, W. (2026). HARLA-ED: Resolving information asymmetry and enhancing algorithmic symmetry in intelligent educational assessment via hybrid reinforcement learning. Symmetry, 18(1), 58. https://doi.org/10.3390/sym18010058.
[26] Teng, D. (2025). RSRBench: Reproducibility and performance benchmarking for regional seismic risk workflows. Academic Journal of Architecture and Civil Engineering, 3(1), 40–52.
[27] Feng, M., Heffernan, N., & Koedinger, K. (2009). Addressing the assessment challenge with an online system that tutors as it assesses. User Modeling and User-Adapted Interaction, 19(3), 243–266.
[28] Chang, H.-S., Hsu, H.-J., & Chen, K.-T. (2015). Modeling exercise relationships in e-learning: A unified approach. Proceedings of the 8th International Conference on Educational Data Mining.
[29] Guo, Z. (2026). Closed-Loop Audio Neuroregulation for ADHD: Graph-Based Attention Decoding and Rhythm Entrainment. IEEE Access. https://doi.org/10.1109/ACCESS.2026.3731552.
[30] Ren, S. (2025). ARCS: Adaptive Reinforcement Learning Framework for Automated Cybersecurity Incident Response Strategy Optimization. Applied Sciences, 15, 951. https://doi.org/10.3390/app15020951.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Academic Journal of Education
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
