Non-stationarity Detection in Model-Free Reinforcement Learning via Value Function Monitoring

Maryem Hussein, Marwa Keshk, Aya Hussein

Research output: A Conference proceeding or a Chapter in BookConference contributionpeer-review

Abstract

The remarkable success achieved by Reinforcement learning (RL) in recent years is mostly confined to stationary environments. In realistic settings, RL agents can encounter non-stationarity when the environmental dynamics change over time. Detecting when this change occurs is crucial for activating adaptation mechanisms at the right time. Existing research on change detection mostly relies on model-based techniques which are challenging for tasks with large state and action spaces. In this paper, we propose a model-free, low-cost approach based on value functions (V or Q) for detecting non-stationarity. The proposed approach calculates the change in the value function (ΔV or ΔQ ) and monitors the distribution of this change over time. Statistical hypothesis testing is used to detect if the distribution of ΔV or ΔQ changes significantly over time, reflecting non-stationarity. We evaluate the proposed approach in three benchmark RL environments and show that it can successfully detect non-stationarity when changes in the environmental dynamics are introduced at different magnitudes and speeds. Our experiments also show that changes in ΔV or ΔQ can be used for context identification leading to a classification accuracy of up to 88%.

Original languageEnglish
Title of host publicationAI 2023
Subtitle of host publicationAdvances in Artificial Intelligence - 36th Australasian Joint Conference on Artificial Intelligence, AI 2023, Proceedings
EditorsTongliang Liu, Geoff Webb, Lin Yue, Dadong Wang
Place of PublicationSingapore
PublisherSpringer
Pages350-362
Number of pages13
ISBN (Print)9789819983872
DOIs
Publication statusPublished - 2024
Externally publishedYes
Event36th Australasian Joint Conference on Artificial Intelligence, AJCAI 2023 - Brisbane, Australia
Duration: 28 Nov 20231 Dec 2023

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume14472 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference36th Australasian Joint Conference on Artificial Intelligence, AJCAI 2023
Country/TerritoryAustralia
CityBrisbane
Period28/11/231/12/23

Fingerprint

Dive into the research topics of 'Non-stationarity Detection in Model-Free Reinforcement Learning via Value Function Monitoring'. Together they form a unique fingerprint.

Cite this