allowfullscreen>

Poster

BibTeX


                @inproceedings{malekzadeh2023unified,
                title={A Unified Uncertainty-Aware Exploration: Combining Epistemic and Aleatory Uncertainty},
                author={Malekzadeh, Parvin and Hou, Ming and Plataniotis, Konstantinos N},



    
    
    
    
    A UNIFIED UNCERTAINTY-AWARE EXPLORATION: COMBINING
    EPISTEMIC AND ALEATORY UNCERTAINTY

    

    
    
    
    
    
    

    
    
    
    
    
    
    





    

        

            

                

                    


                        
 UUaE: A Unified Uncertainty-aware Exploration
                             combining epistemic and aleatory uncertainty

                        

                            
                                Parvin Malekzadeh1,
                                
                                    Ming Hou2,
                            
                                Konstantinos N.
                                    Plataniotis1,
                        


                        

                            1University of Toronto,
                            1Defence Research and Development Canada (DRDC) 
                                Toronto Research Centre, ON, Canada,
                            
 ICASSP 2023 
                        


                        

                            

                                
                                
                                    
                                        
                                            
                                        
                                        arXiv
                                    
                                

                                
                                    
                                        
                                            
                                        
                                        Paper
                                    
                                

                                
                                
                                    
                                        
                                            
                                        
                                        Video
                                    
                                

                                
                                    
                                        
                                            
                                        
                                        Poster
                                    
                                
               

                            


                        

                    

                

            

        

    




    

        

            
        

            
            UUaE method architecture showing the steps for deriving the unified uncertainty-aware exploration approach
        

        

            Steps taken to derive our proposed UUaE method.
            Orange and green shaded areas show learning of
            epistemic and aleatory uncertainty in UUaE.
        

        

    

    



    

        

            
            
Abstract

            

                

                    Exploration is a significant challenge in practical reinforcement learning (RL), 
                    and uncertainty-aware exploration that
                    incorporates the quantification of epistemic and
                    aleatory uncertainty has been recognized as an effective exploration strategy.
                     However, capturing the combined effect
                    of aleatory and epistemic uncertainty for decision-making
                    is difficult. Existing works estimate aleatory and epistemic
                    uncertainty separately and consider the composite uncertainty as an additive 
                    combination of the two. Nevertheless, the
                    additive formulation leads to excessive risk-taking
                    behavior, causing instability.  

                

                    In this paper, we propose
                    an algorithm that clarifies the theoretical connection between aleatory and 
                    epistemic uncertainty, unifies aleatory
                    and epistemic uncertainty estimation, and quantifies the
                    combined effect of both uncertainties for a risk-sensitive
                    exploration. Our method builds on a novel extension of
                    distributional RL that estimates a parameterized return
                    distribution whose parameters are random variables encoding epistemic uncertainty. 
                     

                
 Experimental results on tasks
                with exploration and risk challenges show that our method
                outperforms alternative approaches.

Comparison with other uncertainty-aware methods

In this section, we evaluate the performance of UUaE on two Atari games with sparse reward functions. To test the proposed algorithm in a more realistic setting, we also run our algorithm on an autonomous vehicle driving simulator [29] in a highway domain, where rewards are designed to penalize unsafe driving behavior. The sparsity and risk-sensitivity of rewards in these tasks make uncertainty-aware exploration challenging.

Fig. 2 presents the results averaged over 10 random seeds.

Performance comparison of UUaE with other uncertainty-aware methods on Atari games and autonomous vehicle driving simulator booktitle={ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, pages={1--5}, year={2023}, organization={IEEE} }

Video

Poster

BibTeX


            @inproceedings{malekzadeh2023unified,
            title={A Unified Uncertainty-Aware Exploration: Combining Epistemic and Aleatory Uncertainty},
            author={Malekzadeh, Parvin and Hou, Ming and Plataniotis, Konstantinos N},
            booktitle={ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
            pages={1--5},
            year={2023},
            organization={IEEE}
            }