Research Article
Occlusion Aware Safety Augmentation for Partially Observed Reinforcement Learning
Downloads
Abstract
Occlusion can hide scene information needed for safe action selection without changing the physical hazard. This manuscript develops COSA, a design for paired observation augmentation, visibility-sensitive representations, ensemble cost estimation, and constrained policy updates. Physical safety remains defined by a latent state-action cost; observation-dependent surrogates estimate that cost under partial observability. The proposed visual architecture is distinguished from a tested tabular realization on a 10×10 lava gridworld. The latter uses observation re-masking, additional Q-learning replay, a blended bootstrap penalty, and a runtime fallback. Across 20 training seeds allocated to five fixed maps, its reported severe-occlusion violation rate is 0.059 versus 0.341 for domain randomization and 0.152 for a belief-augmented baseline; its return is 0.766 versus 0.767 at k=0. Removing replay substantially reduces performance, while removing the fallback slightly improves it at k=2. These observations support the tabular combination on the tested maps. They do not validate learned visual encoders, calibrated uncertainty bounds, trust-region constraint guarantees, or transfer to new occluder classes.
Keywords
Safe reinforcement learning Visual reinforcement learning Occlusion robustness Counterfactual data augmentation Constrained Markov decision process Image inpaintingReferences (42)
- [1] Altman, E. (2021). Constrained Markov decision processes. Routledge. https://doi.org/10.1201/9781315140223
- [2] Garcıa, J., & Fernández, F. (2015). A comprehensive survey on safe reinforcement learning. Journal of machine learning research, 16(1), 1437–1480. https://www.jmlr.org/papers/volume16/garcia15a/garcia15a.pdf
- [3] Achiam, J., Held, D., Tamar, A., & Abbeel, P. (2017). Constrained policy optimization. International conference on machine learning (pp. 22–31). PMLR. https://proceedings.mlr.press/v70/achiam17a
- [4] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., & Bellemare, M. G. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529–533. https://doi.org/10.1038/nature14236
- [5] Levine, S., Finn, C., Darrell, T., & Abbeel, P. (2016). End-to-end training of deep visuomotor policies. Journal of machine learning research, 17(39), 1–40. https://www.jmlr.org/papers/v17/15-522.html
- [6] Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017). Domain randomization for transferring deep neural networks from simulation to the real world. 2017 IEEE/rsj international conference on intelligent robots and systems (IROS) (pp. 23–30). IEEE. https://doi.org/10.1109/IROS.2017.8202133
- [7] Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., & Srinivas, A. (2020). Reinforcement learning with augmented data. Advances in neural information processing systems, 33, 19884–19895. https://doi.org/10.48550/arXiv.2004.14990
- [8] Kostrikov, I., Yarats, D., & Fergus, R. (2020). Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. https://doi.org/10.48550/arXiv.2004.13649
- [9] Cobbe, K., Hesse, C., Hilton, J., & Schulman, J. (2020). Leveraging procedural generation to benchmark reinforcement learning. International conference on machine learning (pp. 2048-2056). PMLR. https://proceedings.mlr.press/v119/cobbe20a.html
- [10] Hendrycks, D., & Dietterich, T. (2019). Benchmarking neural network robustness to common corruptions and perturbations. https://doi.org/10.48550/arXiv.1903.12261
- [11] Michaelis, C., Mitzkus, B., Geirhos, R., Rusak, E., Bringmann, O., Ecker, A. S., … & Brendel, W. (2019). Benchmarking robustness in object detection: Autonomous driving when winter is coming. https://doi.org/10.48550/arXiv.1907.07484
- [12] Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., & Song, D. (2021). Natural adversarial examples. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 15262–15271). https://doi.org/10.48550/arXiv.1907.07174
- [13] Schulman, J., Levine, S., Abbeel, P., Jordan, M., & Moritz, P. (2015). Trust region policy optimization. International conference on machine learning (pp. 1889–1897). PMLR. https://proceedings.mlr.press/v37/schulman15.html
- [14] Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016). Concrete problems in AI safety. https://arxiv.org/pdf/1606.06565
- [15] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. https://doi.org/10.48550/arXiv.1707.06347
- [16] Ray, A., Achiam, J., & Amodei, D. (2019). Benchmarking safe exploration in deep reinforcement learning (Vol. 7). https://cdn.openai.com/safexp-short.pdf
- [17] Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018). Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. International conference on machine learning (pp. 1861–1870). PMLR. https://proceedings.mlr.press/v80/haarnoja18b/haarnoja18b.pdf
- [18] Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998). Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1–2), 99–134. https://doi.org/10.1016/S0004-3702(98)00023-X
- [19] Zhong, Z., Zheng, L., Kang, G., Li, S., & Yang, Y. (2020). Random erasing data augmentation. Proceedings of the aaai conference on artificial intelligence (Vol. 34, pp. 13001–13008). PKP. https://ojs.aaai.org/index.php/AAAI/article/view/7000
- [20] Peng, X. Bin, Andrychowicz, M., Zaremba, W., & Abbeel, P. (2018). Sim-to-real transfer of robotic control with dynamics randomization. 2018 IEEE international conference on robotics and automation (ICRA) (pp. 3803–3810). IEEE. https://doi.org/10.1109/ICRA.2018.8460528
- [21] Sadeghi, F., & Levine, S. (2016). Cad2rl: Real single-image flight without a single real image. https://arxiv.org/abs/1611.04201
- [22] Pearl, J. (2009). Causality. Cambridge University Press. https://archive.illc.uva.nl/cil/uploaded_files/inlineitem/Pearl_2009_Causality.pdf
- [23] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., … & Bengio, Y. (2014). Generative adversarial nets. Advances in neural information processing systems, 27. https://proceedings.neurips.cc/paper_files/paper/2014/hash/f033ed80deb0234979a61f95710dbe25-Abstract.html
- [24] Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016). Context encoders: feature learning by inpainting. Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2536–2544). IEEE. https://www.cv-foundation.org/openaccess/content_cvpr_2016/html/Pathak_Context_Encoders_Feature_CVPR_2016_paper.html
- [25] Liu, G., Reda, F. A., Shih, K. J., Wang, T. C., Tao, A., & Catanzaro, B. (2018). Image inpainting for irregular holes using partial convolutions. European conference on computer vision (pp. 89–105). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-030-01252-6_6%0A%0A
- [26] Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. S. (2019). Free-form image inpainting with gated convolution. Proceedings of the IEEE/CVF international conference on computer vision (pp. 4471–4480). IEEE. https://openaccess.thecvf.com/content_ICCV_2019/html/Yu_Free-Form_Image_Inpainting_With_Gated_Convolution_ICCV_2019_paper.html
- [27] Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in neural information processing systems, 33, 6840–6851. https://arxiv.org/abs/2006.11239
- [28] Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2021). Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1), 99–106. https://dl.acm.org/doi/abs/10.1145/3503250
- [29] Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3431–3440). IEEE. https://doi.org/10.1109/TPAMI.2016.2572683
- [30] Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. Proceedings of the 31st international conference on neural information processing systems (Vol. 30, pp. 6405–6417). Curran Associates Inc. https://dl.acm.org/doi/10.5555/3295222.3295387
- [31] Gal, Y., & Ghahramani, Z. (2016). Dropout as a bayesian approximation: representing model uncertainty in deep learning. International conference on machine learning (pp. 1050–1059). PMLR. https://proceedings.mlr.press/v48/gal16.html?trk=public_post_comment-text
- [32] Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-cam: visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE international conference on computer vision (pp. 618–626). IEEE. https://doi.org/10.1109/iccv.2017.74
- [33] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778). IEEE. https://doi.org/10.1109/cvpr.2016.90
- [34] Dosovitskiy, A. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. https://doi.org/10.48550/arXiv.2010.11929
- [35] Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017). Curiosity-driven exploration by self-supervised prediction. International conference on machine learning (pp. 2778–2787). PMLR. https://arxiv.org/abs/1705.05363
- [36] Sutton, R. S., Barto, A. G., & Barto, A. (1998). Reinforcement learning: An introduction (Vol. 1). MIT Press, Cambridge. https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf
- [37] Seabold, S., & Perktold, J. (2010). Statsmodels: Econometric and statistical modeling with python. Scipy, 7(1), 92–96. https://doi.org/10.25080/majora-92bf1922-011
- [38] Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018). Deep reinforcement learning that matters. Proceedings of the AAAI conference on artificial intelligence (Vol. 32). AAAI. https://doi.org/10.1609/aaai.v32i1.11694
- [39] Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics bulletin, 1(6), 80–83. https://www.jstor.org/stable/3001968
- [40] Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, 65–70. https://www.jstor.org/stable/4615733
- [41] Friedman, M. (1937). The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American statistical association, 32(200), 675–701. https://doi.org/10.1080/01621459.1937.10503522
- [42] Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., & Casas, D. de Las. (2018). Deepmind control suite. https://arxiv.org/abs/1801.00690
Article Metrics
- Published 2026-06-15
- Issue Vol. 3 No. 2 (2026): Soft Computing Fusion with Applications (SCFA)
- Section Articles
- Document Type Research Article
- Views 0
- Downloads 0
How to Cite
Daliri, A. ., & Mensah, K. A. . (2026). Occlusion Aware Safety Augmentation for Partially Observed Reinforcement Learning. Soft Computing Fusion With Applications , 3(2), 126-138. https://doi.org/10.22105/scfa.v3i2.105