Constrained MDPs model return maximization subject to accumulated-cost constraints [1]. García and Fernández [2] survey safety criteria and exploration mechanisms. Constrained policy optimization establishes bounds and near-constraint satisfaction under its stated update assumptions [3], extending trust-region ideas [13]. Other algorithms and benchmarks offer alternative implementations [14–17]. These guarantees do not transfer automatically to an estimated-cost surrogate or a generic dual-ascent procedure.

Partial observability separates the true state from the information available to the policy. Belief-state planning provides a formal approach [18], while learned image encoders supply practical approximations. Simulator cost labels can train an estimator despite incomplete observations. The question is whether its error and uncertainty are controlled well enough for the intended safety constraint, rather than whether state-based physical cost is invalid.

Visual reinforcement learning includes control from pixels [4] and visuomotor policies [5]. Input augmentation can improve data efficiency and generalization [7], while DrQ regularizes value estimates through image augmentation [8]. Random Erasing creates rectangular occlusions for image-recognition training [19]; it does not by itself establish safe closed-loop behavior under hidden hazards. Domain and dynamics randomization [6], [20], rendered training [21], and procedural scene generation [9] supply other forms of variation. These works motivate the design without establishing that every generic augmentation fails to address hazards.

Corruption and distribution-shift benchmarks document visual fragility [10–12]. Their protocols differ from closed-loop safety evaluation, but this does not make them uniformly corrupted or categorically irrelevant. The present protocol focuses on masking hazard-relevant observations and measuring resulting policy outcomes.

Simulator edits can change a rendering while preserving a specified latent state. Calling them counterfactual observations identifies that intervention; causal attribution requires the other relevant variables to remain fixed [22]. Image generation and inpainting methods provide possible editing tools [23–26], while diffusion models and view synthesis offer further design options [27–29]. Photorealism alone does not verify that the hazard state or its cost label has been preserved.

Ensemble disagreement is a practical uncertainty proxy [30], and dropout offers another approximation [31]. Neither spread nor a heatmap [32] proves calibration or safety. Encoder choices [33], [34] and auxiliary objectives [35] motivate the visual design. The assembled architecture requires separate implementation and validation; its novelty is framed as a proposed integration, without an exhaustive absence-of-prior-work claim.

Table 1 contrasts the selected works with the proposed design. These entries describe those works, not every member of each method family. The visual COSA row states design capabilities; the experiments test a distinct tabular combination. Paired data, uncertainty-aware cost estimation, and constrained updates must each be evaluated before the full integration can be claimed effective.

The questions concern cost estimation under hidden hazards, the value of paired training views, and uncertainty-based conservatism. The hypotheses below are proposed tests. The tabular campaign does not execute learned encoder, calibrated-ensemble, or held-out-occluder evaluations.

RQ1 asks how to estimate latent physical safety cost from observations that hide hazard-relevant state, and how to calibrate uncertainty about that estimate.

RQ2 asks whether paired observation edits improve representations and safety compared with unpaired edits at matched data and optimizer-update budgets.

RQ3 asks whether uncertainty inflation adapts conservatism to masking severity and transfers to occluder classes absent from training.

H1 predicts lower violation rates from the proposed surrogate under occluded evaluation, relative to a comparator with matched learning and observation conditions.

H2 predicts improved held-out-occluder generalization from paired versus unpaired augmentation. That test is not executed here.

H3 predicts that calibrated uncertainty inflation changes behavior with severity. The tested tabular penalty is not an ensemble-uncertainty implementation of this hypothesis.

Let M=(S,A,P,r,c,γ,μ₀) describe the physical constrained process. The policy receives oₜ=R(sₜ,ωₜ) rather than sₜ, so the observation model makes the decision problem partially observable. The occluder configuration ωₜ affects rendering only. Physical hazard cost c(s,a) is unchanged by observation edits; κ(o,a) is an estimate used by the agent. History-dependent policies or beliefs may be needed because an individual observation need not be Markov. Table 2 collects the notation.

f_k(o, a) k-th member of the cost-estimator ensemble mu_hat_K, sigma_hat_K Ensemble mean and standard deviation of cost estimates kappa_bar(o, a) Uncertainty-inflated surrogate cost, Eq. (2) beta Uncertainty-inflation coefficient d Expected-cost budget in Eq. (1) lambda Dual variable of the constrained update D_cf Buffer of factual-counterfactual observation pairs L_pair, L_disc Consistency and visibility-discrimination losses in the proposed visual design, Eq. (3)

The physical goal is to maximize expected discounted return while bounding expected discounted latent cost. The proposed optimizer instead bounds a surrogate built from ensemble mean and spread. Eq. (1) states this surrogate constraint. It implies a physical-cost bound only if the surrogate conservatively covers the relevant latent cost or a suitable error bound is incorporated. Agreement among ensemble members does not ensure that condition. A positive β term biases estimated cost upward relative to its mean, but common-mode errors can remain.

t (1)

Jκ ‾ (θ) = π [∑ γ κ ‾(ot, at)] ≤ d.

κ ‾(o, a) = μ ̂K(o, a) + βσ ̂K(o, a).

2 (3)

ℒenc = ℒpair + αℒdisc, ℒpair =∥ zirr(o) − zirr(ocf) ∥ .

In Eq. (3), zirr denotes a hazard-irrelevant representation head. Its consistency loss preserves content that should be unaffected by the observation edit. Ldisc discriminates visibility status using renderer-derived mask labels, rather than imposing different physical cost targets. A separate cost-regression objective assigns both paired views the same c(s,a). These objectives form a proposed design; their calibration and interaction are not tested by the tabular experiments.

The visual design has a paired-observation generator, encoder, ensemble cost estimator, and constrained policy optimizer. Training would interleave rollouts and edited observations. Simulator masks and physical cost labels provide supervision, while inference uses available observations. Algorithm 1 specifies the proposed loop; its neural and constrained-update components are not implemented in Section 5.

Training requires a simulator exposing rendering, depth, and object masks for controlled observation edits. The physical latent cost must be computable for simulator supervision even when the policy cannot observe the hazard. Occlusion edits are assumed to affect observations only; physical state, transition dynamics, and cost labels remain fixed. Simulator re-rendering or screened inpainting must preserve the intended latent state. Visual plausibility alone is insufficient. The design assumes a single camera. Additional sensing or active viewpoints are possible extensions.

A reveal edit removes a visual occluder and a conceal edit inserts one, while preserving physical state, action, and non-designated scene factors. Simulator rendering is preferred when it permits those controls. Inpainting [25], [26] and diffusion-based generation [27] are possible alternatives, but they can introduce artifacts or change inferred scene content. A taxonomy of occluder classes is a proposed extension; the tabular study uses only rectangular masking.

Pairs are annotated with a mask and a visibility measure, such as the fraction of hazard area covered. A pixel audit checks non-designated regions, and a state audit checks that physical hazards and costs are unchanged. Both observations receive the same c(s,a) target. These controls define the observation intervention; they do not by themselves isolate its learning effect from additional data or updates.

Algorithm 1. COSA training loop.

Input: simulator G with segmentation access; latent cost c(s, a); ensemble size K; inflation coefficient beta; pair-generation probability p_cf; expected-cost budget d 1: initialize policy pi_theta, encoder g_psi, cost ensemble f_1..f_K, dual variable lambda 2: repeat 3: collect rollout batch B with pi_theta in G 4: for each transition (o_t, a_t, s_t) in B do 5: with probability p_cf sample occluder config omega' and occlusion intensity rho 6: o_cf <- composite(G, s_t, omega') or inpaint(o_t, mask) 7: run mask audit and perceptual screening; discard on failure 8: store (o_t, o_cf, mask, visibility, c(s_t, a_t)) in D_cf 9: end for 10: update the encoder with irrelevant-content consistency and visibility discrimination 11: regress each paired cost prediction to the same physical c(s_t, a_t) 12: kappa_bar <- mu_hat_K + beta * sigma_hat_K // Eq. (2) 13: update pi_theta by constrained trust-region step on (J_r, J_c(kappa_bar)) with dual ascent on lambda // Eq. (1) 14: until training budget exhausted Output: policy pi_theta, encoder g_psi, cost ensemble f_1..f_K, fitted dual variable lambda

The encoder uses separate heads for hazard-irrelevant content and visibility-relevant features. A consistency term is applied to the irrelevant head, while renderer-derived visibility labels guide Ldisc on the relevant head. Concealing a hazard does not change the underlying hazard state or its c(s,a) value. Thus cost estimates should target the same physical label for a conceal-and-reveal pair, although uncertainty may differ. Any measured distinction in prediction must be interpreted through estimator accuracy and calibration, not forced difference in true costs.

The K cost estimators are supervised by simulator costs. Their standard deviation is used as an uncertainty proxy in Eq. (2), rather than a certified upper bound. A constrained trust-region or Lagrangian implementation may optimize Jr−λ(Jc−d) [3], [15], but dual ascent alone does not ensure convergence or physical constraint satisfaction. Validation must assess cost-prediction error, spread calibration, feasibility, and realized physical costs under shifts. These checks are requirements for the proposed visual implementation; the tabular study does not perform them.

Generation costs grow with the number of edits, and ensemble scoring grows with K. Pair storage adds memory and replay adds optimizer work. Inpainting, encoder training, calibration, and policy updates contribute further costs. The tabular campaign quantifies neither the neural overhead nor a compute-matched benefit of the full architecture.

The following are anticipated failure modes and proposed mitigations, rather than controls all executed by the present study:

I. Inpainting artifacts may become shortcuts; simulator rendering, region audits, and artifact controls can reduce this risk.

II. Synthetic occluders may differ from real clutter. Held-out-class and physical-scene tests remain necessary. III. Ensemble agreement may conceal shared errors. Calibration and realized-cost audits are required.

IV. Conservatism may reduce return or impede deadlines. A return–cost frontier should be measured across β and budget choices.

V. Counterfactual replay can add supervision for occluded views, but it must retain the same physical state-action cost label.

All experiments run on a purpose-built 10 x 10 gridworld navigation task implemented in NumPy. Each of the five fixed maps contains twelve lava cells, a goal in the top-right region, and episode starts drawn uniformly from the bottom-left region; stepping costs -0.01, reaching the goal yields +1.0 and terminates the episode with success, and entering lava yields -1.0 and terminates it with a safety violation, with episodes capped at 200 steps. The agent observes an egocentric 5 x 5 view plus four goal-direction bits, and occlusion is applied as k random 2 x 2 patches of the view masked per step, with a 5% per-step blackout probability retained at every occlusion level. The five maps were generated once with a fixed master seed and every training seed is assigned to one of them, so map layout acts as a controlled blocking factor.

Four tabular Q-learning agents [36] share the same state abstraction: agent position, coarse goal sector, four orthogonal-neighbour flags in {unknown, safe, hazard, wall}, and a caution bucket that coarsens the fraction of the local window that is currently unobserved. The naive baseline trains entirely on clean views; the domain-randomization baseline draws k uniformly from {0, ..., 4} each episode [6]; the belief-augmented baseline replaces last-seen flags with a persistent per-episode memory map that encodes unseen cells as hazards; and COSA trains on clean views while replaying every experienced transition under counterfactually re-masked observations (k' drawn from {1, 2, 4}) with a conservative bootstrap that blends the clean and occluded next-state values with weight lambda = 0.5, together with a risk-threshold fallback (tau = 0.15) that switches to a widest-margin detour action whenever the value gap between the observed and the worst-case state exceeds the threshold. All agents use alpha = 0.2, gamma = 0.95, and an epsilon schedule decaying linearly from 1.0 to 0.05.

The campaign reports 20 training seeds, assigned as four repeats on each of five fixed maps, and 3,000 training episodes per seed. Each policy is evaluated for 200 episodes at k∈{0,1,2,4}, using paired streams. Pilot training increased the episode budget from 600; no preregistration record is established. The stated runtime is 367.5 s, with Python 3.12, NumPy 2.1.3, SciPy 1.14.1, statsmodels 0.14.5 [37], and matplotlib 3.9.2. Exact reproduction requires the maps, streams, code, and outputs; variance and experimental reporting warrant careful control in reinforcement learning [38]. Matching episodes does not match additional replay updates or total compute. Absolute position in the state abstraction also permits map memorization; no map is held out.

Tables 3 and 4 report returns and outcome rates over the 20 seeds. At k=1 the source reports naive return −1.133 and violation rate 0.994, whereas domain randomization and the belief baseline perform better. At k=4, domain randomization has return 0.056 and violation rate 0.341, and the belief baseline has return 0.463 and violations 0.152. These observations concern these implementations on five fixed maps. Unknown-state behavior is a plausible explanation for naive failure, but identifying it requires Q-table and action-trace inspection.

Each cell aggregates 200 test episodes per seed on shared evaluation streams.

levels k = 0, 2, and 4.

Policy Success k = 0 Success k = 2 Success k = 4 Naive Q (clean-trained) 0.945 +/- 0.022 0.002 +/- 0.003 0.010 +/- 0.012 Domain randomization 0.941 +/- 0.022 0.682 +/- 0.102 0.659 +/- 0.092 Belief-augmented Q 0.936 +/- 0.021 0.913 +/- 0.025 0.848 +/- 0.036 COSA (proposed) 0.944 +/- 0.019 0.932 +/- 0.024 0.941 +/- 0.023

and seed controls.

COSA Variant (K = 2) Return Full COSA 0.749 +/- 0.047 Minus counterfactual augmentation -1.134 +/- 0.045 Minus conservative penalty 0.717 +/- 0.039 Minus risk fallback 0.759 +/- 0.051 k = 2 k = 4 -1.139 +/- 0.041 -1.143 +/- 0.045 0.069 +/- 0.231 0.056 +/- 0.205 0.664 +/- 0.053 0.463 +/- 0.077 0.749 +/- 0.047 0.766 +/- 0.044

Violation k = 0 Violation k = 2 Violation k = 4 0.055 +/- 0.022 0.998 +/- 0.003 0.990 +/- 0.012 0.059 +/- 0.022 0.317 +/- 0.103 0.341 +/- 0.091 0.064 +/- 0.021 0.086 +/- 0.025 0.152 +/- 0.036 0.056 +/- 0.019 0.068 +/- 0.024 0.059 +/- 0.023

Success Rate Violation Rate 0.932 +/- 0.024 0.068 +/- 0.024 0.020 +/- 0.011 0.980 +/- 0.011 0.925 +/- 0.020 0.075 +/- 0.020 0.937 +/- 0.026 0.063 +/- 0.026

COSA reports k=4 return 0.766 and violation rate 0.059. The source traces describe a cautious Q-function, but the comparisons do not isolate that explanation from masked-state coverage or additional replay updates. The naive method has slightly higher k=0 return, so COSA does not dominate every cell. The runtime fallback and blended target are specific implementation choices, not a calibrated ensemble cost model.

Table 5 compares tabular variants at k=2. Removing re-masked replay leaves return −1.134 and violations 0.980; this changes both masked-state coverage and replay updates, so it does not isolate causal pairing. Removing the blended penalty changes return from 0.749 to 0.717 and violations from 0.068 to 0.075. The reported differences are consistent with a benefit of that variant at this level, subject to the common update Removing the fallback gives return 0.759 and violations 0.063, slightly better than the full method at k=2. Its protective effect is therefore not demonstrated at this severity. Unseen severities could test its intended role, but that remains an extension. The replay-removal contrast supports the complete replay treatment relative to its absence; a matched-update unpaired replay comparator is required to isolate pairing.

Fig. 2 and Table 4 show COSA violations near 0.06 across the tested levels, with belief-baseline violations increasing from 0.064 at k=0 to 0.152 at k=4, and domain randomization from 0.059 to 0.341. COSA success changes from 0.944 to 0.941. Here k=0 means no patch masking but retains a 5% blackout probability, so it is not a fully unobscured condition.

Every level retains 5% blackout; COSA is approximately flat over the tested levels.

Only one rectangular-mask mechanism is tested. Held-out occluder classes under H2 and calibrated ensemble inflation under H3 remain untested. A nearly flat violation profile is an observation of the tabular realization, not evidence that the proposed ensemble adapts uncertainty correctly.

Table 6 retains paired-seed Wilcoxon [39], Holm [40], and Friedman [41] summaries. For k=4, the reported COSA-versus-randomization return contrast has W=0 and p=1.9e−6; the two-sided all-same-sign exact value is 2/2²⁰≈1.907e−6 under 20 independent nonzero pairs. The reported Friedman χ²(3)=60 is the maximum n(k−1) for n=20 and four methods without rank inconsistencies. Raw ranks, ties, and Holm family must be checked. The 20 seeds reuse five maps, so tests are conditional on this map set and do not establish generalization to independent environments.

The tabular replay-and-penalty combination has lower reported violations under severe masking than the compared implementations. The penalty-removal contrast has a smaller difference, while the fallback-removal contrast slightly favors removal at k=2. COSA return at k=4 is close to its own k=0 value. These comparisons support the tested combination, but unmatched replay updates prevent attributing the main gain solely to paired observation structure.

At k=0, where patch masks are absent but 5% blackout remains, the naive policy has return 0.773 versus 0.767 for COSA and success 0.945 versus 0.944. The belief baseline remains competitive at mild masking. The fallback-removal variant is slightly better at k=2, and any protective effect beyond the trained range remains untested.

The tabular realization replaces image editing with view re-masking, neural cost uncertainty with a blended bootstrap, and constrained trust-region updates with a penalty in the TD target. It does not test the visual architecture. A 100-cell environment, one hazard class, one masking mechanism, and five reused maps provide limited evidence. Photorealistic and continuous-control transfer [42], new maps, encoder behavior, and physical constraint calibration remain open.

The reported observations are proof-of-concept results from the stated gridworld configuration. Code, maps, seeds, and raw outputs are required to establish their traceability and reproducibility.

The visual design depends on simulator supervision and state-preserving observation edits. Expected additive cost does not capture every irreversible safety concern. Ensemble disagreement may be miscalibrated under occlusion, and that calibration is not tested by this tabular study. Real-time overhead, deadlines, multiple sensors, and active viewpoint control also remain outside the evaluation.

Domain randomization, memory-based beliefs, and paired replay are reasonable alternatives with different implementations. The reported randomization result does not settle which succeeds at matched update counts, observation coverage, or tuning budgets. A per-episode memory map is the strongest tested baseline under severe masking. Physical-scene validity of simulator edits and held-out occluder generalization require separate experiments.