Revizyon Diff — access.tex

access.tex revizyon diff +181 eklenen −1 silinen ~85 değişen toplam 267 satır yalnızca değişiklikleri göster 📄 PDF 🖍️ İşaretli PDF

SUBMIT EDİLEN (5 Tem 2026)

REVİZE (çalışma kopyası)

 % IEEE Access LaTeX source package
 % Compile inside the official IEEE Access LaTeX template project, or keep the
 % official ieeeaccess.cls in this directory.
 \documentclass{ieeeaccess}
 \usepackage{silence}
 \WarningFilter{xcolor}{Incompatible color definition}
 \WarningFilter{latexfont}{Font shape}
 \WarningFilter{latexfont}{Some font shapes}
 \usepackage{cite}
 \usepackage{amsmath,amssymb,amsfonts}
 \usepackage{algorithm}
 \usepackage{algorithmic}
 \usepackage{graphicx}
 \usepackage{textcomp}
 \usepackage{xcolor}
 \usepackage{booktabs}
 \usepackage{array}
 \usepackage{tabularx}
 \usepackage{makecell}
 \usepackage{threeparttable}
 \usepackage{url}
 \usepackage{balance}
 \usepackage{comment}
 \makeatletter
 \let\prism@orig@outputpage\@outputpage
 \def\@outputpage{{\hfuzz=1000pt\hbadness=10000\vfuzz=1000pt\vbadness=10000\prism@orig@outputpage}}
 \makeatother
 \def\BibTeX{{\rm B\kern-.05em{\sc i\kern-.025em b}\kern-.08em
     T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}
 \begin{document}
 \vol{14}
 \year{2026}
 \showboxbreadth=0
 \showboxdepth=0
 \hbadness=10000
 \vbadness=10000
 \hfuzz=1000pt
 \vfuzz=1000pt
 \emergencystretch=3em
 \history{Date of submission July 5, 2026. Publication and current-version dates to be assigned by IEEE Access.}
 \doi{To be assigned by IEEE Access}
 \title{Evaluating Self-Reflective LLM Agents for\\ Autonomous Kubernetes Troubleshooting}
 \author{\uppercase{Mustafa Mert Süerkan}\authorrefmark{1},
 \uppercase{Emin Kugu}\authorrefmark{2,3} \IEEEmembership{Member, IEEE}, and
 \uppercase{Hakan Emekci}\authorrefmark{4} \IEEEmembership{Member, IEEE}}
 \address[1]{TED University, Graduate School, 06420, Ankara, Turkiye}
 \address[2]{TED University, Software Engineering Department, 06420, Ankara, Turkiye }
 \address[3]{Universitat Politècnica de València, Telecommunication and Multimedia Applications University Research Institute (iTEAM), València, 46730, Spain}
 \address[4]{Hacettepe University Institute of Informatics, Ankara, Turkiye }
 \corresp{Corresponding author: Emin Kugu (emin.kugu@tedu.edu.tr, ekugu@upvnet.upv.es).}
 \markboth
 {Süerkan \headeretal: Evaluating Self-Reflective LLM Agents for Autonomous Kubernetes Troubleshooting}
 {Süerkan \headeretal: Evaluating Self-Reflective LLM Agents for Autonomous Kubernetes Troubleshooting}
 \begin{abstract}
-Kubernetes is now a central platform for deploying cloud-native applications, yet resolving cluster incidents continues to demand considerable operational knowledge. Modern large language model (LLM) agents can use command-line tools and interpret logs and configuration files, but their dependability for autonomous remediation is still unclear. This paper experimentally examines whether Reflexion, a verbal reinforcement learning method driven by self-produced failure analysis, improves the performance of LLM agents on Kubernetes troubleshooting tasks. We built an autonomous remediation system composed of an LLM actor, a deterministic programmatic evaluator, a self-reflection module, and bounded episodic memory. We assessed the system in 1,500 controlled runs covering five Claude models, ten Kubernetes fault scenarios, and three configurations: a one-shot baseline, a two-try configuration without reflection, and a complete Reflexion configuration. The findings are mixed: permitting a second attempt raised overall success from 38.6\% to 68.4\%, while adding Reflexion increased success only slightly to 69.2\%. A preliminary complexity analysis showed a small decline in single-fault cases (-2.0 percentage points) and a +12.0 percentage-point gain in each of the two multi-fault cases, but the multi-fault result is hypothesis-generating because only one double-fault and one triple-fault scenario were tested. Within the evaluated Claude models, the largest gains appeared for mid-capability models; weaker models did not use reflections effectively, and the strongest model showed a ceiling effect. Cost and latency results suggest that mid-tier models can provide the most favorable production-oriented trade-off. These findings apply to a single-round reflection setting under a two-attempt budget, where each run produces at most one reflection. Overall, the results support selective deployment of self-reflective agents for complex or unresolved incidents rather than routine use for every failure.
+Kubernetes is central to cloud-native deployment, yet resolving cluster incidents still demands considerable operational knowledge, and the dependability of large language model (LLM) agents for autonomous remediation is unclear. This paper examines whether Reflexion, a verbal reinforcement learning method driven by self-produced failure analysis, improves LLM agents on Kubernetes troubleshooting tasks. We built a Reflexion-based remediation system with a deterministic programmatic evaluator and assessed it in 1,500 controlled runs over five Claude models, ten fault scenarios, and three configurations (one-shot baseline, two-try without reflection, and full Reflexion with at most one reflection per run). Permitting a second attempt raised overall success from 38.6\% to 68.4\%, whereas adding Reflexion increased it only to 69.2\%. Reflexion slightly reduced single-fault success (-2.0 percentage points) but raised it by 12.0 points in both multi-fault cases; with only one double-fault and one triple-fault scenario tested, this result is hypothesis-generating and needs validation on a larger benchmark. Two post-hoc controls on the three currently available models qualify these results. Forwarding the evaluator's verbatim failure reason without reflection outperformed full Reflexion on identically instantiated multi-fault scenarios (76.4\% versus 26.0\%), so most of the multi-fault gain comes from the external failure signal. Allowing the actor to verify its own fix within one attempt recovered about half of the retry effect on single-fault scenarios (67.5\% to 80.8\%), while the externally verified second attempt remained superior (92.5\%). Mid-capability models benefited most and offered the best cost-latency trade-off, weaker models did not use reflection effectively, and the strongest model showed a ceiling effect. The results support selective use of self-reflection for complex or unresolved incidents rather than routine use for every failure. 📄 s.1
 \end{abstract}
 \begin{keywords}
 AIOps, autonomous operations, DevOps, Kubernetes, large language model agents, Reflexion, self-reflection, incident remediation.
 \end{keywords}
 \titlepgskip=-15pt
 \maketitle
 \section{Introduction}
 \label{sec:introduction}
-Modern cloud-native systems increasingly depend on Kubernetes to manage containerized workloads over distributed infrastructure. Although Kubernetes offers declarative deployment, service discovery, scaling, and self-healing mechanisms, its operating model also makes production incident diagnosis more difficult. A fault can stem from a pod, deployment, service, ConfigMap, probe, port mapping, image definition, or resource constraint, while the visible symptom may be separated from the actual root cause. In typical practice, operators examine cluster state, events, logs, and YAML manifests before making changes with \texttt{kubectl} commands or infrastructure-as-code workflows. This process demands specialized expertise and has a direct effect on mean time to recovery (MTTR).
+Modern cloud-native systems increasingly depend on Kubernetes to manage containerized workloads over distributed infrastructure. Although Kubernetes offers declarative deployment, service discovery, scaling, and self-healing mechanisms, its operating model also makes production incident diagnosis more difficult. A fault can stem from a pod, deployment, service, ConfigMap, probe, port mapping, image definition, or resource constraint, while the visible symptom may be separated from the actual root cause. In typical practice, operators examine cluster state, events, logs, and YAML manifests before making changes with \texttt{kubectl} commands or infrastructure-as-code workflows. This process demands specialized expertise and has a direct effect on mean time to recovery (MTTR). Empirical studies of large-scale cloud services underline the weight of this burden: most production incidents are mitigated not by code changes but by configuration fixes, rollbacks, and infrastructure-level interventions, which makes misconfiguration repair a first-order operational concern \cite{ghosh2022incidents}. 📄 s.1
-Native Kubernetes tools, including the Horizontal Pod Autoscaler and Vertical Pod Autoscaler, automate selected scaling decisions but do not offer broad diagnosis and repair for arbitrary configuration errors. Monitoring stacks such as Prometheus and Grafana increase observability, yet they typically do not perform autonomous remediation. LLM-based agents create a new possibility because they can inspect cluster state, call tools, reason about symptoms, and apply fixes. This possibility raises an important reliability question: can an LLM agent learn from an unsuccessful remediation attempt, or does self-reflection mostly add latency and cost without improving the outcome?
+Native Kubernetes tools, including the Horizontal Pod Autoscaler and Vertical Pod Autoscaler, automate selected scaling decisions but do not offer broad diagnosis and repair for arbitrary configuration errors. Monitoring stacks such as Prometheus and Grafana increase observability, yet they typically do not perform autonomous remediation. LLM-based agents can inspect cluster state, call tools, reason about symptoms, and apply fixes, which raises a reliability question: does an agent learn from a failed remediation attempt, or does self-reflection mainly add latency and cost without improving the outcome? 📄 s.1
 Reflection in LLM agents refers to an explicit post-hoc reasoning step in which the agent reviews its own prior output, trajectory, or failure and converts that review into guidance for a later attempt. It is related to broader self-improvement methods such as Self-Refine, where a model critiques and revises its own answer, and Tree of Thoughts, where alternative reasoning paths are explored before selecting a solution \cite{madaan2023selfrefine,yao2023treeofthoughts}. Reflexion formalizes this idea as verbal reinforcement learning: instead of updating model parameters, the agent produces a natural-language reflection after an unsuccessful trajectory, stores it as episodic memory, and conditions the next trial on that memory \cite{shinn2023reflexion}. In agentic systems, this mechanism is especially important because the model does not merely generate text; it selects tools, observes external feedback, and changes its plan across multiple interaction steps \cite{wang2024surveyagents,yao2023react}. Prior work on generative agents also shows that memory and reflection can help agents synthesize past observations into higher-level behavioral guidance, suggesting that reflection can function as a lightweight substitute for experience when retraining is not feasible \cite{park2023generativeagents}.\par
-In this paper, reflection is therefore treated as a structured failure-analysis component rather than as a generic second thought. A useful reflection should identify which hypothesis was wrong, which evidence was overlooked, which action partially changed the system state, and what concrete repair should be attempted next. This distinction matters for Kubernetes remediation because failures are interactive and stateful: an agent may fix an image tag but leave a service targetPort mismatch, repair a ConfigMap reference but still face a probe error, or misinterpret a symptom that appears far from the underlying resource. Reflection may help in such cases by turning the first failed attempt into actionable diagnostic memory, but it can also add irrelevant context, cost, and latency if the incident is simple or if the model lacks enough capability to use the feedback. Consequently, the central question is not whether reflection is conceptually useful, but under which operational conditions it improves verified remediation beyond a simple retry \cite{shinn2023reflexion,madaan2023selfrefine}.\par
+We therefore treat reflection as a structured failure-analysis step, not as a generic second thought. A useful reflection identifies which hypothesis was wrong, which evidence was overlooked, which action partially changed the system state, and which repair to attempt next. This distinction matters for Kubernetes remediation because failures are interactive and stateful: an agent may fix an image tag but leave a service targetPort mismatch, repair a ConfigMap reference but still face a probe error, or misinterpret a symptom that appears far from the underlying resource. Reflection may help in such cases by turning the first failed attempt into actionable diagnostic memory, but it can also add irrelevant context, cost, and latency if the incident is simple or if the model lacks enough capability to use the feedback. Consequently, the central question is not whether reflection is conceptually useful, but under which operational conditions it improves verified remediation beyond a simple retry \cite{shinn2023reflexion,madaan2023selfrefine}.\par 📄 s.2
-This work studies Reflexion for Kubernetes incident remediation using a controlled experimental setup. We compare three configurations: a one-attempt baseline, a two-try setting without reflection, and a full Reflexion setting. Separating these conditions is necessary because simply allowing another attempt may improve results even when no explicit reflection is used. Thus, we distinguish the benefit of retrying from the benefit of reflection and test whether self-reflection contributes additional measurable value.
+This work studies Reflexion for Kubernetes incident remediation using a controlled experimental setup. We compare three configurations: a one-attempt baseline, a two-try setting without reflection, and a full Reflexion setting. Separating these conditions is necessary because simply allowing another attempt may improve results even when no explicit reflection is used. Thus, we distinguish the benefit of retrying from the benefit of reflection and test whether self-reflection contributes additional measurable value. Because the trial budget is two attempts, each run produces at most one reflection; the study therefore evaluates single-round, Reflexion-style reflection---the marginal value of the first reflection---rather than the full multi-trial Reflexion loop with accumulated episodic memory \cite{shinn2023reflexion}. 📄 s.2
 The study is guided by four research questions:
 \begin{itemize}
-    \item \textbf{RQ1:} How much does Reflexion improve success on Kubernetes troubleshooting tasks?
+    \item \textbf{RQ1:} How much does a single round of Reflexion-style reflection improve success on Kubernetes troubleshooting tasks beyond a verified retry? 📄 s.2
     \item \textbf{RQ2:} How does the effect differ between single-fault and multi-fault incidents?
     \item \textbf{RQ3:} How does model capability affect the value of reflection?
     \item \textbf{RQ4:} Which model--configuration pair provides the best balance of success, cost, and runtime?
 \end{itemize}
 The results indicate that Reflexion is best viewed as a conditional mechanism. Most of the overall improvement comes from allowing a second attempt, whereas reflection is most helpful for complex incidents with multiple interacting faults. This has practical implications for AIOps and DevOps systems: reflection can be saved for complex or unresolved cases instead of being applied automatically to every failure.
 This paper makes the following contributions:
 \begin{itemize}
     \item We introduce an autonomous Kubernetes remediation architecture based on Reflexion, combining an LLM actor, deterministic evaluator, self-reflection module, and bounded episodic memory.
     \item We conduct 1,500 controlled runs over five Claude models, ten Kubernetes incidents, and three configurations.
-    \item We disentangle retrying from reflection and show that the second attempt accounts for most aggregate improvement.
+    \item We disentangle retrying from reflection and show that the second attempt accounts for most aggregate improvement; a post-hoc self-verification ablation (Section~\ref{sec:selfverify}) further shows that allowing the actor to verify its own fix recovers about half of this retry effect on single-fault scenarios, while an externally verified second attempt remains superior. 📄 s.2
-    \item We observe preliminary evidence that Reflexion may help multi-fault incidents and mid-capability models, while noting in Section~\ref{sec:threats} that the multi-fault finding requires confirmation on a larger scenario set.
+    \item We observe preliminary evidence that Reflexion may help multi-fault incidents and mid-capability models, while noting in Section~\ref{sec:threats} that the multi-fault finding requires confirmation on a larger scenario set. A post-hoc feedback-only control (Section~\ref{sec:feedback_control}) further shows that supplying the evaluator's verbatim failure reason without any reflection is substantially more effective than reflection on identically instantiated multi-fault scenarios, attributing most of the benefit to the failure signal itself rather than to verbal self-reflection. 📄 s.2
     \item We discuss cost, latency, and model-selection implications for production-focused AIOps deployments.
 \end{itemize}
+The remainder of this paper is organized as follows. Section~\ref{sec:related_work} reviews related work on Kubernetes operations, LLM agents, self-reflection methods, and operational benchmarks. Section~\ref{sec:methods} describes the experimental design, agent architecture, prompts, scenarios, and evaluation procedure. Section~\ref{sec:results} reports the quantitative and qualitative results. Section~\ref{sec:discussion} discusses their implications and the threats to validity, and Section~\ref{sec:conclusion} concludes the paper and outlines future work. 📄 s.2
+ 📄 s.2
 \section{Related Work}
 \label{sec:related_work}
+This section provides a narrative review that positions the study within four strands of prior work: Kubernetes operations and AIOps, LLM agents and self-reflection methods, benchmarks for operational agents, and recent journal studies on LLM-based operations. The review is selective rather than systematic; it covers the work most directly related to the research questions instead of following a formal search protocol. 📄 s.2
 \subsection{Kubernetes Operations and AIOps}
 Kubernetes grew out of earlier cluster-management systems and is now widely used for cloud-native deployment \cite{burns2016borg,kubernetes_docs}. Its declarative API and controller-driven design support scalable operation, but they also expand the space of possible configuration errors. Frequent failure patterns include image-pull failures, CrashLoopBackOff behavior, service endpoint mismatches, liveness-probe faults, volume and ConfigMap problems, and resource-scheduling issues.
 AIOps seeks to automate fault detection, root-cause analysis, and remediation by using telemetry, logs, traces, and learned models \cite{zhang2024aiops}. Automated log analysis is a key part of reliability engineering for distributed systems \cite{he2021logsurvey}. Many current tools assist with monitoring or workflow automation, but they remain limited when unfamiliar combinations of misconfigurations occur. Within Kubernetes, K8sGPT provides AI-supported diagnostics \cite{jones2023k8sgpt}, and newer multi-agent systems such as KubeIntellect investigate end-to-end management \cite{ardebili2025kubeintellect}. The specific contribution of self-reflection and episodic memory to autonomous Kubernetes remediation, however, has not been studied in depth.
 \subsection{LLM Agents and Self-Reflection}
-LLM agents augment language models with tools, memory, and repeated interaction with external environments \cite{wang2024surveyagents,qin2024toollearning}. ReAct combines reasoning with action, enabling a model to choose what information to gather and what operation to perform \cite{yao2023react}. Prior tool-learning research also shows that foundation models can use APIs and command-line interfaces when suitable constraints are provided \cite{schick2023toolformer}.
+LLM agents augment language models with tools, memory, and repeated interaction with external environments \cite{wang2024surveyagents,qin2024toollearning}. Multi-agent frameworks such as AutoGen extend this paradigm by orchestrating several conversing agents \cite{wu2024autogen}; the present study instead evaluates a single-agent design, so that the effect of reflection is not confounded with inter-agent coordination. ReAct combines reasoning with action, enabling a model to choose what information to gather and what operation to perform \cite{yao2023react}. Prior tool-learning research also shows that foundation models can use APIs and command-line interfaces when suitable constraints are provided \cite{schick2023toolformer}. 📄 s.3
-Self-reflection methods try to improve future actions by examining previous failures. Self-Refine has a model critique and revise its own output \cite{madaan2023selfrefine}; Tree of Thoughts explores multiple reasoning paths \cite{yao2023treeofthoughts}; and Reflexion records verbal feedback after unsuccessful trajectories \cite{shinn2023reflexion}. Such methods are appealing for operations because they avoid retraining. Their success nevertheless depends on whether the reflection is concrete, actionable, and useful to the model on the following attempt.
+Self-reflection methods try to improve future actions by examining previous failures. Self-Refine has a model critique and revise its own output \cite{madaan2023selfrefine}; Tree of Thoughts explores multiple reasoning paths \cite{yao2023treeofthoughts}; and Reflexion records verbal feedback after unsuccessful trajectories \cite{shinn2023reflexion}. Such methods are appealing for operations because they avoid retraining. Their success depends on whether the reflection is specific enough for the model to use on the next attempt.\par An important design dimension of these methods is the number of reflection rounds. In the original Reflexion loop, the agent may iterate over many trials, accumulating several verbal reflections in a bounded episodic memory so that later attempts are conditioned on a growing body of self-generated guidance \cite{shinn2023reflexion}. A single-round variant, in contrast, generates and consumes exactly one reflection between two attempts. The present study deliberately evaluates the single-round setting: it isolates the marginal value of the first reflection, which is the most cost-relevant case for production remediation because every additional trial multiplies latency and API cost. Whether accumulated reflective memory yields compounding benefits over larger trial budgets is a separate question that this study does not measure. 📄 s.3
 \subsection{Benchmarks for Operational Agents}
-Broad software-engineering benchmarks, including HumanEval and SWE-bench, assess code generation or issue resolution \cite{chen2021humaneval,jimenez2024swebench}; cloud operations, however, require interaction with a specific environment and external validation. Kubernetes-focused benchmarks such as KubeLLMBench cover common faults such as port mismatches, selector mistakes, liveness-probe errors, incorrect image names, and missing environment variables \cite{dejesus2025kubellmbench}. AIOpsLab likewise evaluates autonomous cloud agents with realistic fault injection \cite{chen2025aiopslab}. This study uses Kubernetes misconfiguration scenarios and adds multi-fault cases to test whether reflection helps when faults must be diagnosed in sequence.
+Broad software-engineering benchmarks, including HumanEval and SWE-bench, assess code generation or issue resolution \cite{chen2021humaneval,jimenez2024swebench}, and AgentBench evaluates LLMs as agents across interactive environments such as operating systems and databases \cite{liu2024agentbench}; cloud operations, however, require interaction with a specific environment and external validation. Kubernetes-focused benchmarks such as KubeLLMBench cover common faults such as port mismatches, selector mistakes, liveness-probe errors, incorrect image names, and missing environment variables \cite{dejesus2025kubellmbench}. AIOpsLab likewise evaluates autonomous cloud agents with realistic fault injection \cite{chen2025aiopslab}, and ITBench benchmarks AI agents on real-world IT automation tasks, including site-reliability scenarios on Kubernetes, and reports that even the strongest agents resolve only a small fraction of them \cite{jha2025itbench}. This study uses Kubernetes misconfiguration scenarios and adds multi-fault cases to test whether reflection helps when faults must be diagnosed in sequence. 📄 s.3
 \subsection{Recent AIOps and LLM-Based Operations Studies}
 \label{sec:recent_journal_studies}
-Diaz-de-Arcaya \emph{et al.} \cite{diazdearcaya2023mlopsaiops} provide a systematic journal survey of MLOps and AIOps and frame both areas as complementary operational disciplines for deploying, monitoring, and maintaining intelligent systems. Their review emphasizes that organizational adoption requires more than model accuracy: collaboration, lifecycle governance, automation pipelines, and operational feedback loops are also necessary. For the present study, this work is relevant because autonomous Kubernetes remediation can be interpreted as an AIOps lifecycle capability in which detection, diagnosis, repair, and post-failure learning must be integrated into a repeatable operational process rather than treated as isolated model calls.\par
+Diaz-de-Arcaya \emph{et al.} \cite{diazdearcaya2023mlopsaiops} provide a systematic journal survey of MLOps and AIOps and frame both areas as complementary operational disciplines for deploying, monitoring, and maintaining intelligent systems. Their review emphasizes that organizational adoption requires more than model accuracy: collaboration, lifecycle governance, automation pipelines, and operational feedback loops are also necessary. Autonomous Kubernetes remediation fits this view: detection, diagnosis, repair, and post-failure learning form one repeatable operational process instead of isolated model calls.\par 📄 s.3
-Zhang \emph{et al.} \cite{zhang2025aiopsllm} survey AIOps in the era of large language models and organize recent work around failure data sources, AIOps tasks, LLM-based methods, and evaluation practices. Their analysis shows that LLMs are increasingly used not only for text interpretation but also for incident understanding, runbook assistance, and operator-facing decision support. This directly supports the motivation of this paper: Kubernetes incident remediation is a setting where an LLM must combine heterogeneous evidence, choose actions, and remain verifiable under operational constraints such as cost, latency, and safety.\par
+Zhang \emph{et al.} \cite{zhang2025aiopsllm} survey AIOps in the era of large language models and organize recent work around failure data sources, AIOps tasks, LLM-based methods, and evaluation practices. Their analysis shows that LLMs are now used for incident understanding, runbook assistance, and operator-facing decision support as well as for text interpretation. Kubernetes incident remediation is such a setting: the model must combine heterogeneous evidence, choose actions, and remain verifiable under cost, latency, and safety constraints.\par 📄 s.3
-Zha \emph{et al.} \cite{zha2024alertaggregation} study LLM-assisted alert aggregation for AIOps and propose a two-phase approach that first groups alerts by temporal and spatial locality and then uses an LLM to trace cascading service-failure effects. Their results show that LLMs can add value when operational evidence must be connected across dependent services, especially during alert storms. This is closely related to multi-fault Kubernetes remediation because both problems require the system to reason beyond a single symptom and infer whether multiple observations share an underlying cause.\par
+Zha \emph{et al.} \cite{zha2024alertaggregation} study LLM-assisted alert aggregation for AIOps and propose a two-phase approach that first groups alerts by temporal and spatial locality and then uses an LLM to trace cascading service-failure effects. Their results show that LLMs can add value when operational evidence must be connected across dependent services, especially during alert storms. Multi-fault Kubernetes remediation poses the same problem: the system must look beyond a single symptom and decide whether several observations share one cause.\par 📄 s.3
-Zhou \emph{et al.} \cite{zhou2024llmbertlog} examine the use of large language models and BERT for log parsing and anomaly detection. Their journal article treats logs as a primary operational signal and shows that combining LLM-based parsing with transformer-based anomaly detection can improve downstream fault-analysis performance. This is relevant to the current work because Kubernetes remediation agents frequently rely on logs and event streams to infer why pods crash, services lose endpoints, or configurations fail; therefore, the quality of log interpretation can strongly influence the success of autonomous repair.\par
+Zhou \emph{et al.} \cite{zhou2024llmbertlog} examine the use of large language models and BERT for log parsing and anomaly detection. Their journal article treats logs as a primary operational signal and shows that combining LLM-based parsing with transformer-based anomaly detection can improve downstream fault-analysis performance. Remediation agents likewise depend on logs and event streams to infer why pods crash, services lose endpoints, or configurations fail, so the quality of log interpretation directly affects repair success.\par 📄 s.3
-Ma \emph{et al.} \cite{ma2026adaptivelog} introduce AdaptiveLog, a journal framework that coordinates large and small language models for adaptive log analysis. The key idea is that different log-analysis subtasks do not always require the same model capacity, so a system can combine cheaper lightweight models with more capable models when deeper reasoning is needed. This insight aligns with the model-selection findings of the present paper, where mid-tier models provide strong cost-performance trade-offs and high-capability models should be reserved for cases where their additional cost is justified.\par
+Ma \emph{et al.} \cite{ma2026adaptivelog} introduce AdaptiveLog, a journal framework that coordinates large and small language models for adaptive log analysis. The key idea is that different log-analysis subtasks do not always require the same model capacity, so a system can combine cheaper lightweight models with more capable models when deeper reasoning is needed. Our model-selection results point the same way: mid-tier models offer a strong cost-performance trade-off, and high-capability models should be reserved for cases where their additional cost is justified.\par 📄 s.4
-LogLAA \cite{loglaa2026} proposes an adaptive integrated log anomaly analysis framework that combines log parsing, anomaly detection, and LLM-based anomaly interpretation. The study highlights a practical issue for operational LLM use: feeding large volumes of raw logs directly into a model is costly and can reduce analysis quality, whereas structured filtering and targeted anomaly extraction can make LLM explanations more useful. This observation supports the verification-driven design used here, where the agent should not depend on unconstrained free-form reasoning alone but should be guided by structured evidence and deterministic evaluation.\par
+LogLAA \cite{loglaa2026} proposes an adaptive integrated log anomaly analysis framework that combines log parsing, anomaly detection, and LLM-based anomaly interpretation. The study raises a practical issue for operational LLM use: feeding large volumes of raw logs directly into a model is costly and can reduce analysis quality, whereas structured filtering and targeted anomaly extraction can make LLM explanations more useful. The verification-driven design used here follows the same principle: the agent is guided by structured evidence and deterministic evaluation instead of free-form reasoning alone.\par 📄 s.4
+ 📄 s.4
+Table~\ref{tab:related_comparison} summarizes the LLM-based Kubernetes-oriented systems most closely related to this study and contrasts them along the dimensions that define our design: reasoning strategy, memory use, tool orchestration, and verification. The comparison shows the gap this study addresses: prior systems diagnose without applying repairs, coordinate several agents without isolating the contribution of self-reflection, or benchmark agents without a learning-from-failure mechanism. None of them separates the value of a verified retry from the value of verbal self-reflection, which is the central question of this paper. 📄 s.4
+ 📄 s.4
+\begin{table*}[t] 📄 s.4
+\centering 📄 s.5
+\caption{Related LLM-based systems for Kubernetes and cloud operations.} 📄 s.5
+\label{tab:related_comparison} 📄 s.5
+\footnotesize 📄 s.5
+\setlength{\tabcolsep}{4pt} 📄 s.5
+\begin{tabularx}{\textwidth}{l X X X X X} 📄 s.5
+\toprule 📄 s.5
+\textbf{System} & \textbf{Reasoning strategy} & \textbf{Memory} & \textbf{Tool orchestration} & \textbf{Verification} & \textbf{Gap addressed here} \\ 📄 s.5
+\midrule 📄 s.5
+K8sGPT \cite{jones2023k8sgpt} & Rule-based analyzers with LLM-generated explanations & None & Built-in cluster scanners & None; operator applies fixes & Diagnosis only; no autonomous repair loop \\ 📄 s.5
+KubeIntellect \cite{ardebili2025kubeintellect} & Supervisor-coordinated multi-agent workflow & Session context & LLM-orchestrated \texttt{kubectl} and API tools & Human oversight in the loop & End-to-end management; reflection not isolated \\ 📄 s.5
+KubeLLMBench \cite{dejesus2025kubellmbench} & Multi-agent with RAG over Kubernetes documentation & Chat history & \texttt{kubectl} tool calls & Benchmark checks & No learning-from-failure mechanism under test \\ 📄 s.5
+AIOpsLab \cite{chen2025aiopslab} & Agent-agnostic evaluation harness & Depends on agent & Fault-injection framework APIs & Programmatic oracles & Benchmark infrastructure, not an agent design \\ 📄 s.5
+This work & Single ReAct-style actor with Reflexion & Bounded episodic memory of verbal reflections & Claude Agent SDK with \texttt{kubectl} & Deterministic programmatic evaluator & Disentangles verified retry from self-reflection \\ 📄 s.5
+\bottomrule 📄 s.5
+\end{tabularx} 📄 s.5
+\end{table*} 📄 s.5
 \section{Materials and Methods}
 \label{sec:methods}
+This section describes the controlled study design, the four-component agent architecture, the prompt design, the ten fault scenarios, the evaluated models and configurations, and the metrics and statistical procedures used in the analysis. 📄 s.4
 \subsection{Study Design}
 We use a controlled design where the independent variables are the agent configuration, the model/capability tier, and the incident scenario. The main dependent variable is binary remediation success. Additional measurements include the number of trials consumed, execution time, token use, and API cost. To account for variability in LLM behavior, every model--configuration--scenario combination was run ten times.
 The experiment compares three configurations. In the baseline, the agent receives one attempt and no reflection. In the two-try setting, the agent receives a second attempt but no self-reflection, allowing the retry effect to be measured separately. In the full Reflexion setting, the agent receives a second attempt together with natural-language feedback produced after the failed first attempt. Because the budget is fixed at two attempts, each run yields at most one reflection; this isolates a single reflection between two tries rather than the cumulative effect of Reflexion's multi-trial memory loop. With ten scenarios, five models, three configurations, and ten repetitions, the experiment comprises 1,500 runs. The complete thesis version of the experimental protocol, implementation details, appendices, and raw-data schema is documented in the corresponding master's thesis \cite{suerkan2026thesis}.
+The full Reflexion condition combines two factors: access to an external failure signal and the LLM-generated reflection itself. To separate them, we additionally ran a post-hoc \emph{feedback-only} control configuration. In this control, the second attempt is preceded by no reflection call; instead, the episodic memory receives a single verbatim string of the form ``\texttt{The previous attempt failed. Evaluator feedback: \{reason\} (Pod status: \{status\})}'', where \emph{reason} is the programmatic evaluator's textual failure explanation. The Actor prompt and memory-injection mechanism are identical across all retry-capable configurations; only the memory content differs. Because Claude 3 Haiku and Claude 3.5 Haiku had been retired from the Anthropic API by the time of revision, this control was run on the three models that remained available (Haiku 4.5, Sonnet 4.5, and Opus 4.5) on the multi-fault scenarios, where the reflection benefit was concentrated (Section~\ref{sec:feedback_control}). 📄 s.4
+ 📄 s.4
+A second post-hoc ablation addresses the complementary confound on the baseline side. Because the Actor is prohibited from verifying its own fix (Section~\ref{sec:prompts}), the baseline measures a single \emph{unverified} attempt, so the difference between the baseline and the two-try condition bundles an additional attempt with an external verification signal. The \emph{self-verification} configuration keeps the single-attempt, no-reflection design of the baseline but replaces the prompt's stop instruction with an explicit allowance to verify the fix and, if the problem persists, to repeat the diagnose--fix--verify cycle at most three times within the same context window. This ablation was run on the same three models over the eight single-fault scenarios with five repetitions per cell (120 runs) and is compared against the matrix baseline and two-try runs in the same cells (Section~\ref{sec:selfverify}). 📄 s.4
+ 📄 s.4
 \subsection{Agent Architecture}
-As illustrated in Fig.~\ref{fig:architecture}, the remediation system has four components. The Actor is an LLM-driven Kubernetes troubleshooting agent built with the Claude Agent SDK \cite{anthropic2024agentsdk}. It investigates incidents through \texttt{kubectl}, applies repairs, and ends with a fixed completion signal. The Actor prompt follows three phases: diagnosis with \texttt{kubectl get}, \texttt{describe}, \texttt{logs}, and \texttt{get events}; repair with non-interactive commands such as \texttt{patch}, \texttt{set image}, \texttt{apply}, and \texttt{delete}; and termination with a fixed \texttt{FIX COMPLETE} signal. The prompt explicitly forbids interactive editors such as \texttt{kubectl edit}, \texttt{vim}, \texttt{nano}, and \texttt{vi}, and it instructs the Actor not to enter retry loops because external validation is handled separately.
+As illustrated in Fig.~\ref{fig:architecture}, the remediation system has four components. The Actor is an LLM-driven Kubernetes troubleshooting agent built with the Claude Agent SDK, version 0.1.6 \cite{anthropic2024agentsdk}. It investigates incidents through \texttt{kubectl}, applies repairs, and ends with a fixed completion signal. The Actor prompt follows three phases: diagnosis with \texttt{kubectl get}, \texttt{describe}, \texttt{logs}, and \texttt{get events}; repair with non-interactive commands such as \texttt{patch}, \texttt{set image}, \texttt{apply}, and \texttt{delete}; and termination with a fixed \texttt{FIX COMPLETE} signal. The prompt explicitly forbids interactive editors such as \texttt{kubectl edit}, \texttt{vim}, \texttt{nano}, and \texttt{vi}, and it instructs the Actor not to enter retry loops because external validation is handled separately. 📄 s.4
 The Evaluator is deterministic and programmatic, checking cluster state without any LLM call. The Actor never receives the Evaluator's binary success/failure signal; it observes only raw cluster state, including pod status, events, logs, and endpoints, so success labels cannot leak into the Actor's remediation policy. The Evaluator verifies that pods are running, all containers are ready, relevant error states are absent, and service endpoint or HTTP connectivity requirements are met. This avoids both evaluation nondeterminism and extra evaluation cost. Table~\ref{tab:evaluator_checks} summarizes the checks used to decide whether a run is successful or should continue to another attempt.
 \begin{table}[t]
 \centering
 \caption{Programmatic evaluator checks.}
 \label{tab:evaluator_checks}
 \setlength{\tabcolsep}{2pt}
 \begin{tabular}{p{1.65cm} p{2.85cm} p{2.15cm}}
 \toprule
 \textbf{Check} & \textbf{Evidence inspected} & \textbf{Failure signal} \\
 \midrule
 Pod phase & Pod status from Kubernetes API & Not running \\
 Readiness & Ready container count & Not all ready \\
 Error state & Events and crash/image errors & Error reason present \\
 Reachability & Endpoints or HTTP test & No endpoint or failed request \\
 \bottomrule
 \end{tabular}
 \end{table}
 If an attempt fails under the full Reflexion configuration, the Self-Reflection component makes one LLM call to analyze the Actor's trajectory and the evaluator result. The reflection prompt includes the attempted trajectory, failure reason, pod status, readiness information, and previous episodic memories. It requires three structured parts: credit assignment, root-cause analysis, and concrete next steps. Reflections are written in the first person and are constrained to be specific, actionable, analytical, and concise. This text is saved in episodic memory and passed to the Actor on the next trial. Episodic memory is reset at the start of every run, so no reflection is shared across repetitions, scenarios, or models. Because the present protocol permits at most two attempts per run, only one reflection can be generated within a run; the three-reflection cap is an implementation safeguard inherited from the general Reflexion framework and is not exercised by the reported two-attempt experiments.
 \begin{figure*}[t]
 \centering
 \includegraphics[width=0.82\textwidth]{figures/akis.png}
 \caption{Reflexion-based remediation architecture.}
 \label{fig:architecture}
 \end{figure*}
 \subsection{Prompt Design}
 \label{sec:prompts}
 The Actor follows a three-phase system prompt: (1)~\emph{diagnose} with
 read-only \texttt{kubectl get}, \texttt{describe}, \texttt{logs}, and
 \texttt{get events}; (2)~\emph{fix} with non-interactive \texttt{patch},
 \texttt{set image}, \texttt{apply}, or \texttt{delete}; and (3)~\emph{stop} by
 emitting \texttt{FIX COMPLETE}. The prompt explicitly forbids interactive editors
 (\texttt{kubectl edit}, \texttt{vim}, \texttt{nano}, \texttt{vi}) and instructs
 the Actor not to verify its own fix or enter retry loops, because verification is
-delegated to the deterministic Evaluator.
+delegated to the deterministic Evaluator. The self-verification ablation 📄 s.5
+(Section~\ref{sec:selfverify}) uses a variant of this prompt in which Phase~3 📄 s.5
+instead instructs the Actor to re-check the pod after the fix and, if the problem 📄 s.5
+persists, to return to diagnosis and repeat the diagnose--fix--verify cycle at 📄 s.5
+most three times before emitting \texttt{FIX COMPLETE}; all other prompt content 📄 s.5
+is identical. 📄 s.5
 The Self-Reflection module is prompted to act as a verbal reinforcement signal.
 Its system prompt requires the reflection to be specific, actionable, analytical,
 first-person, and concise (200--400 words). Its user template is populated with
 the failed trajectory, the Evaluator reason, pod status and readiness, and prior
 reflections, and it mandates three structured parts: (i)~\emph{credit assignment}
 (which action or decision caused the failure), (ii)~\emph{root cause} (what state
 was misunderstood or missed), and (iii)~\emph{concrete next steps} (what to check
 or change next). These reflections are stored in episodic memory and retrieved
 (top-$k=3$) for the following trial.
 \subsection{Test Scenarios}
-The benchmark includes ten Kubernetes misconfiguration scenarios. Eight are single-fault cases derived from Kubernetes troubleshooting benchmarks, and two are multi-fault cases intended to resemble production incidents in which an initial repair exposes or leaves other faults. Case 9 combines an image tag typo with a service \texttt{targetPort} mismatch: after the image tag is corrected, the service can still route traffic to the wrong port. Case 10 combines an image typo, a missing ConfigMap mount, and a wrong service \texttt{targetPort}, requiring several resources to be repaired in sequence. Automated setup and cleanup scripts ensure that every run begins from an identical state. Table~\ref{tab:scenarios} summarizes these cases and makes the 8/1/1 split across single-, double-, and triple-fault scenarios explicit.
+The benchmark includes ten Kubernetes misconfiguration scenarios. Eight are single-fault cases derived from Kubernetes troubleshooting benchmarks, and two are multi-fault cases intended to resemble production incidents in which an initial repair exposes or leaves other faults. Case 9 combines an image tag typo with a service \texttt{targetPort} mismatch: after the image tag is corrected, the service can still route traffic to the wrong port. Case 10 combines an image typo, a missing ConfigMap mount, and a wrong service \texttt{targetPort}, requiring several resources to be repaired in sequence. Automated setup and cleanup scripts ensure that every run begins from an identical state. Table~\ref{tab:scenarios} summarizes these cases and makes the 8/1/1 split across single-, double-, and triple-fault scenarios explicit. Instead of a priori difficulty labels, the table reports the empirical success rate observed across all models and configurations, because hand-assigned labels aligned only loosely with the difficulty actually observed in the experiments. 📄 s.6
 \begin{table*}[t]
 \centering
 \caption{Evaluated Kubernetes incidents.}
 \label{tab:scenarios}
 \setlength{\tabcolsep}{3pt}
-\begin{tabularx}{0.94\textwidth}{c l c X l}
+\begin{tabularx}{0.94\textwidth}{c l c X r} 📄 s.7
 \toprule
-\textbf{Case} & \textbf{Scenario} & \textbf{Faults} & \textbf{Main symptom} & \textbf{Difficulty} \\
+\textbf{Case} & \textbf{Scenario} & \textbf{Faults} & \textbf{Main symptom} & \textbf{Overall SR} \\ 📄 s.7
 \midrule
-1 & Wrong port & 1 & Pod running but unreachable through the service & Hard \\
+1 & Wrong port & 1 & Pod running but unreachable through the service & 38.7\% \\ 📄 s.7
-2 & Incorrect selector & 1 & Service has no endpoints & Easy \\
+2 & Incorrect selector & 1 & Service has no endpoints & 67.3\% \\ 📄 s.7
-3 & Liveness probe & 1 & Pod enters CrashLoopBackOff after probe failures & Medium \\
+3 & Liveness probe & 1 & Pod enters CrashLoopBackOff after probe failures & 64.0\% \\ 📄 s.7
-4 & Wrong interface & 1 & Application binds to localhost and cannot be reached by service & Hard \\
+4 & Wrong interface & 1 & Application binds to localhost and cannot be reached by service & 60.7\% \\ 📄 s.7
-5 & Port mismatch & 1 & Connection refused due to inconsistent ports & Medium \\
+5 & Port mismatch & 1 & Connection refused due to inconsistent ports & 74.0\% \\ 📄 s.7
-6 & Misspelling & 1 & Typo in image, label, or selector causes a failure symptom & Easy \\
+6 & Misspelling & 1 & Typo in image, label, or selector causes a failure symptom & 92.0\% \\ 📄 s.7
-7 & Volume mount & 1 & CreateContainerConfigError due to ConfigMap reference mismatch & Medium \\
+7 & Volume mount & 1 & CreateContainerConfigError due to ConfigMap reference mismatch & 44.0\% \\ 📄 s.7
-8 & Environment variable & 1 & Application crashes at startup due to missing variables & Medium \\
+8 & Environment variable & 1 & Application crashes at startup due to missing variables & 70.7\% \\ 📄 s.7
-9 & Double trouble & 2 & Image tag typo plus service targetPort mismatch & Hard \\
+9 & Double trouble & 2 & Image tag typo plus service targetPort mismatch & 45.3\% \\ 📄 s.7
-10 & Triple threat & 3 & Image typo, missing ConfigMap mount, and wrong targetPort & Very hard \\
+10 & Triple threat & 3 & Image typo, missing ConfigMap mount, and wrong targetPort & 30.7\% \\ 📄 s.7
 \bottomrule
 \end{tabularx}
+\par\vspace{3pt}\begin{minipage}{0.94\textwidth}\footnotesize\raggedright 📄 s.7
+Overall SR: empirical success rate over all models and configurations (150 runs per case; see Table~\ref{tab:case_success}). 📄 s.7
+\end{minipage} 📄 s.7
 \end{table*}
 \subsection{Models and Configurations}
-We evaluated five Claude models representing different capability levels: Claude Haiku 3.0, Claude Haiku 3.5, Claude Haiku 4.5, Claude Sonnet 4.5, and Claude Opus 4.5. These models are treated as a spectrum from lightweight to high-capability systems. All runs used the Claude Agent SDK's default API parameters.
+We evaluated five Anthropic Claude models representing different capability levels, listed here with their official names: Claude 3 Haiku, Claude 3.5 Haiku, Claude Haiku 4.5, Claude Sonnet 4.5, and Claude Opus 4.5 (the exact API identifiers are given in the Data and Code Availability section). For brevity, tables and figures refer to them as Haiku 3.0, Haiku 3.5, Haiku 4.5, Sonnet 4.5, and Opus 4.5. These models are treated as a spectrum from lightweight to high-capability systems. The evaluation was deliberately restricted to a single model family: the Actor's tool-use behavior is mediated by the Claude Agent SDK, so holding the SDK, prompts, and tool interface constant isolates the effect of model capability from framework differences, at the acknowledged cost of cross-family generality (Section~\ref{sec:discussion}-C). All runs used the Claude Agent SDK's default API parameters. Table~\ref{tab:configs} summarizes the three main experimental configurations and the two post-hoc controls (feedback-only and self-verification) described in Section~\ref{sec:methods}-A. 📄 s.6
 \begin{table}[t]
 \centering
 \caption{Experimental configurations.}
 \label{tab:configs}
 \begin{tabular}{p{1.7cm} c c p{2.25cm}}
 \toprule
 \textbf{Config.} & \textbf{Trials} & \textbf{Refl.} & \textbf{Purpose} \\
 \midrule
 Baseline & 1 & No & Measure one-shot remediation performance \\
 Two-try & 2 & No & Measure retry value separately from reflection \\
 Full Reflexion & 2 & Yes & Measure reflection and memory value \\
+Feedback-only & 2 & No & Isolate evaluator feedback from reflection (post-hoc control) \\ 📄 s.7
+Self-verify & 1 & No & Permit in-attempt verification and revision (post-hoc ablation) \\ 📄 s.7
 \bottomrule
 \end{tabular}
 \end{table}
 \subsection{Evaluation Metrics and Statistical Analysis}
 The main metric is success rate, defined as the share of runs that the programmatic evaluator judged to be remediated. For configuration $c$, success rate is calculated as
 \begin{equation}
 SR_c = \frac{1}{N_c}\sum_{i=1}^{N_c}\mathbb{I}(S_i=\text{success}),
 \end{equation}
 where $N_c$ denotes the number of runs for configuration $c$, $S_i$ is the outcome for run $i$, and $\mathbb{I}(\cdot)$ denotes the indicator function.
-The second-attempt effect is measured as the difference between the two-try and baseline settings. Reflexion benefit is measured as the difference between full Reflexion and the two-try setting without reflection. Cost is computed from token counts and model prices, and execution time is recorded in seconds. The experiment logs also store trials used, total token counts, command trajectories, evaluator reasons, and per-run JSON records that were exported into an SQLite database for analysis. Statistical comparisons use chi-square independence tests, Wilson 95\% confidence intervals for proportions, and Cohen's $h$ to interpret effect sizes. These tests treat runs as Bernoulli observations; however, repeated runs of the same model--scenario pair and differences in scenario difficulty introduce clustering, so the inferential results should be interpreted as descriptive rather than as a fully independent-sample analysis. A mixed-effects logistic model or clustered generalized estimating equation with scenario and model effects is a rigorous next step.
+The second-attempt effect is measured as the difference between the two-try and baseline settings. Reflexion benefit is measured as the difference between full Reflexion and the two-try setting without reflection. Cost is computed from token counts and model prices, and execution time is recorded in seconds. The experiment logs also store trials used, total token counts, command trajectories, evaluator reasons, and per-run JSON records that were exported into an SQLite database for analysis. Statistical comparisons use chi-square independence tests, Wilson 95\% confidence intervals for proportions, and Cohen's $h$ to interpret effect sizes. These tests treat runs as Bernoulli observations; however, repeated runs of the same model--scenario pair and differences in scenario difficulty introduce clustering, so the pooled inferential results should be interpreted as descriptive. To address this, Section~\ref{sec:results}-G additionally reports a mixed-effects logistic model with random intercepts for scenario and model and a clustered generalized estimating equation fitted on the per-run records. 📄 s.6
 \section{Experiments And Results}
 \label{sec:results}
+This section reports the empirical findings: overall configuration performance, model- and scenario-level breakdowns, the retry-versus-reflection ablation, the complexity analysis, cost and runtime measurements, statistical tests, and a qualitative analysis of the generated reflections. 📄 s.7
 \subsection{Overall Performance}
-Table~\ref{tab:overall} and Fig.~\ref{fig:overall} present the aggregate results. The one-attempt baseline achieved a 38.6\% success rate. Providing a second attempt without reflection raised success to 68.4\%, an improvement of 29.8 percentage points. Full Reflexion produced the highest overall success rate at 69.2\%, but this was only 0.8 percentage points above the non-reflective two-try setting. Because Reflexion adds another LLM call after failed attempts, it also increased both mean runtime and cost.
+Table~\ref{tab:overall} and Fig.~\ref{fig:overall} present the aggregate results. The one-attempt baseline achieved a 38.6\% success rate. Providing a second attempt without reflection, triggered by the external evaluator's failure signal, raised success to 68.4\%, an improvement of 29.8 percentage points. Full Reflexion produced the highest overall success rate at 69.2\%, but this was only 0.8 percentage points above the non-reflective two-try setting. Because Reflexion adds another LLM call after failed attempts, it also increased both mean runtime and cost. 📄 s.7
 \begin{table}[t]
 \centering
 \caption{Aggregate configuration performance.}
 \label{tab:overall}
 \begin{tabular}{l r r r r r}
 \toprule
 \textbf{Configuration} & \textbf{Runs} & \textbf{Succ.} & \textbf{SR} & \textbf{Cost} & \textbf{Time} \\
 \midrule
 Baseline & 500 & 193 & 38.6\% & \$0.108 & 83 s \\
 Two-try & 500 & 342 & 68.4\% & \$0.155 & 110 s \\
 Full Reflexion & 500 & 346 & 69.2\% & \$0.179 & 173 s \\
 \bottomrule
 \end{tabular}
 \end{table}
 \begin{figure}[t]
 \centering
 \includegraphics[width=\columnwidth]{figures/basari.png}
 \caption{Overall success by configuration.}
 \label{fig:overall}
 \end{figure}
 \subsection{Model-Level Performance}
-Remediation success was strongly influenced by model capability. Opus 4.5 achieved the best overall success rate, whereas Sonnet 4.5 and Haiku 4.5 performed at similar levels. The older and less capable Haiku models lagged well behind. The cost-efficiency results show that the strongest model is not automatically the most economical option: among the models with strong performance, Haiku 4.5 combined high success with the lowest average cost.
+Remediation success was strongly influenced by model capability, as shown in Table~\ref{tab:model_overall}. Opus 4.5 achieved the best overall success rate, whereas Sonnet 4.5 and Haiku 4.5 performed at similar levels. The older and less capable Haiku models lagged well behind. The cost-efficiency results show that the strongest model is not automatically the most economical option: among the models with strong performance, Haiku 4.5 combined high success with the lowest average cost. 📄 s.7
 \begin{table}[t]
 \centering
 \caption{Model-level success and cost.}
 \label{tab:model_overall}
 \begin{tabular}{l r r r}
 \toprule
 \textbf{Model} & \textbf{Success rate} & \textbf{Avg. cost} & \textbf{Runs} \\
 \midrule
 Claude Opus 4.5 & 87.3\% & \$0.191 & 300 \\
 Claude Sonnet 4.5 & 73.3\% & \$0.184 & 300 \\
 Claude Haiku 4.5 & 73.0\% & \$0.105 & 300 \\
 Claude Haiku 3.5 & 34.7\% & \$0.080 & 300 \\
 Claude Haiku 3.0 & 25.3\% & \$0.176 & 300 \\
 \bottomrule
 \end{tabular}
 \end{table}
 \subsection{Scenario-Level Performance}
-Incident difficulty differed considerably across scenarios. Typographical errors and selector or port cases were generally easier, whereas wrong-port, volume-mount, and multi-fault cases were more challenging. The triple-fault case had the lowest overall success rate, suggesting that concurrent defects substantially weaken autonomous remediation reliability.
+Incident difficulty differed considerably across scenarios, as reported in Table~\ref{tab:case_success}. Typographical errors and selector or port cases were generally easier, whereas wrong-port, volume-mount, and multi-fault cases were more challenging. The triple-fault case had the lowest overall success rate, suggesting that concurrent defects substantially weaken autonomous remediation reliability. 📄 s.7
 \begin{table}[t]
 \centering
 \caption{Test-case success rates.}
 \label{tab:case_success}
 \begin{tabular}{l l c r}
 \toprule
 \textbf{Case} & \textbf{Description} & \textbf{Faults} & \textbf{SR} \\
 \midrule
 case6 & Image typo & 1 & 92.0\% \\
 case5 & Port mismatch & 1 & 74.0\% \\
 case8 & Environment variable & 1 & 70.7\% \\
 case2 & Incorrect selector & 1 & 67.3\% \\
 case3 & Liveness probe & 1 & 64.0\% \\
 case4 & Wrong interface & 1 & 60.7\% \\
 case9 & Double trouble & 2 & 45.3\% \\
 case7 & Volume mount & 1 & 44.0\% \\
 case1 & Wrong port & 1 & 38.7\% \\
 case10 & Triple threat & 3 & 30.7\% \\
 \bottomrule
 \end{tabular}
 \end{table}
 \subsection{Ablation: Retry Versus Reflexion}
-Table~\ref{tab:ablation} separates the contribution of retrying from the contribution of explicit reflection. Every model benefited from a second attempt, with improvements ranging from +15 to +37 percentage points. Reflexion produced clear additional gains only for mid-capability models: Haiku 4.5 increased from 84\% to 88\%, and Sonnet 4.5 increased from 84\% to 87\%. Opus 4.5 did not improve further because it had already reached 98\% with two attempts. Haiku 3.0 and Haiku 3.5 showed negligible or slightly negative changes, indicating that weaker models may create reflections they cannot use effectively.
+Table~\ref{tab:ablation} separates the contribution of retrying from the contribution of explicit reflection. Every model benefited from a second attempt, with improvements ranging from +15 to +37 percentage points. Reflexion produced clear additional gains only for mid-capability models: Haiku 4.5 increased from 84\% to 88\%, and Sonnet 4.5 increased from 84\% to 87\%. Opus 4.5 did not improve further because it had already reached 98\% with two attempts. Haiku 3.0 and Haiku 3.5 showed negligible or slightly negative changes, indicating that weaker models may create reflections they cannot use effectively. Fig.~\ref{fig:model_comparison} visualizes this non-linear relationship between model capability and reflection benefit. 📄 s.7
+ 📄 s.7
+Table~\ref{tab:breakdown} reports the attempt-level breakdown underlying these aggregates, restoring the recovery-rate analysis from the thesis protocol \cite{suerkan2026thesis}. The table supports three observations. First, first-attempt success in the retry-capable configurations (38.0\% and 40.2\%) is consistent with the one-attempt baseline (38.6\%), confirming that the conditions differ only in what happens after a failed first attempt. Second, the full Reflexion condition generated exactly 310 reflections, one per failed first attempt; no reflection follows a failed second attempt because no further trial exists. Third, the recovery rate after a failed first attempt was 50.3\% (156/310) with reflection versus 47.2\% (141/299) without, a modest difference consistent with the small aggregate gap between the two configurations. 📄 s.8
 \begin{table}[t]
 \centering
-\caption{Model-level ablation results.}
+\caption{Attempt-level breakdown of the retry-capable configurations.} 📄 s.8
+\label{tab:breakdown} 📄 s.8
+\setlength{\tabcolsep}{4pt} 📄 s.8
+\begin{tabular}{l r r r r r} 📄 s.8
+\toprule 📄 s.8
+\textbf{Model} & \textbf{1st} & \textbf{Refl.} & \textbf{2nd} & \textbf{Fail} & \textbf{SR} \\ 📄 s.8
+\midrule 📄 s.8
+\multicolumn{6}{l}{\emph{Full Reflexion}} \\ 📄 s.8
+Haiku 3.0 & 22 & 78 & 7 & 71 & 29\% \\ 📄 s.8
+Haiku 3.5 & 16 & 84 & 28 & 56 & 44\% \\ 📄 s.8
+Haiku 4.5 & 41 & 59 & 47 & 12 & 88\% \\ 📄 s.8
+Sonnet 4.5 & 44 & 56 & 43 & 13 & 87\% \\ 📄 s.8
+Opus 4.5 & 67 & 33 & 31 & 2 & 98\% \\ 📄 s.8
+Total & 190 & 310 & 156 & 154 & 69.2\% \\ 📄 s.8
+\midrule 📄 s.8
+\multicolumn{6}{l}{\emph{Two-try (no reflection)}} \\ 📄 s.8
+Haiku 3.0 & 16 & -- & 15 & 69 & 31\% \\ 📄 s.8
+Haiku 3.5 & 17 & -- & 28 & 55 & 45\% \\ 📄 s.8
+Haiku 4.5 & 49 & -- & 35 & 16 & 84\% \\ 📄 s.8
+Sonnet 4.5 & 50 & -- & 34 & 16 & 84\% \\ 📄 s.8
+Opus 4.5 & 69 & -- & 29 & 2 & 98\% \\ 📄 s.8
+Total & 201 & -- & 141 & 158 & 68.4\% \\ 📄 s.8
+\bottomrule 📄 s.8
+\end{tabular} 📄 s.8
+\par\vspace{3pt}\begin{minipage}{\columnwidth}\footnotesize\raggedright 📄 s.8
+1st = successful first attempts; Refl.\ = reflections generated; 2nd = successful second attempts; Fail = final failures. Each model--configuration cell comprises 100 runs. 📄 s.8
+\end{minipage} 📄 s.8
+\end{table} 📄 s.8
+ 📄 s.8
+\begin{table}[t] 📄 s.8
+\centering 📄 s.8
+\caption{Model-level ablation results ($\boldsymbol{\Delta}$Two = Two-try minus Baseline success rate; $\boldsymbol{\Delta}$Refl.\ = Full Reflexion minus Baseline success rate, in percentage points).} 📄 s.8
 \label{tab:ablation}
 \begin{tabular}{l r r r r r}
 \toprule
 \textbf{Model} & \textbf{Base} & \textbf{Two} & \textbf{Refl.} & $\Delta$Two & $\Delta$Refl. \\
 \midrule
 Opus 4.5 & 66\% & 98\% & 98\% & +32 pp & +32 pp \\
 Sonnet 4.5 & 49\% & 84\% & 87\% & +35 pp & +38 pp \\
 Haiku 4.5 & 47\% & 84\% & 88\% & +37 pp & +41 pp \\
 Haiku 3.5 & 15\% & 45\% & 44\% & +30 pp & +29 pp \\
 Haiku 3.0 & 16\% & 31\% & 29\% & +15 pp & +13 pp \\
 \bottomrule
 \end{tabular}
 \end{table}
 \begin{figure}[t]
 \centering
 \includegraphics[width=\columnwidth]{figures/karsilastirma.png}
 \caption{Model capability and Reflexion benefit.}
 \label{fig:model_comparison}
 \end{figure}
 \subsection{Problem Complexity and Reflexion Benefit}
-The strongest preliminary signal for Reflexion emerges after grouping scenarios by complexity. As shown in Table~\ref{tab:complexity} and Fig.~\ref{fig:complexity}, Reflexion lowered success by 2.0 percentage points for single-fault cases, but raised success by 12.0 percentage points for the double-fault case and by 12.0 percentage points for the triple-fault case. Because each multi-fault category contains only one scenario, these values should be interpreted as scenario-level evidence that motivates further testing, not as a definitive complexity effect. The pattern suggests that reflection may be most beneficial when an incomplete repair forces the agent to revise its causal explanation or when several resources must be fixed sequentially.
+Grouping scenarios by complexity gives the clearest, though preliminary, signal for Reflexion. As shown in Table~\ref{tab:complexity} and Fig.~\ref{fig:complexity}, Reflexion lowered success by 2.0 percentage points for single-fault cases, but raised success by 12.0 percentage points for the double-fault case and by 12.0 percentage points for the triple-fault case. Because each multi-fault category contains only one scenario, these values should be interpreted as scenario-level evidence that motivates further testing, not as a definitive complexity effect. The pattern suggests that reflection may be most beneficial when an incomplete repair forces the agent to revise its causal explanation or when several resources must be fixed sequentially. 📄 s.8
 \begin{table}[t]
 \centering
 \caption{Reflexion benefit by problem complexity.}
 \label{tab:complexity}
 \begin{tabular}{l l r r r}
 \toprule
 \textbf{Complexity} & \textbf{Cases} & \textbf{Two-try} & \textbf{Reflexion} & \textbf{Benefit} \\
 \midrule
 Single-fault & case1--8 & 72.8\% & 70.8\% & -2.0 pp \\
 Double-fault & case9 & 62.0\% & 74.0\% & +12.0 pp \\
 Triple-fault & case10 & 40.0\% & 52.0\% & +12.0 pp \\
 \bottomrule
 \end{tabular}
 \end{table}
 \begin{figure}[t]
 \centering
 \includegraphics[width=\columnwidth]{figures/karsilastirma-2.png}
 \caption{Complexity-level Reflexion gain.}
 \label{fig:complexity}
 \end{figure}
+ 📄 s.8
+\subsection{Feedback-Only Control on Multi-Fault Scenarios} 📄 s.8
+\label{sec:feedback_control} 📄 s.8
+The full Reflexion condition combines two factors: an external failure signal and the LLM-generated verbal reflection. To separate them, the post-hoc feedback-only control (Section~\ref{sec:methods}-A) supplies the second attempt with the evaluator's verbatim failure reason and pod status, but no reflection. The control comprises 142 runs on the three current models: the two multi-fault scenarios (110 runs) and an additional exploratory two-fault scenario outside the ten-scenario matrix (32 runs), discussed below. 📄 s.8
+ 📄 s.8
+One difference in scenario provenance constrains the comparison. The control ran on the current replication-package instantiation of the multi-fault scenarios, which differs from the instantiation used for the original 1,500-run matrix in one respect: the agent's diagnostic entry point. In the matrix runs, the agent was pointed at a neighboring client pod and had to explore the namespace to locate the faults; in the current instantiation, the agent is pointed directly at the faulty deployment. This change substantially increases difficulty, because agents entering at the faulty pod typically repair the pod-level fault, observe a \texttt{Running} pod, and terminate without examining the service path. All comparisons in this subsection therefore use only runs executed on the identical current instantiation (the post-freeze replication runs of the two-try and full Reflexion arms and the new feedback-only runs), and the values in Table~\ref{tab:feedback_control} are consequently not comparable to the matrix values in Table~\ref{tab:complexity}. 📄 s.8
+ 📄 s.8
+Table~\ref{tab:feedback_control} reports the outcome. Under this instantiation, the plain two-try condition never recovered on the multi-fault scenarios (0/50), full Reflexion recovered a minority of runs (13/50, 26.0\%), and the feedback-only control recovered 84/110 (76.4\%). The feedback-only advantage is significant against both plain two-try ($\chi^2(1)=80.4$, $p<0.001$) and full Reflexion ($\chi^2(1)=36.5$, $p<0.001$), and it replicates on the exploratory two-fault scenario (a ConfigMap reference combined with a selector mismatch): 96.9\% (31/32) versus 54.8\% (17/31) for Reflexion ($\chi^2(1)=15.3$, $p<0.001$). Reflexion itself still improved markedly over plain retry under this instantiation ($\chi^2(1)=14.9$, $p<0.001$), confirming that reflection transmits part of the failure signal; the verbatim feedback simply transmits it more completely. The pattern is capability-dependent in the same direction as the main results: on the two multi-fault scenarios, Sonnet 4.5 (40/40) and Opus 4.5 (30/30) converted the verbatim failure reason into a successful repair in every run, whereas Haiku 4.5 remained well below ceiling on Case 9 (10/25) and Case 10 (4/15), indicating that an explicit failure signal does not compensate for limited tool-use capability. 📄 s.9
+ 📄 s.9
+A second caveat is that the control receives more information than the reflection module. In the implemented pipeline, the reflection module conditions on the failed trajectory and the pod status but not on the evaluator's textual failure reason, whereas the feedback-only memory contains that reason verbatim. The comparison therefore does not show that verbal reflection is harmful; it shows that the evaluator's signal is what matters most and that the reflection stage, as implemented here, does not reliably reconstruct it from the trajectory alone. 📄 s.9
+ 📄 s.9
+\begin{table}[t] 📄 s.9
+\centering 📄 s.9
+\caption{Feedback-only control on the current scenario instantiation.} 📄 s.9
+\label{tab:feedback_control} 📄 s.9
+\setlength{\tabcolsep}{4pt} 📄 s.9
+\begin{tabular}{l r r r} 📄 s.9
+\toprule 📄 s.9
+\textbf{Config.} & \textbf{Case 9} & \textbf{Case 10} & \textbf{Case 11} \\ 📄 s.9
+\midrule 📄 s.9
+Two-try & 0.0\% (0/35) & 0.0\% (0/15) & 53.1\% (17/32) \\ 📄 s.9
+Full Reflexion & 34.3\% (12/35) & 6.7\% (1/15) & 54.8\% (17/31) \\ 📄 s.9
+Feedback-only & 76.9\% (50/65) & 75.6\% (34/45) & 96.9\% (31/32) \\ 📄 s.9
+\bottomrule 📄 s.9
+\end{tabular} 📄 s.9
+\par\vspace{3pt}\begin{minipage}{\columnwidth}\footnotesize\raggedright 📄 s.9
+Models: Haiku 4.5, Sonnet 4.5, and Opus 4.5. Cases 9--10 are the multi-fault scenarios in their current instantiation (see text); Case 11 is an exploratory two-fault scenario outside the ten-scenario matrix. All three configurations are evaluated on the identical instantiation, so these values are not comparable to Table~\ref{tab:complexity}. 📄 s.9
+\end{minipage} 📄 s.9
+\end{table} 📄 s.9
+ 📄 s.9
+\subsection{Self-Verification Ablation on Single-Fault Scenarios} 📄 s.9
+\label{sec:selfverify} 📄 s.9
+The baseline prohibits the Actor from verifying its own fix, so the retry effect reported in Section~\ref{sec:results}-D combines two factors: a second attempt and an external verification signal. To bound the share attributable to verification alone, the post-hoc self-verification ablation (Section~\ref{sec:methods}-A) gives the Actor a single attempt without reflection but permits up to three diagnose--fix--verify cycles within that attempt. The ablation comprises 120 runs (the three current models on the eight single-fault scenarios, five repetitions per cell) and is compared against the matrix baseline and two-try runs in the same 24 model--scenario cells (240 runs each). The single-fault scenarios were not modified between the matrix runs and the revision, so, unlike the feedback-only control, this comparison uses the identical scenario instantiation. 📄 s.9
+ 📄 s.9
+Table~\ref{tab:selfverify} reports the outcome. Permitting self-verification raised single-attempt success from 67.5\% (162/240) to 80.8\% (97/120; $\chi^2(1)=7.05$, $p=0.008$), recovering roughly half of the 25.0-percentage-point gap between the baseline and the two-try condition in these cells (92.5\%, 222/240). The externally verified second attempt nevertheless remained significantly better than the self-verifying single attempt ($\chi^2(1)=10.8$, $p=0.001$). The effect is capability-dependent in the same direction as the main results: Haiku 4.5 did not benefit (58.8\% versus 60.0\%, $p=0.90$), whereas Sonnet 4.5 (61.2\% to 82.5\%, $\chi^2(1)=5.56$, $p=0.018$) and Opus 4.5 (82.5\% to 100\%, Fisher's exact $p=0.005$) converted self-checks into successful repairs; for Sonnet 4.5 the self-verifying single attempt was statistically indistinguishable from its two-try result (87.5\%, $p=0.46$). 📄 s.9
+ 📄 s.9
+Two scenario-level details qualify this result. First, the gain is concentrated in Case~7 (volume mount), where the baseline succeeded in 4 of 30 runs but self-verification in 14 of 15. In all 30 baseline runs the agents applied the correct repair (creating the missing ConfigMap) and terminated after a median of 40 seconds, while the pod was still \texttt{Pending} waiting for the volume to mount; because the Evaluator samples the cluster immediately after the Actor stops, 26 of these runs were scored as failures. Self-verifying agents instead polled the pod until it reached \texttt{Running} (median 83 seconds). Part of the measured value of verification is therefore the value of waiting for the cluster to converge rather than of diagnostic correction. Excluding Case~7, self-verification changed success from 75.2\% to 79.0\% ($\chi^2(1)=0.57$, $p=0.45$), whereas the two-try condition reached 94.3\% and remained superior ($\chi^2(1)=16.9$, $p<0.001$). Second, self-verification can mislead. In Case~4 (wrong interface), Haiku 4.5 failed in all five self-verifying runs but succeeded in 9 of 10 baseline runs: the self-verifying agents probed the application with \texttt{curl} against \texttt{localhost} from inside the pod, which succeeded because the application was bound to the loopback interface, and accepted this local check as confirmation although the service remained unreachable from outside the pod, which is exactly what the external Evaluator tests. Self-verification is also more expensive: \$0.28 and 139 seconds per run, against \$0.12 and 94 seconds for the baseline and \$0.16 and 100 seconds for two-try, because every verification cycle extends a single, growing context window. The ablation therefore indicates that the retry effect should be read as the value of an additional, externally verified attempt, of which in-attempt self-verification captures only part, chiefly for stronger models and at higher cost. 📄 s.10
+ 📄 s.10
+\begin{table}[t] 📄 s.10
+\centering 📄 s.10
+\caption{Self-verification ablation on the single-fault scenarios.} 📄 s.10
+\label{tab:selfverify} 📄 s.10
+\setlength{\tabcolsep}{4pt} 📄 s.10
+\begin{tabular}{l r r r} 📄 s.10
+\toprule 📄 s.10
+\textbf{Model} & \textbf{Baseline} & \textbf{Self-verify} & \textbf{Two-try} \\ 📄 s.10
+\midrule 📄 s.10
+Haiku 4.5 & 58.8\% (47/80) & 60.0\% (24/40) & 90.0\% (72/80) \\ 📄 s.10
+Sonnet 4.5 & 61.2\% (49/80) & 82.5\% (33/40) & 87.5\% (70/80) \\ 📄 s.10
+Opus 4.5 & 82.5\% (66/80) & 100.0\% (40/40) & 100.0\% (80/80) \\ 📄 s.10
+\midrule 📄 s.10
+All & 67.5\% (162/240) & 80.8\% (97/120) & 92.5\% (222/240) \\ 📄 s.10
+Excl.\ Case 7 & 75.2\% (158/210) & 79.0\% (83/105) & 94.3\% (198/210) \\ 📄 s.10
+Cost per run & \$0.12 & \$0.28 & \$0.16 \\ 📄 s.10
+\bottomrule 📄 s.10
+\end{tabular} 📄 s.10
+\par\vspace{3pt}\begin{minipage}{\columnwidth}\footnotesize\raggedright 📄 s.10
+Models: Haiku 4.5, Sonnet 4.5, and Opus 4.5 on Cases 1--8. Baseline and Two-try are the matrix runs in the same model--scenario cells (ten repetitions per cell); Self-verify is the post-hoc single-attempt configuration with in-attempt verification permitted (five repetitions per cell). All three arms use the identical scenario instantiation. 📄 s.10
+\end{minipage} 📄 s.10
+\end{table} 📄 s.10
 \subsection{Cost and Execution Time}
 Table~\ref{tab:cost_efficiency} reports cost efficiency for the retry-capable configurations only. Haiku 4.5 had the lowest cost per successful remediation while still maintaining a high success rate. Opus 4.5 achieved the highest absolute success, but with a higher average cost. Although Haiku 3.0 is lightweight, it was inefficient because its low success rate increased the cost of each successful remediation.
 \begin{table}[t]
 \centering
 \caption{Cost per successful remediation.}
 \label{tab:cost_efficiency}
 \begin{tabular}{l r r r}
 \toprule
 \textbf{Model} & \textbf{Success rate} & \textbf{Avg. cost} & \textbf{Cost/success} \\
 \midrule
 Haiku 4.5 & 86.0\% & \$0.116 & \$0.135 \\
 Haiku 3.5 & 44.5\% & \$0.091 & \$0.205 \\
 Opus 4.5 & 98.0\% & \$0.209 & \$0.213 \\
 Sonnet 4.5 & 85.5\% & \$0.204 & \$0.239 \\
 Haiku 3.0 & 30.0\% & \$0.214 & \$0.713 \\
 \bottomrule
 \end{tabular}
 \end{table}
 Runtime measurements reveal the same trade-off. Since Reflexion makes an extra LLM call after failures, it increases average execution time. For instance, mean runtime rose from 110 seconds in the two-try setting to 173 seconds in the full Reflexion setting. This latency cost supports using reflection selectively for complex, unresolved, or high-impact incidents instead of simple single-fault problems.
 \subsection{Statistical Analysis}
 The overall gain from baseline to two-try was statistically significant, and the gain from baseline to full Reflexion was also significant. In contrast, the aggregate difference between two-try and full Reflexion was not statistically significant. Table~\ref{tab:stats} gives the chi-square comparisons and effect sizes, while Table~\ref{tab:ci} shows that the confidence intervals for two-try and full Reflexion overlap substantially. The complexity-specific Reflexion improvement in the multi-fault cases should therefore be interpreted cautiously: the experimental analysis indicated that roughly 150--200 additional samples per condition would be needed to test that effect size with stronger statistical power at $\alpha=0.05$.
 To avoid the pseudo-replication induced by treating the 500 correlated runs
 per configuration as independent, we additionally analyzed the configuration
 effects using the model as the unit of analysis ($n=5$), as summarized in
 Table~\ref{tab:model_unit}. Paired comparisons across models confirmed the
 pooled findings: the retry effect was large and significant
 (mean $+29.8$ pp, paired $t(4)=7.66$, $p=0.002$, Cohen's $d_z=3.43$),
 whereas the additional effect of reflection over a plain second attempt was
 small and not significant (mean $+0.8$ pp, $t(4)=0.69$, $p=0.53$,
 $d_z=0.31$). Because this model-level test does not rely on the independence
 assumption, it strengthens the central conclusion that retrying rather than
-reflecting accounts for the aggregate improvement. A full mixed-effects
+reflecting accounts for the aggregate improvement. 📄 s.10
-logistic model with random intercepts for scenario and model, fitted on the
+ 📄 s.10
-per-run records in the replication database, is left as a confirmatory step.
+To complete this analysis, we fitted the per-run records with (a) a Bayesian 📄 s.10
+mixed-effects logistic regression with random intercepts for model and 📄 s.10
+scenario and (b) a logistic generalized estimating equation (GEE) with an 📄 s.10
+exchangeable working correlation clustered on model--scenario cells. 📄 s.11
+Table~\ref{tab:mixed} summarizes the configuration contrasts. Both estimators 📄 s.11
+agree with the pooled and model-as-unit analyses: the retry effect is large 📄 s.11
+and significant, whereas the additional effect of reflection over a plain 📄 s.11
+second attempt is not. The estimated random-effect standard deviations for 📄 s.11
+model (1.59) and scenario (1.40) are both substantial, confirming that the 📄 s.11
+clustered analysis is warranted. Finally, a GEE interaction model restricted 📄 s.11
+to the retry-capable configurations estimates the reflection-by-multi-fault 📄 s.11
+interaction at $\beta=0.558$ (SE $0.281$, $p=0.047$): the complexity-specific 📄 s.11
+Reflexion benefit of Section~\ref{sec:results}-E remains statistically 📄 s.11
+significant under clustering, although it still rests on only two multi-fault 📄 s.11
+scenarios and therefore retains the hypothesis-generating caveat. 📄 s.11
+ 📄 s.10
+\begin{table}[t] 📄 s.10
+\centering 📄 s.11
+\caption{Mixed-effects and GEE configuration contrasts.} 📄 s.11
+\label{tab:mixed} 📄 s.11
+\scriptsize 📄 s.11
+\setlength{\tabcolsep}{3pt} 📄 s.11
+\begin{tabular}{l c c c c} 📄 s.11
+\toprule 📄 s.11
+\textbf{Contrast} & \textbf{Mixed OR [95\% CI]} & $p$ & \textbf{GEE OR [95\% CI]} & $p$ \\ 📄 s.11
+\midrule 📄 s.11
+Two-try vs.\ Baseline & 9.06 [6.19, 13.26] & $<0.001$ & 3.44 [2.33, 5.08] & $<0.001$ \\ 📄 s.11
+Reflexion vs.\ Baseline & 9.55 [6.51, 14.00] & $<0.001$ & 3.54 [2.27, 5.54] & $<0.001$ \\ 📄 s.11
+Reflexion vs.\ Two-try & 1.07 [0.75, 1.52] & 0.71 & 1.03 [0.82, 1.30] & 0.80 \\ 📄 s.11
+\bottomrule 📄 s.11
+\end{tabular} 📄 s.11
+\par\vspace{3pt}\begin{minipage}{\columnwidth}\footnotesize\raggedright 📄 s.11
+Mixed = Bayesian mixed-effects logistic regression on per-run outcomes (conditional odds ratios); GEE = logistic GEE clustered on model--scenario cells (marginal odds ratios); the two differ by construction but support identical conclusions. 📄 s.11
+\end{minipage} 📄 s.11
+\end{table} 📄 s.11
 \begin{table}[t]
 \centering
 \caption{Pooled configuration significance tests.}
 \label{tab:stats}
 \begin{tabular}{l r r r l}
 \toprule
 \textbf{Comparison} & $\chi^2$ & $p$ & Cohen's $h$ & \textbf{Sig.} \\
 \midrule
 Baseline vs. Two-try & 89.24 & $<0.001$ & 0.607 & Yes \\
 Baseline vs. Reflexion & 94.21 & $<0.001$ & 0.624 & Yes \\
 Two-try vs. Reflexion & 0.07 & 0.796 & 0.017 & No \\
 \bottomrule
 \end{tabular}
 \end{table}
 \begin{table}[!t]
 \caption{Model-as-unit paired tests.}
 \label{tab:model_unit}
 \centering
 \scriptsize
 \setlength{\tabcolsep}{2pt}
 \begin{tabular}{lccccc}
 \hline
 \textbf{Comparison} & \textbf{Mean} $\Delta$ \textbf{(pp)} & $t(4)$ & $p$ & \textbf{Cohen's} $d_z$ & \textbf{Sig.} \\
 \hline
 Baseline vs.\ Two-try      & $+29.8$ & $7.66$ & $0.002$ & $3.43$ & Yes \\
 Two-try vs.\ Full Reflexion & $+0.8$  & $0.69$ & $0.53$  & $0.31$ & No  \\
 Baseline vs.\ Full Reflexion & $+30.6$ & $6.26$ & $0.003$ & $2.80$ & Yes \\
 \hline
 \end{tabular}
 \end{table}
 \begin{table}[t]
 \centering
 \caption{Success-rate confidence intervals.}
 \label{tab:ci}
 \begin{tabular}{l r l}
 \toprule
 \textbf{Configuration} & \textbf{Success rate} & \textbf{95\% CI} \\
 \midrule
 Baseline & 38.6\% & [34.3\%, 42.9\%] \\
 Two-try & 68.4\% & [64.2\%, 72.3\%] \\
 Full Reflexion & 69.2\% & [65.0\%, 73.1\%] \\
 \bottomrule
 \end{tabular}
 \end{table}
-\subsection{Qualitative Analysis of Reflections}
+\subsection{Exploratory Qualitative Analysis of Reflections} 📄 s.11
-To examine the mechanism behind the quantitative results, we analyzed 462 reflection texts generated in the full Reflexion condition: 205 from runs that eventually succeeded and 257 from runs that remained unsuccessful. Table~\ref{tab:reflection_quality} summarizes reflection characteristics by model.
+To examine the mechanism behind the quantitative results, we conducted an exploratory automated content analysis of the 310 reflection texts generated in the full-Reflexion condition of the experimental matrix (one reflection per failed first attempt; no reflection follows a failed second attempt). Of these, 156 preceded a successful second attempt (``recovered'') and 154 did not. We report three surface metrics per model: mean character length (Len.), the share of reflections containing at least one Markdown heading as a proxy for structured output (Head.), and the share containing an explicit credit-assignment section (Credit, detected by the case-insensitive marker \emph{credit assignment}). Table~\ref{tab:reflection_quality} summarizes these characteristics, and Fig.~\ref{fig:refl_examples} contrasts a representative recovered reflection with a representative unrecovered one. Because these metrics are automated surface measures rather than a validated human-coded scheme, we treat this analysis as exploratory and hypothesis-generating rather than confirmatory. 📄 s.11
 \begin{figure*}[!t]
 \centering
 \footnotesize
 \renewcommand{\arraystretch}{1.3}
 \begin{tabular}{|p{0.46\textwidth}|p{0.46\textwidth}|}
 \hline
-\textbf{Successful reflection (action-oriented)} &
+\textbf{Recovered reflection (action-oriented)} & 📄 s.12
-\textbf{Failed reflection (vague acknowledgment)} \\
+\textbf{Unrecovered reflection (vague acknowledgment)} \\ 📄 s.12
 \hline
 % REPLACE with verbatim reflection from results.db (representative text below)
 ``I corrected the pod's \texttt{containerPort} to 8765, but the Service still
 targets the previous port, so traffic never reaches the container. The fix was
 incomplete: I repaired one side of the port mapping and did not verify
 end-to-end reachability. Next attempt: patch the Service \texttt{targetPort} to
 8765 and confirm the endpoint responds before stopping.'' &
 % REPLACE with verbatim reflection from results.db (representative text below)
 ``The pod is still not ready and the previous fix did not resolve the problem.
 Something in the configuration is incorrect and the service is missing an
 endpoint. I should re-examine the pod and try a different change next time.'' \\
 \hline
 \multicolumn{2}{|p{0.94\textwidth}|}{\emph{Representative reflections
-illustrating the patterns quantified in Table~\ref{tab:reflection_quality}: successful
+illustrating the patterns quantified in Table~\ref{tab:reflection_quality}: reflections 📄 s.12
-reflections perform explicit credit assignment and name concrete verification
+followed by a successful second attempt perform explicit credit assignment and name 📄 s.12
-steps (``verify''/``test'' terms appear markedly more often), whereas failed
+concrete verification steps (``verify''/``test'' terms appear markedly more often), 📄 s.12
-reflections restate the symptom without identifying the wrong action or the next
+whereas unrecovered reflections restate the symptom without identifying the wrong 📄 s.12
-repair. Verbatim logs are available in the replication database.}} \\
+action or the next repair. Verbatim logs are available in the replication database.}} \\ 📄 s.12
 \hline
 \end{tabular}
 \caption{Representative self-reflection contrast.}
 \label{fig:refl_examples}
 \end{figure*}
  The pattern supports the model-capability result: stronger and mid-tier models produced longer and more structured reflections, while lower-capability models more often produced vague failure acknowledgments that did not translate into effective follow-up actions.
 \begin{table}[t]
 \centering
-\caption{Reflection quality by model.}
+\caption{Reflection characteristics by model.} 📄 s.11
 \label{tab:reflection_quality}
 \setlength{\tabcolsep}{2pt}
 \begin{tabular}{l r r r r r}
 \toprule
-\textbf{Model} & \textbf{N} & \textbf{Succ.} & \textbf{Len.} & \textbf{Head.} & \textbf{Credit} \\
+\textbf{Model} & \textbf{N} & \textbf{Recov.} & \textbf{Len.} & \textbf{Head.} & \textbf{Credit} \\ 📄 s.11
 \midrule
-Haiku 3.0 & 97 & 8.2\% & 1951 & 0.0\% & 18.6\% \\
+Haiku 3.0 & 78 & 9.0\% & 1949 & 0.0\% & 20.5\% \\ 📄 s.11
-Haiku 3.5 & 117 & 24.8\% & 1866 & 12.8\% & 62.4\% \\
+Haiku 3.5 & 84 & 33.3\% & 1906 & 14.3\% & 52.4\% \\ 📄 s.11
-Haiku 4.5 & 98 & 58.2\% & 2254 & 100.0\% & 100.0\% \\
+Haiku 4.5 & 59 & 79.7\% & 2280 & 100.0\% & 100.0\% \\ 📄 s.11
-Sonnet 4.5 & 87 & 71.3\% & 2134 & 100.0\% & 81.6\% \\
+Sonnet 4.5 & 56 & 76.8\% & 2140 & 100.0\% & 82.1\% \\ 📄 s.11
-Opus 4.5 & 63 & 77.8\% & 2117 & 100.0\% & 100.0\% \\
+Opus 4.5 & 33 & 93.9\% & 2126 & 100.0\% & 100.0\% \\ 📄 s.11
 \bottomrule
 \end{tabular}
+\par\vspace{3pt}\begin{minipage}{\columnwidth}\footnotesize\raggedright 📄 s.11
+$N=310$ in-matrix reflections. Recov.\ = share of reflections followed by a successful second attempt; Len.\ = mean length in characters; Head.\ = share containing at least one Markdown heading; Credit = share containing an explicit credit-assignment section. 📄 s.11
+\end{minipage} 📄 s.11
 \end{table}
-The content analysis also showed that successful reflections were more action-oriented. Words associated with validation and concrete follow-up actions appeared more frequently in successful reflections: for example, uses of ``verify'' were 35.2\% higher, ``test'' 137.4\% higher, and ``wait'' 144.1\% higher than in failed reflections. In contrast, failed reflections more often contained general problem terms such as ``incorrect,'' ``failed,'' or ``missing'' without specifying what should be checked or changed next. Representative cases followed the same pattern. High-capability reflections identified specific technical mismatches, such as the need to coordinate pod ports and service \texttt{targetPort} values, and proposed explicit end-to-end validation steps. Low-capability reflections tended to acknowledge that the diagnosis was unclear but did not identify the wrong command, the overlooked evidence, or the concrete repair to try next.
+A keyword comparison, normalized to occurrences per 1{,}000 words within each group and tested with $\chi^2$ tests on the underlying counts, showed that recovered reflections were markedly more action-oriented. Terms associated with validation and concrete follow-up appeared significantly more often in recovered reflections: ``verify'' 64\% more frequently (10.0 vs.\ 6.1 per 1{,}000 words, $p<0.001$), ``test'' 98\% more ($9.8$ vs.\ $5.0$, $p<0.001$), ``wait'' 475\% more ($3.3$ vs.\ $0.6$, $p<0.001$), and ``confirm'' 80\% more ($4.9$ vs.\ $2.7$, $p<0.001$). Conversely, the diagnostic label ``incorrect'' was 64\% \emph{less} frequent in recovered reflections ($1.1$ vs.\ $3.1$, $p<0.001$): unrecovered reflections more often restated that something was incorrect, failed, or missing without specifying what should be checked or changed next. Representative cases followed the same pattern. High-capability reflections identified specific technical mismatches, such as the need to coordinate pod ports and service \texttt{targetPort} values, and proposed explicit end-to-end validation steps. Low-capability reflections tended to acknowledge that the diagnosis was unclear but did not identify the wrong command, the overlooked evidence, or the concrete repair to try next. 📄 s.11
+ 📄 s.11
+Categorizing the evaluator's failure reason for the 310 failed first attempts that triggered these reflections yields a coarse taxonomy of first-attempt failure modes: the pod was not running or not ready in 49.4\% of cases, the service was unreachable on its advertised port in 41.0\%, the agent had deleted a resource without recreating it in 4.8\%, and the service had no matching endpoints in 3.2\% (remaining 1.6\% other). The third category is a pure process failure: the agent's destructive repair step, not the original fault, caused the failing state that the subsequent reflection had to diagnose. 📄 s.11
 \subsection{Worked Example: One Reflection Loop}
 \label{sec:worked_example}
-To make the mechanism concrete, we trace a single full-Reflexion run on
+As an illustration, we trace a single full-Reflexion run on 📄 s.12
 Case~1 (wrong port) with Claude Haiku~4.5. In this incident the pod runs but is
 unreachable through its Service because of a port mismatch. The run reached
 success after two trials in $168.6$~s at a total cost of \$0.094.
 \textbf{Trial 1 (diagnosis and partial repair).}
 The Actor executed its diagnosis phase (\texttt{kubectl get}, \texttt{describe},
 \texttt{logs}, \texttt{get events}) and found that the pod declared one port
 while the application listened on another. It deleted and recreated the pod with
 the corrected \texttt{containerPort}. The deterministic Evaluator, however,
 returned failure: the endpoint was still unreachable, with reason
 \emph{``Service `wrong-port-service' not responding on port~80.''} The repair had
 addressed the pod side of the mapping but left the Service targeting the old
 port.
 \textbf{Reflection.}
 The Self-Reflection module analyzed the failed trajectory and the Evaluator
 reason and produced a first-person reflection that (i)~assigned credit to the
 incomplete fix, (ii)~identified the unrepaired Service \texttt{targetPort} as the
 root cause, and (iii)~prescribed a concrete next step:
 % REPLACE with verbatim reflection from results.db (representative text below)
 \begin{quote}
 \emph{``I fixed the pod's container port but did not update the Service, so
 traffic is routed to the wrong port. Next time I should patch the Service
 \texttt{targetPort} to match the container and confirm reachability before
 stopping.''}
 \end{quote}
 \textbf{Trial 2 (guided repair and success).}
 Conditioned on this reflection from episodic memory, the Actor targeted the
 Service directly, aligned its \texttt{targetPort} with the container, and stopped
 with \texttt{FIX COMPLETE}. The Evaluator then confirmed that pods were running,
-containers ready, and the endpoint reachable, marking the run successful. This
+containers ready, and the endpoint reachable, marking the run successful. This single run shows the paper's central finding on a small scale: the second attempt 📄 s.12
-example illustrates the paper's central finding in miniature: the second attempt
 provides most of the value, and reflection contributes when the first attempt
 leaves a partially repaired, still-inconsistent state that must be reasoned about
 rather than merely retried.
 \section{Discussion}
 \label{sec:discussion}
+This section discusses four themes: the value of retrying, the conditional nature of reflection, the moderating role of model capability, and the operational implications for AIOps deployments. It closes with the threats to validity. 📄 s.12
 \subsection{The Dominant Value of Retrying}
-The main aggregate result is that an additional attempt is highly valuable. The second attempt improved success by 29.8 percentage points, far exceeding the overall incremental gain from explicit reflection. This implies that LLM agents often gather useful evidence during the first attempt, and a further chance to act can correct many errors even without a separate reflection step.
+The main aggregate result is that an additional, externally verified attempt accounts for most of the improvement. The second attempt improved success by 29.8 percentage points, far exceeding the overall incremental gain from explicit reflection. This implies that LLM agents often gather useful evidence during the first attempt, and a further chance to act can correct many errors even without a separate reflection step. 📄 s.12
-For system designers, this finding suggests that agentic remediation systems should prioritize reliable retry and validation loops before adding more elaborate self-improvement mechanisms. A deterministic evaluator is particularly important because it stops the agent from prematurely claiming success and ensures that retries are driven by external evidence rather than the model's own judgment.
+For system designers, this finding suggests that agentic remediation systems should prioritize reliable retry and validation loops before adding more elaborate self-improvement mechanisms. A deterministic evaluator is particularly important because it stops the agent from prematurely claiming success and ensures that retries are driven by external evidence rather than the model's own judgment. The self-verification ablation (Section~\ref{sec:selfverify}) supports this point: letting the Actor verify and revise within one attempt recovered about half of the retry effect on single-fault scenarios, but only for the stronger models, largely by waiting for the cluster to converge, and at more than twice the baseline cost; a misleading local check also led Haiku 4.5 to accept a broken service in every Case~4 run. An external evaluator that probes the service from outside the pod remains the more reliable and cheaper source of the verification signal. 📄 s.12
-This interpretation is consistent with the broader agent literature, where tool-using LLM systems are usually most effective when reasoning is coupled with environmental feedback rather than treated as a single-pass generation problem \cite{yao2023react,wang2024surveyagents,qin2024toollearning}. In Kubernetes troubleshooting, the first attempt often functions as an exploratory interaction: commands reveal pod events, service endpoints, logs, and configuration mismatches that were not available in the initial prompt. The observed jump from baseline to two-try therefore suggests that verified interaction itself is a major source of performance improvement, aligning with ReAct-style findings that alternating between action and observation can improve task completion. At the same time, our results refine that literature by showing that an additional cycle of action is not equivalent to a successful self-reflection mechanism; the evaluator-driven retry produced most of the aggregate gain without requiring a separate reflective prompt.\par
+This interpretation is consistent with the broader agent literature, where tool-using LLM systems are usually most effective when reasoning is coupled with environmental feedback rather than treated as a single-pass generation problem \cite{yao2023react,wang2024surveyagents,qin2024toollearning}. In Kubernetes troubleshooting, the first attempt often functions as an exploratory interaction: commands reveal pod events, service endpoints, logs, and configuration mismatches that were not available in the initial prompt. The observed jump from baseline to two-try therefore suggests that verified interaction itself is a major source of performance improvement, aligning with ReAct-style findings that alternating between action and observation can improve task completion. Our results also qualify that literature: an additional action cycle is not the same as successful self-reflection, because the evaluator-driven retry produced most of the aggregate gain without a separate reflective prompt.\par 📄 s.12
 \subsection{Reflection is Conditional, Not Universal}
-Considered by itself, the aggregate Reflexion gain of 0.8 percentage points is not practically large. This average, however, masks a preliminary relationship with problem complexity. Reflection slightly reduced performance in single-fault scenarios but improved results in the two tested multi-fault scenarios. Because the double-fault and triple-fault categories each contain one scenario, this pattern should be read as hypothesis-generating rather than definitive. A likely explanation is that single-fault cases often require a direct correction, and extra reflection may introduce irrelevant context or distract the model. Multi-fault cases instead require the agent to revise its hypothesis after a partial repair. In those settings, explicit credit assignment and concrete next actions may help prevent the agent from repeating an incomplete strategy.
+On its own, the aggregate Reflexion gain of 0.8 percentage points is small in practical terms, but the average hides a preliminary dependence on problem complexity. Reflection slightly reduced performance in single-fault scenarios but improved results in the two tested multi-fault scenarios. Because the double-fault and triple-fault categories each contain one scenario, this pattern should be read as hypothesis-generating rather than definitive. A likely explanation is that single-fault cases often require a direct correction, and extra reflection may introduce irrelevant context or distract the model. Multi-fault cases instead require the agent to revise its hypothesis after a partial repair. In those settings, explicit credit assignment and concrete next actions may help prevent the agent from repeating an incomplete strategy. 📄 s.13
+ 📄 s.13
+The feedback-only control (Section~\ref{sec:feedback_control}) narrows this interpretation. On the multi-fault scenarios, supplying the evaluator's verbatim failure reason without any reflection outperformed full Reflexion on the three models tested. The multi-fault benefit previously attributed to reflection is therefore better described as a benefit of \emph{access to a concrete external failure signal}: the verbal reflection stage adds cost and latency, and, at least as implemented here without direct access to the evaluator's textual reason, it transmits that signal less faithfully than the raw feedback itself. The design recommendation changes accordingly: before adding a reflection module, a remediation system should first make sure that whatever failure evidence the evaluator produces reaches the next attempt verbatim. Reflection then becomes a candidate mechanism for the residual cases in which no informative external signal exists and the agent must reconstruct the failure from its own trajectory. 📄 s.13
 This result supports complexity-aware orchestration. A production-oriented AIOps system does not need to call reflection for every incident. Reflection can instead be activated after a failed first attempt, when several resources show inconsistent states, or when the evaluator observes that a partial fix changed the symptom without resolving it.
-The complexity-dependent pattern also agrees with, but qualifies, prior work on Reflexion and self-refinement. Reflexion reports that verbal feedback can help agents convert failed trajectories into improved subsequent behavior, and Self-Refine similarly shows that critique-and-revision can improve outputs without parameter updates \cite{shinn2023reflexion,madaan2023selfrefine}. Our Kubernetes results provide preliminary support for this principle in the tested multi-fault incidents, where the first attempt may repair only part of the causal chain and reflection can summarize what remains unresolved. However, the slight decline in single-fault cases indicates that reflection is not automatically beneficial: when the correct action is local and direct, additional natural-language analysis may increase cognitive load, introduce irrelevant hypotheses, or bias the model away from a simple fix. Thus, reflection should be understood as a targeted mechanism for credit assignment in stateful, multi-step failures rather than as a universal performance enhancer.\par
+The complexity-dependent pattern is consistent with prior work on Reflexion and self-refinement but also qualifies it. Reflexion reports that verbal feedback can help agents convert failed trajectories into improved subsequent behavior, and Self-Refine similarly shows that critique-and-revision can improve outputs without parameter updates \cite{shinn2023reflexion,madaan2023selfrefine}. Our Kubernetes results provide preliminary support for this principle in the tested multi-fault incidents, where the first attempt may repair only part of the causal chain and reflection can summarize what remains unresolved. However, the slight decline in single-fault cases indicates that reflection is not automatically beneficial: when the correct action is local and direct, additional natural-language analysis may increase cognitive load, introduce irrelevant hypotheses, or bias the model away from a simple fix. Reflection is therefore best understood as a targeted credit-assignment mechanism for stateful, multi-step failures, not as a universal performance enhancer.\par 📄 s.13
 \subsection{Model Capability Moderates Reflection}
-Within the five Claude models evaluated, the link between model capability and reflection effectiveness is non-linear. Because all models come from a single family, capability here is confounded with version-specific training and tool-use behavior, so the pattern should be read as within-family evidence rather than a capability law that generalizes across model families. Lower-capability models appear to generate or use reflections poorly, so the added context does not translate into better actions. Mid-capability models can reason over the feedback and still have meaningful room for improvement. Higher-capability models solve many incidents without reflection, creating a ceiling effect. This pattern indicates that reflection cannot replace sufficient model capability; it appears most useful when the model can understand the feedback but is not already strong enough to succeed with a simple retry.
+Within the five Claude models evaluated, the link between model capability and reflection effectiveness is non-linear. Because all models come from a single family, capability here is confounded with version-specific training and tool-use behavior, so the pattern is within-family evidence, not a general rule across model families. Lower-capability models appear to generate or use reflections poorly, so the added context does not translate into better actions. Mid-capability models can reason over the feedback and still have meaningful room for improvement. Higher-capability models solve many incidents without reflection, creating a ceiling effect. This pattern indicates that reflection cannot replace sufficient model capability; it appears most useful when the model can understand the feedback but is not already strong enough to succeed with a simple retry. 📄 s.13
-This non-linear effect is important for interpreting recent AIOps and LLM-for-operations studies. Prior AIOps research emphasizes that operational automation depends not only on model intelligence but also on telemetry quality, action constraints, and feedback loops \cite{zhang2024aiops,zhang2025aiopsllm}. Our findings are consistent with that view: reflection helped when the model had enough capability to map evaluator feedback to concrete Kubernetes actions, but it did not compensate for weak tool-use or incomplete domain reasoning. The result also suggests that benchmark reports should separate model capability, retry budget, and reflection design, because combining them into a single success number can overstate the independent contribution of reflective memory.\par
+This non-linear effect also bears on recent AIOps and LLM-for-operations studies. Prior AIOps research emphasizes that operational automation depends not only on model intelligence but also on telemetry quality, action constraints, and feedback loops \cite{zhang2024aiops,zhang2025aiopsllm}. Our findings are consistent with that view: reflection helped when the model had enough capability to map evaluator feedback to concrete Kubernetes actions, but it did not compensate for weak tool-use or incomplete domain reasoning. The result also suggests that benchmark reports should separate model capability, retry budget, and reflection design, because combining them into a single success number can overstate the independent contribution of reflective memory.\par 📄 s.13
 \subsection{Operational Implications}
 The findings favor a tiered remediation policy. For straightforward incidents, a low-latency two-try setup may be sufficient. For complex or unresolved incidents, the system can escalate to Reflexion with a mid-capability model. For high-severity incidents where success outweighs cost, a stronger model such as Opus may be appropriate, although reflection may add little at that level. In every case, remediation should remain verification-driven, with a programmatic evaluator deciding whether the agent continues, reflects, escalates, or stops.
+In an enterprise environment, the policy's inputs and outputs would connect to existing systems. The evaluator's failure signal can come from the observability and alerting platforms already in place instead of custom probes, and the repairs an agent proposes can be routed through GitOps- or CI/CD-style review pipelines so that cluster changes remain declarative, versioned, and approvable instead of being applied imperatively. Connecting the escalation step to incident-management tooling would let unresolved cases be handed to on-call engineers with the full diagnostic trajectory attached. 📄 s.13
+ 📄 s.13
 Algorithm~\ref{alg:tiered} summarizes this policy.
 \begin{algorithm}[!t]
-\caption{Tiered remediation policy}
+\caption{Proposed tiered remediation policy derived from the findings (deployment recommendation; not the experimental protocol of Section~\ref{sec:methods}).} 📄 s.14
 \label{alg:tiered}
 \begin{algorithmic}[1]
 \REQUIRE incident $I$, model tier ladder $M_1 \prec M_2 \prec \dots \prec M_k$
 \REQUIRE deterministic evaluator $E$, attempt budget $B$
 \STATE $m \gets M_1$ \COMMENT{start with a fast, low-cost model}
 \STATE $r_1 \gets \textsc{Actor}(I, m)$ \COMMENT{one-shot remediation attempt}
 \IF{$E(r_1) = \textsc{success}$}
     \RETURN \textsc{success} \COMMENT{most incidents resolve on a verified first/second try}
 \ENDIF
 \STATE $s \gets E.\textsc{signal}(r_1)$ \COMMENT{partial progress / multi-resource inconsistency?}
-\IF{$s$ indicates a partial or multi-fault state \AND attempts $< B$}
+\IF{attempts $< B$} 📄 s.14
-    \STATE $\rho \gets \textsc{Reflect}(r_1, s)$ \COMMENT{credit assignment + next steps}
+    \IF{$s$ indicates a partial or multi-fault state} 📄 s.14
-    \STATE $r_2 \gets \textsc{Actor}(I, m, \rho)$ \COMMENT{reflection-guided retry}
+        \STATE $\rho \gets \textsc{Reflect}(r_1, s)$ \COMMENT{credit assignment + next steps} 📄 s.14
-    \IF{$E(r_2) = \textsc{success}$} \RETURN \textsc{success} \ENDIF
+        \STATE $r_2 \gets \textsc{Actor}(I, m, \rho)$ \COMMENT{reflection-guided retry} 📄 s.14
-\ELSE
+    \ELSE 📄 s.14
-    \STATE $r_2 \gets \textsc{Actor}(I, m)$ \COMMENT{plain verified retry; no reflection}
+        \STATE $r_2 \gets \textsc{Actor}(I, m)$ \COMMENT{plain verified retry; no reflection} 📄 s.14
+    \ENDIF 📄 s.14
     \IF{$E(r_2) = \textsc{success}$} \RETURN \textsc{success} \ENDIF
 \ENDIF
 \IF{severity high \AND cost budget allows}
     \STATE escalate to a stronger tier $m \gets M_{j>1}$ and repeat
 \ELSE
     \STATE escalate to a human operator \COMMENT{dry-run, approve, or edit the repair}
 \ENDIF
 \end{algorithmic}
 \end{algorithm}
 \subsection{Threats to Validity}
 \label{sec:threats}
-Several validity threats remain. First, LLM behavior is non-deterministic. We addressed this by repeating each condition ten times, but more repetitions would narrow the confidence intervals. Second, the benchmark includes ten scenarios and therefore cannot represent the full range of production Kubernetes incidents. The complexity analysis is especially limited because only two scenarios are multi-fault cases; consequently, the observed gains for double- and triple-fault incidents should be treated as evidence of a promising pattern rather than as a definitive general result. Third, all runs used a single-node Minikube v1.32 environment on Docker Desktop; multi-node clusters could introduce scheduling, networking, storage, and permission problems not present here. Fourth, the evaluation covered only Claude models because the Actor was implemented with the Claude Agent SDK. Whether the findings generalize to GPT, Gemini, LLaMA, or other model families remains open. Fifth, success was measured as a binary outcome. An agent that found the root cause but failed to apply the repair was scored the same as one that made an unrelated change. Future work should use graded metrics for diagnostic accuracy, action safety, repeated mistakes, rollback behavior, and partial remediation progress.
+Several validity threats remain. First, LLM behavior is non-deterministic. We addressed this by repeating each condition ten times, but more repetitions would narrow the confidence intervals. Second, the benchmark includes ten scenarios and therefore cannot represent the full range of production Kubernetes incidents. The complexity analysis is especially limited because only two scenarios are multi-fault cases; consequently, the observed gains for double- and triple-fault incidents should be treated as evidence of a promising pattern rather than as a definitive general result. Third, all runs used a single-node Minikube v1.32 environment on Docker Desktop; multi-node clusters could introduce scheduling, networking, storage, and permission problems not present here. Fourth, the evaluation covered only Claude models because the Actor was implemented with the Claude Agent SDK. Whether the findings generalize to GPT, Gemini, LLaMA, or other model families remains open. Fifth, success was measured as a binary outcome. An agent that found the root cause but failed to apply the repair was scored the same as one that made an unrelated change. Future work should use graded metrics for diagnostic accuracy, action safety, repeated mistakes, rollback behavior, and partial remediation progress. Sixth, the Actor is prohibited from verifying its own fix because verification is delegated to the deterministic Evaluator. This design choice may inflate the measured value of the second attempt: the baseline effectively measures a single unverified attempt, whereas the retry conditions add both another attempt and an external verification signal, so part of the 29.8-percentage-point retry effect may reflect the value of verification itself rather than retrying alone. The post-hoc self-verification ablation (Section~\ref{sec:selfverify}) bounds this concern rather than removing it: on the single-fault scenarios and the three current models, permitting in-attempt verification recovered about half of the retry effect (+13.3 of +25.0 percentage points), most of it in one scenario where the baseline failure was a convergence-timing artifact, and the externally verified second attempt remained significantly better. The ablation uses five repetitions per cell, excludes the two retired models, and does not cover the multi-fault scenarios where the reflection benefit was concentrated; the retry effect should therefore be interpreted as the value of an additional, externally verified attempt, of which self-verification alone captures only part. Seventh, the fault patterns in the eight single-fault scenarios are pervasive in tutorials, community forums, and public benchmarks, so descriptions of these misconfigurations and their fixes plausibly appear in the models' training data. Absolute success rates should therefore be read as an upper bound on performance for genuinely novel incidents rather than as an estimate of general troubleshooting ability. The comparisons between configurations, however, remain internally valid: the baseline, retry, and Reflexion conditions used identical scenarios, models, and prompts, so any familiarity with a fault pattern is shared equally by all conditions and cannot by itself produce the observed between-condition differences. In addition, the two multi-fault scenarios, where the reflection benefit is concentrated, were composed for this study by combining faults across resources, making verbatim memorization of their solutions unlikely, although the constituent single-fault patterns remain individually well known. Eighth, the feedback-only control was added post hoc and could be run only on the three models still available through the API, on the multi-fault scenarios; its comparison against the other arms uses only matched runs executed on the identical current scenario instantiation, which differs from the instantiation of the original five-model matrix in its diagnostic entry point (Section~\ref{sec:feedback_control}), and the control receives the evaluator's textual reason, which the implemented reflection pipeline does not. Its results should therefore be read as isolating the value of the evaluator's failure signal in the tested cells, not as a full factorial re-run of the study. 📄 s.13
 \section{Conclusion and Future Work}
 \label{sec:conclusion}
-This study assessed Reflexion for autonomous Kubernetes incident remediation using 1,500 controlled experiments. The results show that allowing a second attempt substantially improves LLM-agent success, whereas explicit self-reflection adds only a small aggregate benefit. Its value is most visible in specific settings: the two tested multi-fault incidents and mid-capability models. Lower-capability models do not exploit reflection well, while higher-capability models often succeed without it. The cost and latency analyses suggest that self-reflection should be used selectively, together with deterministic evaluation and dynamic model selection. More broadly, self-reflective LLM agents can support autonomous operations, but their design should account for incident complexity, model capability, and verification requirements. This conclusion concerns single-round reflection under a two-attempt budget; whether reflection's value grows under larger budgets, where episodic memory can accumulate across trials, remains an open question.
+This study assessed Reflexion for autonomous Kubernetes incident remediation using 1,500 controlled experiments. The results show that allowing a second attempt substantially improves LLM-agent success, whereas explicit self-reflection adds only a small aggregate benefit. Its value is most visible in specific settings: the two tested multi-fault incidents and mid-capability models. Because the multi-fault evidence rests on only two scenarios, this complexity-specific benefit remains preliminary and requires confirmation on a broader multi-fault benchmark. The post-hoc feedback-only control further shows that, on identically instantiated multi-fault scenarios, forwarding the evaluator's verbatim failure reason to the second attempt is clearly more effective than verbal self-reflection, so most of the observed benefit is due to the failure signal itself. Lower-capability models do not exploit reflection well, while higher-capability models often succeed without it. The cost and latency analyses suggest that self-reflection should be used selectively, together with deterministic evaluation and dynamic model selection. More broadly, self-reflective LLM agents can support autonomous operations, but their design should account for incident complexity, model capability, and verification requirements. This conclusion concerns single-round reflection under a two-attempt budget; whether reflection's value grows under larger budgets, where episodic memory can accumulate across trials, remains an open question. 📄 s.14
-In direct response to the research questions, \textbf{RQ1} shows that Reflexion adds only a small aggregate gain beyond a verified retry (69.2\% versus 68.4\%). \textbf{RQ2} shows a preliminary signal of benefit in the two multi-fault scenarios, but this result requires more double- and triple-fault cases. \textbf{RQ3} shows that, within the Claude family evaluated, mid-capability models benefit most from reflection, while weak models underuse it and the strongest model reaches a ceiling; confirming this across other model families remains future work. \textbf{RQ4} indicates that Haiku 4.5 provides the best observed success--cost trade-off among the retry-capable settings, whereas Opus 4.5 provides the highest absolute success when cost is less important.
+In direct response to the research questions, \textbf{RQ1} shows that single-round Reflexion adds only a small aggregate gain beyond a verified retry (69.2\% versus 68.4\%); the self-verification ablation further indicates that roughly half of the retry effect on single-fault scenarios reflects verification rather than the extra attempt itself, with the externally verified retry still superior. \textbf{RQ2} shows a preliminary signal of benefit in the two multi-fault scenarios, but this result requires more double- and triple-fault cases; the feedback-only control further indicates that this benefit is mediated by access to the evaluator's failure signal rather than by the reflection step itself. \textbf{RQ3} shows that, within the Claude family evaluated, mid-capability models benefit most from reflection, while weak models underuse it and the strongest model reaches a ceiling; confirming this across other model families remains future work. \textbf{RQ4} indicates that Haiku 4.5 provides the best observed success--cost trade-off among the retry-capable settings, whereas Opus 4.5 provides the highest absolute success when cost is less important. 📄 s.15
-These findings lead to two main conclusions. First, Reflexion should not be interpreted as a general replacement for careful agent design, reliable tool access, or external validation. In line with prior work on ReAct, Reflexion, and tool-using LLM agents, the strongest improvements arise when the model is embedded in a loop that exposes observations, checks outcomes, and constrains the next action \cite{yao2023react,shinn2023reflexion,wang2024surveyagents}. Second, the results show that self-reflection has practical value when it is matched to the structure of the task. Kubernetes incidents with several interacting faults create a need for causal revision and partial-progress tracking, and this is where reflective memory most clearly helps. For simpler incidents, however, a verified retry can provide most of the benefit with lower cost and latency.\par
+Two conclusions follow. First, Reflexion should not be interpreted as a general replacement for careful agent design, reliable tool access, or external validation. In line with prior work on ReAct, Reflexion, and tool-using LLM agents, the strongest improvements arise when the model is embedded in a loop that exposes observations, checks outcomes, and constrains the next action \cite{yao2023react,shinn2023reflexion,wang2024surveyagents}. Second, the results show that self-reflection has practical value when it is matched to the structure of the task. Kubernetes incidents with several interacting faults create a need for causal revision and partial-progress tracking, and this is where an informative external failure signal most clearly helps; the feedback-only control shows that delivering that signal verbatim to the next attempt can be more effective than reflecting on it. For simpler incidents, a verified retry alone can provide most of the benefit with lower cost and latency, and self-reflection remains a candidate mechanism mainly for settings in which no informative external signal is available.\par 📄 s.15
-From an AIOps perspective, the study therefore supports a selective and evidence-driven deployment model. Rather than enabling reflection for every alert, an operational system can first run a fast remediation attempt, evaluate the cluster state programmatically, and then decide whether to retry, reflect, escalate to a stronger model, or involve a human operator. This conclusion is consistent with AIOps literature that emphasizes closed-loop automation, telemetry quality, and safety-aware decision making over unconstrained model autonomy \cite{zhang2024aiops,zhang2025aiopsllm}. The practical implication is that future production systems should treat reflection as one component in a broader remediation pipeline, not as the sole source of reliability.\par
+For AIOps practice, the study supports selective, evidence-driven deployment. Rather than enabling reflection for every alert, an operational system can first run a fast remediation attempt, evaluate the cluster state programmatically, and then decide whether to retry, reflect, escalate to a stronger model, or involve a human operator. This conclusion is consistent with AIOps literature that emphasizes closed-loop automation, telemetry quality, and safety-aware decision making over unconstrained model autonomy \cite{zhang2024aiops,zhang2025aiopsllm}. The practical implication is that future production systems should treat reflection as one component in a broader remediation pipeline, not as the sole source of reliability.\par 📄 s.15
-Future studies should examine additional LLM families, larger and more varied Kubernetes benchmarks, multi-node production-like clusters, and remediation processes with more than two cycles.\allowbreak Human-in-the-loop remediation is also important, especially for high-risk operations where an agent proposes a repair and a human operator approves or edits it before execution. Safety constraints require further study as well: autonomous remediation agents can delete pods, patch services, and modify deployments, so production use should be controlled through permissions, dry-run execution, rollback plans, audit logs, and policy checks.
+Future studies should examine additional LLM families, larger and more varied Kubernetes benchmarks, multi-node production-like clusters, and remediation processes with more than two cycles.\allowbreak Human-in-the-loop remediation is also important, especially for high-risk operations where an agent proposes a repair and a human operator approves or edits it before execution. Safety constraints require further study as well: autonomous remediation agents can delete pods, patch services, and modify deployments, so production use should be controlled through permissions, dry-run execution, rollback plans, audit logs, and policy checks; recent work on trustworthy LLM agents catalogues these threats and countermeasures systematically \cite{yu2025trustworthy}. The architecture used here already records every command and evaluator verdict per run, which provides a natural audit trail; production readiness would further require explicit approval workflows for destructive actions and automated policy-compliance checks integrated with the organization's change-management process. 📄 s.15
-Future work should also move beyond binary success and failure labels. More detailed metrics could distinguish correct diagnosis, safe but incomplete repair, unsafe action, unnecessary change, rollback success, and time-to-stable-state. Such metrics would make it possible to evaluate whether reflection improves the agent's understanding of the root cause even when the final repair is incomplete. Another important direction is to analyze the content of generated reflections themselves: reflections may be useful when they identify overlooked evidence and concrete next actions, but harmful when they restate symptoms, invent unsupported causes, or overfit to misleading intermediate observations.\par
+Future work should also move beyond binary success and failure labels. More detailed metrics could distinguish correct diagnosis, safe but incomplete repair, unsafe action, unnecessary change, rollback success, and time-to-stable-state. Such metrics would make it possible to evaluate whether reflection improves the agent's understanding of the root cause even when the final repair is incomplete. Because the public replication dataset stores the full command trajectory, evaluator reason, and token accounting for every run, metrics such as root-cause localization accuracy, repair precision, and action efficiency can be computed retrospectively from the released artifacts without rerunning the experiments. Another important direction is to analyze the content of generated reflections themselves: reflections may be useful when they identify overlooked evidence and concrete next actions, but harmful when they restate symptoms, invent unsupported causes, or overfit to misleading intermediate observations. Because each scenario was repeated ten times, the released corpus of 310 reflections also enables a retrospective study of the semantic consistency, diversity, and quality of reflections generated across repeated runs of identical incidents.\par 📄 s.15
-Finally, future research should investigate adaptive orchestration policies. A promising direction is to trigger reflection only when evaluator signals indicate partial progress, repeated failure, or multi-resource inconsistency, and to combine this trigger with dynamic model selection based on incident severity and expected cost. Larger benchmarks should include network policies, persistent volumes, RBAC errors, node failures, autoscaling behavior, and multi-service dependencies so that results better approximate production Kubernetes environments. Evaluating these settings across multiple LLM providers would clarify whether the conditional value of reflection observed here is a general property of agentic remediation or a model-family-specific effect.\par
+Finally, future research should investigate adaptive orchestration policies. A promising direction is to trigger reflection only when evaluator signals indicate partial progress, repeated failure, or multi-resource inconsistency, and to combine this trigger with dynamic model selection based on incident severity and expected cost. The tiered policy in Algorithm~\ref{alg:tiered} is a first step in this direction; confidence-aware or uncertainty-driven triggers that estimate when reflection is worth its added cost and latency would be a natural refinement. Larger benchmarks should include network policies, persistent volumes, RBAC errors, node failures, autoscaling behavior, and multi-service dependencies, as well as Helm-based deployments, StatefulSets, admission controllers, and heterogeneous multi-node workloads, so that results better approximate production Kubernetes environments. Evaluating these settings across multiple LLM providers would clarify whether the conditional value of reflection observed here is a general property of agentic remediation or a model-family-specific effect.\par 📄 s.15
+ 📄 s.16
+\section*{Acknowledgment} 📄 s.16
+The authors thank the anonymous reviewers for their constructive comments, which substantially improved the manuscript. 📄 s.16
 \section*{Data and Code Availability}
-The experimental outputs were saved as JSON files and exported into an SQLite database. Data, scripts, prompts, scenario definitions, and evaluator logic are being prepared for a public replication repository; until anonymized review artifacts are released, they are available from the corresponding author upon reasonable request.
+The experimental outputs were saved as per-run JSON files and exported into an SQLite database. The complete replication package (raw per-run JSON records, the SQLite results database, all agent and reflection prompts, the scenario definitions with automated setup and cleanup scripts, and the programmatic evaluator logic) is publicly available at \url{https://github.com/mmerts/thesis-sdk} (accessed August 13, 2026). The exact API model identifiers used in all experiments were \texttt{claude-3-haiku-20240307}, \texttt{claude-3-5-haiku-20241022}, \texttt{claude-haiku-4-5-20251001}, \texttt{claude-sonnet-4-5-20250929}, and \texttt{claude-opus-4-5-20251101} for Claude 3 Haiku, Claude 3.5 Haiku, Claude Haiku 4.5, Claude Sonnet 4.5, and Claude Opus 4.5, respectively. 📄 s.16
 \bibliographystyle{IEEEtran}
 \bibliography{references}
 \begin{comment}
 \begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{author1.png}}]{First A. Author} received the B.S. and M.S. degrees in aerospace engineering from
 the University of Virginia, Charlottesville, in 2001 and the Ph.D. degree in
 mechanical engineering from Drexel University, Philadelphia, PA, in 2008.
 From 2001 to 2004, he was a Research Assistant with the Princeton Plasma
 Physics Laboratory. Since 2009, he has been an Assistant Professor with the
 Mechanical Engineering Department, Texas A{\&}M University, College Station.
 He is the author of three books, more than 150 articles, and more than 70
 inventions. His research interests include high-pressure and high-density
 nonthermal plasma discharge processes and applications, microscale plasma
 discharges, discharges in liquids, spectroscopic diagnostics, plasma
 propulsion, and innovation plasma applications. He is an Associate Editor of
 the journal \emph{Earth, Moon, Planets}, and holds two patents.
 Dr. Author was a recipient of the International Association of Geomagnetism
 and Aeronomy Young Scientist Award for Excellence in 2008, and the IEEE
 Electromagnetic Compatibility Society Best Symposium Paper Award in 2011.
 \end{IEEEbiography}
 \end{comment}
 \begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{Mert.png}}]{Mustafa Mert Suerkan} is a graduate student at TED University, Ankara, Turkiye. His research interests include large language model agents, autonomous cloud operations, Kubernetes troubleshooting, AIOps, and software engineering automation.
 \end{IEEEbiography}
 \begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{Emin.png}}]{Emin Kugu} (Member, IEEE) received the B.S. degree from Istanbul University Computer Science Engineering Department, his master's degree from Air Force Academy Software Engineering and his doctoral degree from Old Dominion University Electrical and Computer Engineering program. He worked as a part-time lecturer at different universities. He worked as a project manager in the Air Force Command Information System (HvBS) project and took part in the software development processes of international defense projects. His research interests include cyber security, computer networks, artificial intelligence and machine/deep learning.
 \end{IEEEbiography}
 \begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{hakan-emekci.png}}]{Hakan Emekci} (Member, IEEE) received the B.Sc. degree in computer engineering from Middle East Technical University (METU), Ankara, Türkiye, in 2011, the M.Sc. degree from London Business School, and the Ph.D. degree from Hacettepe University, Ankara, Türkiye.
 He is currently a Faculty Member with Hacettepe University, specializing in artificial intelligence. He is also the Founder of NavigaAI, a company focused on AI-driven solutions. He is actively involved in educational technology development. He has led impactful projects in collaboration with the European Bank for Reconstruction and Development (EBRD) and the United Nations Development Program (UNDP), where his work has supported AI-driven solutions for social and economic development initiatives. Beyond his academic and professional achievements, he has been dedicated to bridging advanced AI methodologies with practical applications that address real-world challenges, advancing both the technology and its accessibility. He continues to mentor the next generation of data scientists, focusing on ethical AI practices and innovative problem-solving approaches.
 \end{IEEEbiography}
 % IEEE Access often includes author biographies. Replace or remove before submission.
 % \begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{author1.png}}]{Author One}
 % Biography text for Author One.
 % \end{IEEEbiography}
 %
 % \begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{author2.png}}]{Author Two}
 % Biography text for Author Two.
 % \end{IEEEbiography}
 %
 % \begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{author3.png}}]{Author Three}
 % Biography text for Author Three.
 % \end{IEEEbiography}
 \EOD
 \end{document}