CLAUDE.md

Quick Reference — Do These First or Waste Hours

`deploy/crds/` Is NOT for Operator Version Bumps

deploy/crds/ contains hub-side CRDs (GitOpsCluster, ManagedServiceAccount, etc.) generated by make manifests. During operator manifest updates, CRDs go in gitopsaddon/charts/openshift-gitops-operator/templates/crds/ — that's the embedded chart. Verify before touching deploy/crds/ — it's almost never needed for operator bumps. Only modify it if make manifests or a hub-side CRD change requires it.

Pull Secret Chain (Non-OCP clusters WILL fail without this)

The multiclusterhub-operator-pull-secret on the hub MUST include registry.redhat.io in BOTH multicluster-engine and open-cluster-management namespaces. Without it, ALL non-OCP managed clusters (Kind, EKS) get ImagePullBackOff on every Red Hat image. OCP hubs mask this because nodes have registry.redhat.io at the node level.

# Check
kubectl get secret multiclusterhub-operator-pull-secret -n multicluster-engine -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | python3 -c "import json,sys; print(list(json.load(sys.stdin)['auths'].keys()))"

# Fix: merge OCP global pull-secret into both copies
kubectl get secret pull-secret -n openshift-config -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d > /tmp/ocp-pull.json
# merge auths from /tmp/ocp-pull.json into multiclusterhub-operator-pull-secret in BOTH namespaces
# then delete {cluster}-import secrets to force regeneration

Agent Mode Setup Order (skip a step = broken)

Fix pull secret chain (above) — do this BEFORE anything else for non-OCP clusters
Install OpenShift GitOps operator on hub (OLM, openshift-gitops-operator ns, ARGOCD_CLUSTER_CONFIG_NAMESPACES=openshift-gitops, DISABLE_DEFAULT_ARGOCD_INSTANCE=false)
Wait for ArgoCD server pod Running in openshift-gitops
Patch ArgoCD CR: principal.enabled: true, destinationBasedMapping: true, controller.enabled: true, sourceNamespaces: ["*"] — hybrid mode: the app-controller stays enabled and coexists with the principal in the same namespace/instance; it never conflicts because agent-routed cluster secrets carry skip-reconcile: "true" (see "Hybrid Mode" below)
Create ManagedClusterSetBinding for global in openshift-gitops
Patch default AppProject: destinations: [{name:"*", namespace:"*", server:"*"}] — name:"*" is critical, without it principal won't propagate AppProjects
Apply addon templates: kubectl apply -f gitopsaddon/addonTemplates/addonTemplates.yaml
Create Placement (excluding local-cluster) + GitOpsCluster with argoCDAgent.enabled: true, mode: managed — local-cluster must never be in this Placement; it's not an addon-install target (it already has its own ArgoCD instance)
Wait for GitOpsCluster Ready=True and Policy Compliant
Patch Policy with cluster-admin RBAC for app-controller BEFORE creating AppSet — if apps sync before RBAC exists, ArgoCD exhausts 5 retries and gives up permanently
Grant the hub's own app-controller ServiceAccount (openshift-gitops-argocd-application-controller) cluster-admin too — needed for any Application targeting local-cluster/in-cluster to create resources in its destination namespace; the operator's default ClusterRole is read-only outside openshift-gitops itself. Do this before creating the AppSet for the same reason as step 10 (5-retry exhaustion).
If ApplicationSets should also target local-cluster (hybrid mode), create a separate Placement that includes local-cluster and point the AppSet's clusterDecisionResource generator at it — never reuse the addon-install Placement from step 8 for this
Create ApplicationSet(s)
Verify apps Synced on hub (including any local-cluster-targeted app, reconciled directly by the hub's own app-controller) and guestbook pods on spokes

Kind Cluster Import Order

The import secret has TWO files. Apply CRDs first or the Klusterlet CR fails:

kubectl -n <cluster> get secret <cluster>-import -o jsonpath='{.data.crds\.yaml}' | base64 -d | kubectl apply -f -
kubectl -n <cluster> get secret <cluster>-import -o jsonpath='{.data.import\.yaml}' | base64 -d | kubectl apply -f -

Then install governance addons (governance-policy-framework + config-policy-controller ManagedClusterAddOns) or Policies will never be enforced.

Expected Waits (Not Bugs)

Principal CrashLoopBackOff: Normal when GitOpsCluster is created after ArgoCD CR. Secrets (argocd-agent-ca, argocd-agent-principal-tls, argocd-agent-resource-proxy-tls, argocd-agent-jwt) are generated by the hub controller on GitOpsCluster reconciliation. The JWT secret is required by the v1.21.0+ principal for token signing. Principal recovers automatically — can take 5+ minutes due to exponential backoff.
Spoke agent CrashLoopBackOff with [FATAL]: Error creating remote: could not read secret: secrets "argocd-agent-ca" not found: Looks like a config error but is the same class of startup race as the principal one above, just the next hop over. The hub's argocd-agent-ca secret has to reach the spoke's openshift-gitops namespace via a ManifestWork (argocd-agent-ca-mw, created by PropagateHubCA in propagate_hub_ca.go) delivered by the klusterlet work-agent — this is a second, independent propagation delay on top of (not instead of) the principal's own cert-generation delay in the line above. Confirmed benign: the agent pod simply restarts until the secret lands, then starts normally with no other action needed. Before assuming something is actually broken, check: (1) hub has the source secret (oc get secret argocd-agent-ca -n openshift-gitops on the hub), (2) the ManifestWork exists and shows Applied=True/Available=True (oc get manifestwork argocd-agent-ca-mw -n <managed-cluster-name> -o yaml on the hub), (3) the secret eventually appears on the spoke (oc get secret argocd-agent-ca -n openshift-gitops on the managed cluster). If all three are healthy, just wait out the agent's crash-loop backoff — observed taking several minutes end-to-end, longer than the principal's own cert-generation window since it's gated behind it (the hub can't propagate a CA secret that doesn't exist yet) plus the klusterlet work-agent's own sync interval.
Policy NonCompliant on Kind: The addon agent must install the embedded operator first (creates CRDs + openshift-gitops namespace). Takes 2-5 minutes. The Policy can't create the ArgoCD CR until the CRD exists.
OCP OLM timing: Catalog resolution + InstallPlan + CSV = 30-120s even when healthy. Use 600s timeouts.

Repository Overview

Every binary below is built from the same image (quay.io/stolostron/multicloud-integrations), but shipped as three separate Deployments on the hub — don't assume "the controller" means only one of them:

gitopscluster — Hub-side controller in the multicluster-operators-application deployment. Reconciles GitOpsCluster CRs, creates Policies, MCAs, ADCs, AddOnTemplates, cluster secrets, TLS certs, and handles agent version drift healing. This is the argocd-agent (principal/agent) integration — see "Agent Modes" and "Hybrid Mode" below.
gitopsaddon — Addon agent on each managed cluster. Installs OpenShift GitOps operator (OLM on OCP, embedded Helm chart on non-OCP), creates ArgoCD CR, manages image pull secrets, patches ServiceAccounts, and optionally deploys the argocd-agent. Deployed by the OCM addon framework — runs regardless of which integration model (agent or basic pull model) is in use, since both need an ArgoCD instance actually installed on the spoke.
gitopssyncresc / multiclusterstatusaggregation / propagation — the older, still-shipping basic pull model integration (in the multicluster-integrations deployment on the hub, a third, separate Deployment from the two above). Predates argocd-agent, has no dedicated spoke agent process, and reuses the standard OCM klusterlet work-agent (ManifestWork) that every managed cluster already has. See "Basic Pull Model" under Architecture, and "Basic Pull Model Setup — Full YAML" below, for the full picture. maestropropagation/maestroaggregation are an opt-in, Maestro-backed alternative to this same model for very large scale (not deployed by default).

Common Commands

make build                                    # Build all binaries
make build-images                             # Build container images
make test                                     # Unit tests (self-provisions envtest/kubebuilder assets via setup-envtest)
make lint                                     # Linting
make generate                                 # DeepCopy methods after API changes
make manifests                                # CRD manifests after RBAC/CRD annotation changes

# E2E (creates Kind clusters, builds image, runs tests)
make test-local-e2e-gitopsaddon-embedded
make test-local-e2e-gitopsaddon-embedded-agent
make test-local-e2e-gitopsaddon-olm-override
make test-local-e2e-gitopsaddon-embedded-autonomous
make clean-e2e                                # Delete Kind clusters

Agent Mode Setup — Full YAML

# 1. Configure hub ArgoCD as principal (hybrid mode: controller stays enabled)
kubectl patch argocd openshift-gitops -n openshift-gitops --type=merge -p '{
  "spec": {
    "controller": {"enabled": true},
    "argoCDAgent": {
      "principal": {
        "enabled": true,
        "auth": "mtls:CN=system:open-cluster-management:cluster:([^:]+):addon:gitops-addon:agent:gitops-addon-agent",
        "destinationBasedMapping": true,
        "namespace": {"allowedNamespaces": ["*"]}
      }
    },
    "sourceNamespaces": ["*"]
  }
}'

# 2. Apply addon templates
kubectl apply -f gitopsaddon/addonTemplates/addonTemplates.yaml

# 3. Create ManagedClusterSetBinding
kubectl apply -f - <<'EOF'
apiVersion: cluster.open-cluster-management.io/v1beta2
kind: ManagedClusterSetBinding
metadata:
  name: global
  namespace: openshift-gitops
spec:
  clusterSet: global
EOF

# 4. Patch default AppProject (name:"*" is critical)
kubectl patch appproject default -n openshift-gitops --type=merge -p '{
  "spec": {
    "destinations": [{"name":"*","namespace":"*","server":"*"}],
    "clusterResourceWhitelist": [{"group":"*","kind":"*"}],
    "sourceNamespaces": ["*"],
    "sourceRepos": ["*"]
  }
}'

# 5. Create Placement + GitOpsCluster
kubectl apply -f - <<'EOF'
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: agent-placement
  namespace: openshift-gitops
spec:
  clusterSets: [global]
  predicates:
  - requiredClusterSelector:
      labelSelector:
        matchExpressions:
        - key: name
          operator: NotIn
          values: [local-cluster]
---
apiVersion: apps.open-cluster-management.io/v1beta1
kind: GitOpsCluster
metadata:
  name: mc-gitops-agent
  namespace: openshift-gitops
spec:
  argoServer:
    cluster: local-cluster
    argoNamespace: openshift-gitops
  placementRef:
    kind: Placement
    apiVersion: cluster.open-cluster-management.io/v1beta1
    name: agent-placement
  gitopsAddon:
    enabled: true
    argoCDAgent:
      enabled: true
      mode: managed
EOF

# 6. Wait for Ready + Compliant
kubectl get gitopscluster mc-gitops-agent -n openshift-gitops -w
kubectl get policy mc-gitops-agent-argocd-policy -n openshift-gitops -w

# 7. RBAC Policy patch — DO THIS BEFORE creating AppSet
kubectl patch policy mc-gitops-agent-argocd-policy -n openshift-gitops --type=json -p='[
  {"op":"add","path":"/spec/policy-templates/-","value":{
    "objectDefinition":{
      "apiVersion":"policy.open-cluster-management.io/v1",
      "kind":"ConfigurationPolicy",
      "metadata":{"name":"mc-gitops-agent-argocd-policy-rbac"},
      "spec":{
        "remediationAction":"enforce","severity":"medium",
        "object-templates":[{
          "complianceType":"musthave",
          "objectDefinition":{
            "apiVersion":"rbac.authorization.k8s.io/v1",
            "kind":"ClusterRoleBinding",
            "metadata":{"name":"acm-openshift-gitops-cluster-admin"},
            "roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"cluster-admin"},
            "subjects":[{"kind":"ServiceAccount","name":"acm-openshift-gitops-argocd-application-controller","namespace":"openshift-gitops"}]
          }
        }]
      }
    }
  }}
]'

# 8. Grant the hub's own app-controller RBAC too (needed for local-cluster/in-cluster targets) —
# DO THIS BEFORE creating any AppSet that targets local-cluster, same 5-retry-exhaustion reason.
kubectl create clusterrolebinding openshift-gitops-appcontroller-cluster-admin \
  --clusterrole=cluster-admin \
  --serviceaccount=openshift-gitops:openshift-gitops-argocd-application-controller

# 9. (Hybrid mode only) Create a SEPARATE Placement that includes local-cluster, used only by
# ApplicationSets that need to target it. Never add local-cluster to "agent-placement" above.
kubectl apply -f - <<'EOF'
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: agent-placement-appset
  namespace: openshift-gitops
spec:
  clusterSets: [global]
  predicates:
  - requiredClusterSelector:
      labelSelector:
        matchExpressions:
        - key: name
          operator: In
          values: [local-cluster]   # plus every managed cluster name you want the AppSet to target
EOF

# 10. Create ApplicationSet (AFTER RBAC is in place). Point at "agent-placement-appset" if it
# should also generate an Application for local-cluster; point at "agent-placement" if it should
# stay agent-spokes-only.
kubectl apply -f - <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: guestbook-appset
  namespace: openshift-gitops
spec:
  generators:
  - clusterDecisionResource:
      configMapRef: acm-placement
      labelSelector:
        matchLabels:
          cluster.open-cluster-management.io/placement: agent-placement-appset
      requeueAfterSeconds: 30
  template:
    metadata:
      name: '{{name}}-guestbook'
      namespace: openshift-gitops
    spec:
      project: default
      source:
        repoURL: https://github.com/argoproj/argocd-example-apps
        targetRevision: HEAD
        path: guestbook
      destination:
        name: '{{name}}'
        namespace: guestbook
      syncPolicy:
        automated: {prune: true, selfHeal: true}
        syncOptions: [CreateNamespace=true]
EOF

Basic Pull Model Setup — Full YAML

The basic pull model needs no agent, no principal, no certs — just a GitOpsCluster with gitopsAddon.enabled: true and no argoCDAgent block at all, so the addon installs a plain (non-agent) ArgoCD instance on the spoke, plus an ApplicationSet (or a single Application) carrying the pull label/annotation. OCP-only: this integration is only supported for OCP managed clusters (OLM-installed operator) — it is not validated or supported on non-OCP (embedded Helm chart) spokes.

# 1. Placement selecting only the target (OCP) managed cluster(s) — reuse the same
#    ManagedClusterSetBinding/AppProject setup as agent mode (see "Agent Mode Setup" above).
kubectl apply -f - <<'EOF'
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: pull-model-placement
  namespace: openshift-gitops
spec:
  clusterSets: [global]
  predicates:
  - requiredClusterSelector:
      labelSelector:
        matchExpressions:
        - key: name
          operator: In
          values: [<ocp-managed-cluster-name>]
EOF

# 2. GitOpsCluster with gitopsAddon enabled and argoCDAgent OMITTED (not disabled — omitted).
#    This forces createBlankClusterSecrets=true internally and installs a plain, non-agent
#    ArgoCD via the static "gitops-addon" AddOnTemplate + OLM subscription.
kubectl apply -f - <<'EOF'
apiVersion: apps.open-cluster-management.io/v1beta1
kind: GitOpsCluster
metadata:
  name: pull-model-gitops
  namespace: openshift-gitops
spec:
  argoServer:
    cluster: local-cluster
    argoNamespace: openshift-gitops
  placementRef:
    kind: Placement
    apiVersion: cluster.open-cluster-management.io/v1beta1
    name: pull-model-placement
  gitopsAddon:
    enabled: true
EOF

# 3. Wait for the spoke's ArgoCD (application-controller, redis, repo-server — "server" is
#    disabled by default for both agent and pull-model spokes, this is expected, see Architecture).
#    Then create an ApplicationSet whose template carries the pull label/annotation:
kubectl apply -f - <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: pull-model-guestbook-appset
  namespace: openshift-gitops
spec:
  generators:
  - clusterDecisionResource:
      configMapRef: acm-placement
      labelSelector:
        matchLabels:
          cluster.open-cluster-management.io/placement: pull-model-placement
      requeueAfterSeconds: 30
  template:
    metadata:
      name: '{{name}}-guestbook'
      namespace: openshift-gitops
      labels:
        apps.open-cluster-management.io/pull-to-ocm-managed-cluster: 'true'
      annotations:
        # Required whenever the hub's own app-controller is enabled (hybrid mode) and the
        # template's destination.server is the in-cluster address — otherwise the hub tries to
        # reconcile this placeholder Application locally in ADDITION to it being propagated.
        argocd.argoproj.io/skip-reconcile: 'true'
        apps.open-cluster-management.io/ocm-managed-cluster: '{{name}}'
        apps.open-cluster-management.io/ocm-managed-cluster-app-namespace: openshift-gitops
    spec:
      project: default
      source:
        repoURL: https://github.com/argoproj/argocd-example-apps.git
        targetRevision: HEAD
        path: guestbook
      destination:
        server: https://kubernetes.default.svc   # rewritten to the spoke's in-cluster server on delivery
        namespace: guestbook
      syncPolicy:
        automated: {prune: true, selfHeal: true}
        syncOptions: [CreateNamespace=true]
EOF

# 4. Verify (see "Basic Pull Model" under Architecture for what each layer means):
kubectl get applications.argoproj.io -n openshift-gitops                     # hub placeholder — always specify the group, see Troubleshooting
kubectl get manifestwork -n <ocp-managed-cluster-name> | grep guestbook      # delivery vehicle
kubectl get multiclusterapplicationsetreports -n openshift-gitops           # aggregated status back on the hub
# on the spoke itself:
kubectl get applications.argoproj.io -n openshift-gitops                    # propagated copy, should be Synced
kubectl get pods -n guestbook

Confirmed end-to-end on a real OCP-managed cluster: the ApplicationSet generated the hub placeholder Application, the propagation controller wrapped it into a ManifestWork, the spoke's existing klusterlet work-agent delivered it with no new agent process involved, the spoke's ArgoCD application-controller reconciled it (Synced), and both status paths reported back correctly — the ManifestWork status-feedback path fed multiclusterstatusaggregation, which produced a MulticlusterApplicationSetReport with syncStatus: Synced and the real guestbook-ui Deployment/Service listed under resources, and application_status_controller separately reflected the same sync/health/operation state onto the hub's own Application.status (with a synthetic AdditionalStatusReport condition pointing at the report and the spoke Application for further detail). The demo guestbook-ui pod itself does not reach Healthy on OCP — this is the same pre-existing, non-pull-model-related restricted-SCC limitation already documented under "E2E Testing Gotchas" (non-root process binding port 80); Synced is the correct success bar for this specific demo image on OCP, not Healthy.

Disabling the Basic Pull Model (multicluster-integrations-config)

The basic pull model can be turned off hub-wide via a ConfigMap, without deleting anything or disrupting real workloads. This is the first step of the migration runbook below, but it's a standalone feature in its own right — flip it whenever you want the basic-pull-model controllers to stop doing any work.

ConfigMap name: multicluster-integrations-config. Namespace: the same namespace the multicluster-integrations pod itself runs in — resolved dynamically at runtime (POD_NAMESPACE env var, falling back to reading the pod's mounted service-account namespace file), never hardcoded to open-cluster-management. Package: pkg/pullmodelconfig.
Self-provisioning: if the ConfigMap doesn't exist, whichever of the three containers starts first creates it with defaults (pullModel.basic.disabled: false) — race-safe across the sibling containers in the same pod (Create, falling back to Get on AlreadyExists).
Schema (data["config.yaml"], parsed as YAML):
```
pullModel:
  basic:
    disabled: true
```
Nested under pullModel.basic specifically to disambiguate from argocd-agent, which is also a pull-based model in the generic sense — this switch only ever affects the classic ManifestWork-based basic pull model, never argocd-agent.
Checked live, not cached at startup. Each of the three components re-reads this ConfigMap (a real Get, via pullmodelconfig.LoadOrCreate) at the point where the decision matters — propagation-controller/application's two reconcilers check it as the first thing inside every single Reconcile() call, and gitopssyncresc/multiclusterstatusaggregation check it on every tick of their periodic loop, not once when the process started. This was a deliberate fix, not the original design: an earlier version cached the flag once at process startup, which left a real race during a kubectl rollout restart — the OLD (not-yet-disabled) pod and the NEW (disabled, already-swept) pod briefly run side by side (RollingUpdate's default maxSurge), and the old pod's cached-false reconciler could process one more in-flight event and recreate a ManifestWork with ServerSideApply after the new pod's sweep had just set it to ReadOnly. Reading live on every check closes this window regardless of which pod (old or new) happens to process a given event, since both read the same up-to-date ConfigMap. Toggling the flag therefore takes effect within one reconcile/tick — no restart needed for the guards themselves.
The one-time ManifestWork sweep is the one exception that still needs a restart. It only runs once, at process startup in the propagation container's main.go, gated on the config value read at that moment — so to sweep any ManifestWork that predates flipping the flag, you still need kubectl rollout restart deployment multicluster-integrations -n <namespace>.
What disabled: true actually does, at the code level, across all three containers in the multicluster-integrations Deployment:
- propagation-controller/application's ApplicationReconciler and ApplicationStatusReconciler both return ctrl.Result{}, nil as soon as a live config check says disabled — for every event, not just filtered out at the watch/predicate level. No ManifestWork is created, updated, or deleted by this component while disabled.
- gitopssyncresc's periodic loop (Start()'s wait.Until body) skips syncResources() for any tick where the live check says disabled — no ACM Search API polling that tick.
- multiclusterstatusaggregation's periodic loop skips houseKeeping() the same way — no MulticlusterApplicationSetReport generation/cleanup that tick.
- Once, at process startup (in the propagation container's main.go, before the reconcilers are registered), application.SweepManifestWorksToReadOnly lists every ManifestWork cluster-wide, content-matches the pull-model-owned ones (has the hub-application-name annotation, and carries either the ApplicationSet-owned or standalone-pull label), and — for each one's applications-resource manifestConfigs entry specifically, never by positional index — sets updateStrategy: ReadOnly if it isn't already. This is what stops the spoke's klusterlet work-agent from continuing to enforce already-delivered Application specs, without deleting any ManifestWork. Idempotent: a re-run (e.g. next pod restart) finds nothing left to do and is a fast no-op.
- None of this imports or touches pkg/controller/gitopscluster — verified zero coupling in either direction. Disabling the basic pull model cannot affect argocd-agent.
This is a hub-wide switch, not per-app or per-cluster: it affects every basic-pull-model app across every managed cluster simultaneously. There is no code path to disable it for one app or one cluster only.
Re-enabling (disabled: false + restart) resumes normal reconciliation, but does not automatically revert any ManifestWork the sweep already set to ReadOnly — there is no "undo" sweep. If you want a specific app's ManifestWork back under active enforcement, patch its updateStrategy back explicitly.
RBAC: no changes needed — multicloud-integrations-role already grants full CRUD on configmaps and manifestworks/manifestworks/status cluster-wide (confirmed in deploy/controller/role.yaml).

Migrating an App from Basic Pull Model to argocd-agent

A step-by-step operator runbook with copy-pasteable commands lives in docs/basic-pull-to-argocd-agent-migration.md. This works whether the pull model was set up via gitopsAddon.enabled: true or via GitOpsCluster.spec.createBlankClusterSecrets: true directly — the latter being how the pull model predates and can exist entirely independently of the addon/agent machinery (see "Basic Pull Model" under Architecture).

Goal: argocd-agent takes over the same Application object — same name, same UID, no new Application or ApplicationSet ever created — with zero pod restart/recreation, and without the old pull-model controllers (multicluster-integrations's propagation controller) continuing to touch it afterward. Two earlier, more roundabout designs were tried and rejected; see "Designs that were rejected" below for why creating a second Application (even under a different name) is unnecessary and worse than this one. Getting genuine agent management (not just a hub Application that merely looks migrated) additionally requires step 5 below — see it for why.

Not every pull-model app has an ApplicationSet owner. The propagation controller only requires the pull label + ocm-managed-cluster*/skip-reconcile annotations on the Application itself — a hand-created, standalone Application carrying them gets pulled and reconciled exactly like an ApplicationSet-generated one (the only difference: multiclusterstatusaggregation keys its MulticlusterApplicationSetReport generation off the application-set label, which only an ApplicationSet-owned app has — see Known Limitations #13). Steps 1-3 below are identical for both kinds; only step 4 (the takeover) branches, since a standalone app has no template to edit.

The verified sequence: transform the existing ApplicationSet's template in place

The pull-model ApplicationSet's own template is already the durable, self-enforcing source of truth for the generated Application object (that's why Step 2 below works at all — anything you change in the template, the ApplicationSet controller keeps re-asserting on the live object, including fields you didn't touch). Migrating to argocd-agent means changing what that template says — from "pull-model, hub-side placeholder" to "agent-routed, hub-dispatched" — never replacing the object it manages.

Disable the basic pull model hub-wide via the multicluster-integrations-config ConfigMap (see "Disabling the Basic Pull Model" above) and restart the pod. This freezes every existing pull-model ManifestWork's applications manifestConfig to updateStrategy: ReadOnly in one pass — no need to find and patch it by hand per app. This stops the spoke's klusterlet work-agent from re-asserting/enforcing the delivered Application's spec, without deleting anything: drifting the spoke Application's spec.syncPolicy or a label afterward is not reverted, while statusFeedBackRules (health/sync/operation status) keeps flowing to the hub the whole time. Never delete any pull-model ManifestWork, at any point, ever — see "Designs that were rejected" below; this holds regardless of which overall design you use.
Enable argoCDAgent on the SAME GitOpsCluster (patch it in place; if the cluster never had gitopsAddon at all, this is the first time it gets set, which is fine) — do this before touching the ApplicationSet template's destination (step 4), so the agent cluster secret already exists by the time the destination needs to resolve against it. Wait for AddOnDeploymentConfig's ARGOCD_AGENT_ENABLED=true (a few reconcile passes — cert chain generation is sequential, this is the "Expected Waits (Not Bugs)" pattern, not a stuck reconcile) and for cluster-<name> to appear in the ArgoCD namespace with the argocd-agent.argoproj-labs.io/agent-name label.
If the managed cluster already had its OWN, differently-named ArgoCD CR (e.g. a manually installed default openshift-gitops instance, as opposed to the addon-managed acm-openshift-gitops one that argocd-agent mode creates) — delete that old ArgoCD CR now. ArgoCD's core settings (argocd-cm, argocd-rbac-cm, argocd-secret, etc.) are shared per-namespace, not per-CR-instance; two ArgoCD CRs in the same namespace fight over them continuously (observed live: argocd-server stuck restarting on "url modified. restarting" forever). Deleting the old ArgoCD CR is safe — it does not cascade to the Applications or real workloads it was reconciling (those aren't Kubernetes-owned by the ArgoCD CR itself), only to the ArgoCD control-plane components (application-controller, repo-server, redis, server). Confirmed live: the managed pod's uid was unchanged across this deletion. This step does not apply if the cluster's ArgoCD instance was already the addon-managed acm-openshift-gitops one. Then wait for the agent pod to reach 2 consecutive stable Running samples.
Take over the app — this is the actual takeover, and the only step that touches the app's own identity. Which object you patch depends on whether the app has an ApplicationSet owner (kubectl get applications.argoproj.io $APP_NAME -n $ARGOCD_NS -o jsonpath='{.metadata.ownerReferences}'):

ApplicationSet-owned: patch the ApplicationSet's template in place — the ApplicationSet controller re-asserts the change onto the existing Application:
- remove the pull-specific label (apps.open-cluster-management.io/pull-to-ocm-managed-cluster)
- remove the two ocm-managed-cluster/ocm-managed-cluster-app-namespace annotations
- remove argocd.argoproj.io/skip-reconcile (this one now needs to come OFF, unlike the rejected designs below — the whole point is for this Application to actually be reconciled now, by the agent, instead of sitting inert)
- change spec.destination from the literal server: https://kubernetes.default.svc to name: '{{name}}', which resolves through the agent cluster secret from step 2
```
kubectl patch applicationset $PULL_APPSET -n $ARGOCD_NS --type=json -p='[
  {"op":"remove","path":"/spec/template/metadata/labels/apps.open-cluster-management.io~1pull-to-ocm-managed-cluster"},
  {"op":"remove","path":"/spec/template/metadata/annotations/apps.open-cluster-management.io~1ocm-managed-cluster"},
  {"op":"remove","path":"/spec/template/metadata/annotations/apps.open-cluster-management.io~1ocm-managed-cluster-app-namespace"},
  {"op":"remove","path":"/spec/template/metadata/annotations/argocd.argoproj.io~1skip-reconcile"},
  {"op":"remove","path":"/spec/template/spec/destination/server"},
  {"op":"add","path":"/spec/template/spec/destination/name","value":"{{name}}"}
]'
```
Standalone (no ApplicationSet): there's no template — patch the Application object itself, directly and once (nothing else re-asserts its spec, so this is permanent). Same fields, but on /metadata and /spec directly, and destination.name is the literal cluster name (there's no {{name}} generator to template it):
```
kubectl patch applications.argoproj.io $APP_NAME -n $ARGOCD_NS --type=json -p='[
  {"op":"remove","path":"/metadata/labels/apps.open-cluster-management.io~1pull-to-ocm-managed-cluster"},
  {"op":"remove","path":"/metadata/annotations/apps.open-cluster-management.io~1ocm-managed-cluster"},
  {"op":"remove","path":"/metadata/annotations/apps.open-cluster-management.io~1ocm-managed-cluster-app-namespace"},
  {"op":"remove","path":"/metadata/annotations/argocd.argoproj.io~1skip-reconcile"},
  {"op":"remove","path":"/spec/destination/server"},
  {"op":"add","path":"/spec/destination/name","value":"'"$CLUSTER_NAME"'"}
]'
```
The hub Application's .metadata.uid must not change across this patch — it's a genuine in-place field update, not a delete/recreate: for the ApplicationSet-owned case, the ApplicationSet controller reconciles an existing same-name Application by patching it, not by replacing it; for the standalone case, it's simply the same direct kubectl patch target. If the uid changes, something deleted and recreated the object instead of updating it — stop and investigate before proceeding. The Application goes Synced within seconds — but see step 5 below before trusting that this means the agent is actually managing it. Also confirm the propagation controller is permanently inert with a touch-test (annotate the app, confirm the ManifestWork's .metadata.generation does not bump).
Confirm genuine agent management — do not skip this. The Synced status from step 4 can be misleading. This app was, by definition, delivered to the spoke at least once via the classic pull model — the klusterlet work-agent applied it there directly as a plain object. That pre-existing spoke-side copy is invisible to argocd-agent's own bookkeeping (it has no record of ever creating it), so the agent cannot take real control of it. Step 4 only changes the hub's view; the spoke's copy is not automatically replaced and can keep being reconciled by whatever was already reconciling it before, which still reports back Synced even though the agent isn't managing anything. Check directly:
```
kubectl --kubeconfig $SPOKE_KUBECONFIG get applications.argoproj.io $APP_NAME -n $ARGOCD_NS -o jsonpath='{.spec.destination}'
```
- {"name":"in-cluster", ...} (or another agent-resolved name) — genuine agent control already achieved, nothing more to do.
- {"server":"https://kubernetes.default.svc", ...} (the old pull-model value) — not yet taken over. Replace the stale spoke copy: remove syncPolicy.automated on the spoke's copy first (so the delete below doesn't prune the real resources), then remove only resources-finalizer.argocd.argoproj.io from its finalizers — preserve any other, unrelated finalizer the object might carry rather than clearing the whole list — and read .metadata.finalizers back before deleting — the ApplicationSet controller (or ArgoCD's own defaulting) can silently re-add resources-finalizer.argocd.argoproj.io moments later, the same behavior documented in "Designs that were rejected" below. If the read-back is non-empty, stop and do not delete — automated is off, so there is no destructive cascade in that scenario, but a delete against a re-added finalizer just times out. kubectl has no direct delete-precondition flag, so as the closest practical equivalent, also capture .metadata.resourceVersion from that same read-back and re-check it immediately before deleting — if it changed in that window (e.g. a finalizer got re-added right after the clean read-back), stop instead of deleting an object no longer confirmed clean. Only once both checks pass, delete it directly on the spoke, confirm the real workload's pod uid is unchanged, then nudge a fresh dispatch from the hub (any harmless annotation touch — the first create attempt right after the stale copy disappears can race and silently fail once). Recheck spec.destination on the spoke again after ~30s; it should now be name-based. Confirmed live for both ApplicationSet-owned and standalone apps: the workload's pod uid never changes across this replacement, because automated (and its prune: true) was removed from the spoke copy before it was deleted.

That's the whole migration. No new Application, no new ApplicationSet, nothing to detach or delete afterward — for the ApplicationSet-owned case, the same ApplicationSet that generated the pull-model app is now the permanent, ongoing manager of the agent-dispatched one, exactly like any other agent-mode ApplicationSet; for the standalone case, the Application you patched is now a normal, permanent agent-dispatched app with nothing else managing it. The old, now-ReadOnly ManifestWork is the only leftover either way, and it's inert — leave it in place permanently (see below for why deleting it later is still a bad idea).

Designs that were rejected (and why)

Two earlier designs were tried, both created a second Application/ApplicationSet instead of transforming the original in place, and both were rejected once their actual failure modes were understood:

Strip the old Application's ownerReferences, delete the old ApplicationSet, then create a NEW agent ApplicationSet reusing the exact same Application name. This passed every immediate, automated check across 3 script cycles and then destroyed the live Deployment/Service when the identical steps were replayed manually with realistic pauses between them. Root cause: deleting an ApplicationSet that still (or very recently did) own an Application carrying resources-finalizer.argocd.argoproj.io triggers a delete-cascade that is not instantaneous — it can take real minutes to finish. The fast automated checks sampled a window where the bookkeeping objects were already gone but the cascade toward the real resources hadn't arrived yet.
Create a NEW agent ApplicationSet with a DIFFERENT Application name (e.g. <name>-agent), leaving the old pull-model ApplicationSet/Application alone, then separately detach the old one (finalizer-strip, then delete directly on the spoke). This one was actually safe — proven with 3 full cycles and extended waits — but is objectively worse than transforming in place: it creates a second, differently-named Application object for something that is, from the user's perspective, the same app; it needs an explicit detach step for the old one; and — discovered while tearing down THIS test's own demo apps afterward — deleting an ApplicationSet whose generated Application has resources-finalizer.argocd.argoproj.io and automated.prune: true is always dangerous, permanently, not just during migration — the exact same destructive cascade from design #1 reproduced again, unrelated to migration, just from ordinary test cleanup. Transforming the existing ApplicationSet in place (the sequence above) has no ApplicationSet deletion anywhere in it, so this whole class of hazard never comes up.

Standing caution, not specific to migration: never delete an ApplicationSet whose generated Application has an active resources-finalizer.argocd.argoproj.io while syncPolicy.automated is still set — that combination is what deletes the real managed resources. This applies to decommissioning a still-pull-model app (one you've decided not to migrate): disable syncPolicy.automated on the ApplicationSet's template first, confirm that reached the live Application (spec.syncPolicy no longer has automated), then delete the ApplicationSet, and only then strip the orphaned Application's finalizers to [] and delete it directly. Confirmed live this order the other way around does not reliably work: with the ApplicationSet still present, stripping the Application's finalizers to [] gets silently reverted — something (the ApplicationSet controller's own periodic template reconcile, independent of automated) re-adds resources-finalizer.argocd.argoproj.io moments later, and a kubectl delete --wait=true issued right after the strip times out waiting on a finalizer that's already back. Deleting the ApplicationSet first removes whatever keeps re-adding it, and is safe specifically because automated (and therefore the prune-on-delete cascade) was already confirmed off the live Application before that delete — the real workload's pod uid was confirmed unchanged across this sequence.

Force-stripping the finalizer is wrong once an app has already been migrated — prefer a normal delete, and always verify the spoke copy is actually gone afterward regardless. The force-strip technique above exists because a still-pull-model app's destination is a fake, unreachable URL, so ArgoCD's own finalize-then-remove-finalizer flow can never complete on its own. Once an app is agent-routed (post-migration), its destination is real and reachable, so a normal kubectl delete applications.argoproj.io $APP_NAME -n $ARGOCD_NS --wait=true should be preferred (force-stripping pre-emptively guarantees the relay-to-agent never gets a chance to happen at all).

But even a clean, non-force-stripped delete is not guaranteed to relay to the agent — always explicitly verify the spoke's copy is actually gone, every time. Confirmed live, for apps that existed before argoCDAgent was ever enabled on their cluster (true of any migrated pull-model app, by definition): in one case the relay was rejected outright, logged on the agent as source UID annotation is not found for app: <name>, leaving the hub object stuck Terminating until its finalizer was cleared manually; in another, a --wait=true delete completed on the hub with no error at all, yet the spoke's mirrored Application (and its real workload) was silently left behind regardless. Both were only caught by directly checking the spoke:

kubectl --kubeconfig $SPOKE_KUBECONFIG get applications.argoproj.io $APP_NAME -n $ARGOCD_NS
# if found with no corresponding hub object, it's an orphan -- delete it directly, on the spoke:
kubectl --kubeconfig $SPOKE_KUBECONFIG delete applications.argoproj.io $APP_NAME -n $ARGOCD_NS --wait=true

An orphaned spoke copy isn't just inert leftover clutter — it keeps actively reconciling its resources and will fight any later, unrelated app that reuses the same destination namespace and resource names, both apps repeatedly overwriting each other's argocd.argoproj.io/tracking-id, with the affected app flapping Synced/OutOfSync indefinitely until the orphan is removed.

Other gotchas hit while building and testing this

An immediate post-action check is not sufficient proof of safety — always re-verify after a real wait. This is the single biggest lesson from the rejected design #1 above: a pod uid unchanged 1 second after a risky operation does not mean the operation was safe, only that nothing had cascaded yet. Every step in the sequence above was re-confirmed after an explicit ~3 minute wait, not just immediately.
Deleting the wrong Policy name leaves the ArgoCD CR fighting cleanup forever. The governance root Policy lives in the ArgoCD namespace as <gitopscluster-name>-argocd-policy; the replica in the managed cluster's own namespace is named <argocd-namespace>.<gitopscluster-name>-argocd-policy (root namespace prefixed). Deleting only the replica — or using the un-prefixed name in the managed cluster's namespace — does nothing: the root Policy keeps re-enforcing, so config-policy-controller on the spoke recreates the ArgoCD CR every time a ManagedClusterAddOn pre-delete cleanup Job deletes it, until the Job's own timeout is exhausted and it fails permanently, wedging the MCA in Terminating. Always delete the root Policy + its PlacementBinding in the ArgoCD namespace, before deleting the MCA, per the documented cleanup order.
A default OLM install of the operator does NOT get the same RBAC the addon normally provisions. A manually-installed (not addon-managed) ArgoCD instance's application-controller ServiceAccount is scoped read-only outside its own namespace by default — the very first sync of anything into another namespace fails with services is forbidden / deployments.apps is forbidden and exhausts ArgoCD's 5-retry budget. Grant cluster-admin (or a narrower equivalent) to <argocd-name>-argocd-application-controller before the first sync, same pattern as the Hybrid Mode / RBAC-before-first-sync guidance elsewhere in this doc — and note the SA name changes when the ArgoCD instance's own name changes (e.g. after step 4 above swaps openshift-gitops for acm-openshift-gitops).
A shell script's wait_for-style helper that shells out via bash -c "..." does not inherit parent shell functions or variables. If wait_for (or any retry loop) spawns a new bash process to evaluate its condition, any helper functions (e.g. a hub() { kubectl --kubeconfig ... } wrapper) and the variables they close over must be export -f'd and export'd respectively, or the subshell silently falls through to kubectl's localhost:8080 default and the condition can never become true — the loop just hangs forever with no error, which looks identical to a slow reconcile.
kubectl JSONPath bracket-filters on dotted annotation/label keys are fragile (e.g. {.items[?(@.metadata.annotations.foo\.bar=='x')]}). Prefer -o json | python3 -c "..." for anything beyond a simple field path — it's slower to write but doesn't silently return empty on syntax the kubectl JSONPath parser doesn't like.
ArgoCD component pod label selectors are not always app.kubernetes.io/component=<name> — e.g. the application-controller StatefulSet pod carries app.kubernetes.io/name=<argocd-cr-name>-application-controller, not a generic component label. When scripting a wait condition, prefer matching on the pod name (-o name | grep application-controller) over guessing a label schema.
kubectl get subscription is ambiguous on an ACM hub, same trap as kubectl get application documented under Troubleshooting — both apps.open-cluster-management.io/v1 Subscription (OCM's classic app-lifecycle CRD) and operators.coreos.com/v1alpha1 Subscription (OLM) register the same plural. Always use kubectl get subscriptions.operators.coreos.com when checking OLM install state on a hub or spoke that also has ACM's app-lifecycle CRDs installed.
A duplicate OperatorGroup in the same namespace silently breaks OLM resolution. If a namespace already has one (e.g. a leftover gitops-addon-operator-group from a previous addon-managed install), creating a second one produces MultipleOperatorGroupsFound and the Subscription never progresses past CatalogSourcesUnhealthy: false / no InstallPlan at all — no obvious error on the Subscription itself, only on the OperatorGroup's own status conditions. Check kubectl get operatorgroup -A -o yaml for this condition before assuming a stuck install is a catalog or network problem.

Creating and Importing a Kind Managed Cluster

# 1. Create
kind create cluster --name kind-cluster1
kind get kubeconfig --name kind-cluster1 > ~/Desktop/kind-cluster1

# 2. Register on hub
KUBECONFIG=~/Desktop/hub kubectl apply -f - <<'EOF'
apiVersion: cluster.open-cluster-management.io/v1
kind: ManagedCluster
metadata:
  name: kind-cluster1
spec:
  hubAcceptsClient: true
  leaseDurationSeconds: 60
EOF

# 3. Import (CRDs FIRST, then import.yaml)
KUBECONFIG=~/Desktop/hub kubectl -n kind-cluster1 get secret kind-cluster1-import  # wait for it
KUBECONFIG=~/Desktop/hub kubectl -n kind-cluster1 get secret kind-cluster1-import -o jsonpath='{.data.crds\.yaml}' | base64 -d | KUBECONFIG=~/Desktop/kind-cluster1 kubectl apply -f -
KUBECONFIG=~/Desktop/hub kubectl -n kind-cluster1 get secret kind-cluster1-import -o jsonpath='{.data.import\.yaml}' | base64 -d | KUBECONFIG=~/Desktop/kind-cluster1 kubectl apply -f -

# 4. Wait for Available
KUBECONFIG=~/Desktop/hub kubectl get managedcluster kind-cluster1 -w

# 5. Install governance addons (without these, Policies are NEVER enforced)
KUBECONFIG=~/Desktop/hub kubectl apply -f - <<'EOF'
apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: governance-policy-framework
  namespace: kind-cluster1
spec:
  installNamespace: open-cluster-management-agent-addon
---
apiVersion: addon.open-cluster-management.io/v1alpha1
kind: ManagedClusterAddOn
metadata:
  name: config-policy-controller
  namespace: kind-cluster1
spec:
  installNamespace: open-cluster-management-agent-addon
EOF

Cleanup (Manual Deletion Order)

GitOpsCluster deletion is a no-op (no finalizer). You must delete resources in this order:

Delete ApplicationSet (removes ArgoCD apps)
Delete GitOpsCluster (stops controller from re-creating MCAs)
Delete Policy + PlacementBinding (stops ArgoCD CR enforcement on spokes)
Delete MCAs for each managed cluster (triggers pre-delete cleanup Job)
Wait for MCAs to fully delete (2-5 minutes)
Delete Placement and remaining resources
Optionally delete orphaned ADCs and PlacementDecisions

Key constraints: GitOpsCluster BEFORE MCAs. Policy BEFORE MCAs. ADC/AddOnTemplate must still exist when MCAs are deleted.

Confirm an Application is actually gone (not just --wait=false "issued") before deleting the GitOpsCluster/agent/cluster-secret underneath it — for an agent-routed (post-migration) app, skipping this can wedge it in Terminating forever. If the Application still carries resources-finalizer.argocd.argoproj.io when you request its deletion, some ArgoCD app-controller has to actually finish pruning against its spec.destination before the finalizer clears — for a destination.name-based (agent-routed) app, that means the agent's own ArgoCD instance on the spoke, not the hub. If you delete the GitOpsCluster/MCA (tearing down the agent) or the cluster secret the destination name resolves through while that finalizer is still pending, nothing can ever finish the prune, and the Application sits Terminating indefinitely with status.conditions[].type: DeletionError referencing a destination that no longer exists. Confirmed live: with two migrated Applications from two different test cycles both targeting a cluster named ocp-cluster1, letting cycle 1's stuck deletion linger while cycle 2 created its own (differently-shaped) ocp-cluster1-named cluster secret caused cycle 1's app-controller retry to resolve ocp-cluster1 to cycle 2's secret and report DeletionErrors against a completely unrelated fake URL — a confusing, misleading symptom that's really just "the destination this app needs to reach doesn't exist by that name anymore." Once the real workload is independently confirmed gone (e.g. manually verified on the spoke), it's safe to force-clear the stuck finalizer directly (kubectl patch applications.argoproj.io <name> -n <ns> --type=merge -p '{"metadata":{"finalizers":null}}') — same "target confirmed dead, mechanism to clear it naturally can't work anymore" reasoning as the documented ManagedClusterAddOn addon-pre-delete-finalizer exception above. Prefer not needing this: always kubectl delete applications.argoproj.io <name> -n <ns> --wait=true (or poll for NotFound) before proceeding to the next cleanup step, not --wait=false.

Troubleshooting

Apps Stuck at OutOfSync/Missing After RBAC Fix

ArgoCD auto-sync won't retry a revision that failed 5 times. Fix: delete Application copies on spoke clusters (strip finalizers first). Agent re-dispatches fresh copies.

Image Pull Issues on Non-OCP Clusters

Almost always the MCH pull-secret chain. Check: kubectl get secret open-cluster-management-image-pull-credentials -n openshift-gitops-operator on the spoke, decode it, verify registry.redhat.io is present. See "Pull Secret Chain" at top.

Principal CrashLoopBackOff

Missing secrets. Create a GitOpsCluster with argoCDAgent.enabled: true — the hub controller generates argocd-agent-ca, argocd-agent-principal-tls, argocd-agent-resource-proxy-tls, and argocd-agent-jwt. Principal auto-recovers.

Policy Not Compliant

Check replicated policy: kubectl get policy -n <cluster-name>. Common causes: openshift-gitops namespace doesn't exist yet (addon still installing operator), ArgoCD CRD not registered yet (operator starting up).

ManagedClusterAddOn Stuck Deleting Forever (addon-pre-delete finalizer)

A ManagedClusterAddOn's addon.open-cluster-management.io/addon-pre-delete finalizer only clears once the pre-delete cleanup ManifestWork/Job reports success back to the hub. If the managed cluster is genuinely gone (deregistered, deleted, unreachable — e.g. a Kind cluster whose container was removed outside of OCM's knowledge), that feedback can never arrive and the MCA (and its addon Deployment/Job ManifestWorks) will sit deletionTimestamp-set forever. This is one of the few legitimate exceptions to "never force-strip finalizers": once the target is confirmed dead (not just slow — check ManagedCluster Available condition and try reaching the spoke's own kubeconfig directly first), delete the addon's ManifestWorks in that cluster's namespace and then merge-patch the MCA's finalizers to [] to unblock cleanup. If the whole ManagedCluster is also being decommissioned, deleting the ManagedCluster object itself (which tears down its namespace) is cleaner than clearing individual finalizers one at a time.

Hub Controller UID Mismatch

StorageError: invalid object, UID mismatch after rapid delete/recreate. Fix:

kubectl delete lease multicloud-operators-gitopscluster-leader.open-cluster-management.io -n kube-system --ignore-not-found
kubectl -n ocm rollout restart deployment/multicluster-operators-application

ArgoCD UI Live Manifest "Resource not found"

DBM mismatch: Principal has destinationBasedMapping: true but agent doesn't. Fix: add spec.argoCDAgent.agent.destinationBasedMapping.enabled: true to agent ArgoCD CR in Policy (sibling of agent.client, not nested inside it — confirmed against generateArgoCDSpec in argocd_policy.go and a live managed cluster's ArgoCD CR).
Agent SA missing view ClusterRole: kubectl create clusterrolebinding acm-openshift-gitops-argocd-agent-cluster-reader --clusterrole=view --serviceaccount=openshift-gitops:acm-openshift-gitops-agent-agent on spoke.

Hybrid Mode: local-cluster's ArgoCD UI Resource-Tree/Manifest Drill-Down Is Broken (upstream argocd-agent bug, not fixed here yet)

Symptom: in an ApplicationSet that targets both local-cluster and real managed clusters (the standard hybrid-mode pattern), the managed-cluster app's resource tree/live manifest drill-down works fine in the ArgoCD UI, but local-cluster's does not — clicking into it shows no nodes/manifest data. Sync and health status on the Application object look completely normal the whole time (Synced, real health), which is what makes this easy to miss: it's specifically the UI's resource-tree/manifest API that's broken, not reconciliation.

Confirmed live against the actual ArgoCD /api/v1/applications/{app}/resource-tree endpoint the UI calls (not just the Application's status fields) — this is a real, 100%-reproducible bug in the deployed argocd-agent principal (v0.9.0, logged at principal startup as Starting argocd-agent (server) v0.9.0), not a bug in this repo's code, and not flakiness (ruled out: stale/corrupted Redis data — the cached blob decompresses to valid JSON when read directly with redis-cli; a stale connection pool — a fresh argocd-server restart reproduces it immediately; version skew — argocd-server and application-controller run the identical image digest):

With destinationBasedMapping: true, the principal's newAppCallback and updateAppCallback (principal/callbacks.go in argocd-agent) both call trackAppToAgent(app, app.Spec.Destination.Name) to record which agent an Application's Redis cache traffic should be routed to. Neither validated that Destination.Name actually corresponds to a connected agent in the deployed v0.9.0 (no clusterMgr.HasMapping() check in either callback). Upstream main has since closed the direct create/update path: commit 0554ad8 ("fix: cleanup agent resources after the deletion of the agent cluster secret", #1034, merged 2026-07-29) added a HasMapping guard to both newAppCallback and updateAppCallback's own bodies — so a fresh local-cluster Application created with destination.name: local-cluster from the start is already safe on current main (still not on any released version, including the deployed v0.9.0). The guard does not cover handleAppAgentChange (also in principal/callbacks.go, called from inside updateAppCallback before its own guard runs), which handles the case where an existing app's destination.Name changes — e.g. an ApplicationSet template edit, or any direct kubectl patch, moving an app from a real agent's name onto local-cluster. That function still calls trackAppToAgent for the new agent name unconditionally, so the bug persists specifically for apps that get migrated onto local-cluster after creation, even on current main. Reproduced 100% with a from-scratch Kind-based argocd-agent-only setup (no ACM/OCM): create an app targeting a real agent, confirm its resource tree loads, then patch spec.destination.name to local-cluster — the resource tree breaks immediately after. A fix (add the same HasMapping guard to handleAppAgentChange's new-agent branch) has been written and verified end-to-end (bug reproduced pre-fix, confirmed resolved post-fix, twice, across full Kind-cluster teardown/rebuild cycles) but exists only as an unpushed local commit (a2af478) on a fork checkout at /home/ming/argocd-agent — it is not part of any upstream PR, main, or release yet. Since local-cluster's Application legitimately has destination.name: local-cluster (a real, valid cluster secret — see ensureLocalClusterSecret — just not an agent-connected one), this poisons the principal's in-memory appToAgent map for it as soon as a destination-changing update lands (which happens quickly after creation whenever an ApplicationSet template is what actually sets destination.name: local-cluster, since the generator/template rendering path goes through an update rather than staying on the original create).
The principal's Redis proxy (principal/redisproxy/redisproxy.go, extractAgentNameFromRedisCommandKey) consults that same poisoned map for every app|resources-tree|.../app|managed-resources|... Redis key belonging to that app, and — believing local-cluster is an agent — tries to forward the read to that (nonexistent) agent connection instead of the principal's own Redis. The agent never responds (confirmed in principal logs: msg="no response from agent" ... getKey="app|resources-tree|guestbook-local-cluster|..." followed by msg="exit due to unable to handle agent get" error="unexpected nil in get response"), and argocd-server ends up gunzip'ing an effectively-empty response, producing exactly error getting cached app resource tree: EOF — an empty byte slice hitting gzip.NewReader() returns io.EOF, which matches precisely.
This is a name collision, not a DBM correctness issue: the moment destination.Name is non-empty and matches any real cluster secret name (agent-connected or not), the bug fires. A real spoke's Application is unaffected only because for spokes, being routed to its own agent is the correct behavior — the missing validation happens to not matter there. Restarting argocd-server, hard-refreshing the Application, or waiting does not clear it, because the poisoned mapping lives in the principal's in-memory state, not in argocd-server or Redis itself — nothing in the normal reconcile/refresh loop ever un-poisons it once set.
Confirmed (but not yet implemented anywhere) workaround: give local-cluster's generated Application a destination.server: https://kubernetes.default.svc instead of destination.name: local-cluster — both resolve to the exact same physical destination. Since getAgentNameForApp returns app.Spec.Destination.Name directly when DBM is enabled, an empty Destination.Name (server-based destination) means the resolved agent name is "", and both callbacks' existing if agentName == "" { return } guard correctly skips trackAppToAgent entirely for it. Verified live: 10/10 consecutive resource-tree fetches succeeded after manually switching an ApplicationSet's template to this shape (using goTemplate: true and a Go-template conditional keyed on the cluster name), versus 0/10 before, with the running workload's pod UID confirmed unchanged across the switch (no disruption). This is deliberately not yet applied to gitopsaddon/test-cycle.sh or anywhere else in this repo — the real fix belongs upstream in argocd-agent (adding the missing HasMapping guard to handleAppAgentChange's new-agent branch — see above; a candidate fix already exists as an unpushed local commit, not yet a PR), and that upstream work is still to be done. Only every other (real, agent-routed) cluster should ever use destination.name — switching those to server-based destinations too would break event dispatch to their agents entirely, since destination name is what the principal's dispatch queueing keys off of, not just the Redis proxy lookup.
Nothing in pkg/controller/gitopscluster can fix this: no code in this repo generates the ApplicationSet's destination field — that YAML is authored by whoever creates the ApplicationSet (a real user, or gitopsaddon/test-cycle.sh's own hardcoded template for testing). A code-level fix is not possible from this repo's side; only a workaround at the ApplicationSet-authoring layer (see above) or an upstream argocd-agent fix resolves it.

Agent Not Connecting

Check principal pod: kubectl get pods -n openshift-gitops -l app.kubernetes.io/component=principal
Check principal logs: kubectl logs -n openshift-gitops -l app.kubernetes.io/component=principal --tail=50
Check agent logs on spoke: kubectl logs -n openshift-gitops -l app.kubernetes.io/part-of=argocd-agent --tail=50
If the agent log shows [FATAL]: Error creating remote: could not read secret: secrets "argocd-agent-ca" not found — see "Expected Waits (Not Bugs)" above; this is almost always the hub→spoke CA ManifestWork propagation still in flight, not a config error.

Basic Pull Model: Blank Cluster Secret Creation Fails ("found existing non-ACM ArgoCD clusters secrets")

Symptom: GitOpsCluster status shows ClustersRegistered: False, message found existing non-ACM ArgoCD clusters secrets for cluster: <name>. Cause: a stale/orphaned cluster secret for that cluster already exists in the ArgoCD namespace — most commonly an old argocd-agent secret (argocd-agent.argoproj-labs.io/agent-name label, managed-by: argocd-agent annotation) left behind by a previously deleted GitOpsCluster that had agent mode enabled for the same cluster name. The blank-secret creation path refuses to overwrite a secret it doesn't recognize as its own placeholder. Fix: confirm the existing secret really is orphaned (its owning GitOpsCluster is gone), then delete it — the next reconcile creates a fresh blank secret.

`kubectl get application` Silently Returns Nothing on an ACM Hub

An ACM hub has more than one CRD registered under the bare kind Application (ArgoCD's applications.argoproj.io and the older applications.app.k8s.io from the classic subscription/channel model). An unqualified kubectl get application can resolve to the wrong one and print No resources found even though the ArgoCD Application you're looking for exists. Always use the fully-qualified kubectl get applications.argoproj.io when inspecting pull-model or agent-mode Applications on a real hub.

Hub App-Controller Fights the Principal Over an Autonomous Mirror's Status

Symptom: an autonomous-mode Application mirrored to the hub flips status.sync.status back to Unknown on every refresh cycle even though the spoke's own copy is genuinely Synced, and the hub app-controller logs error getting app project "<agent>-default": appproject.argoproj.io "<agent>-default" not found for it. This is a real ArgoCD version gap, not a bug in this repo's code: skip-reconcile on the agent's cluster secret (confirmed correctly set by CreateArgoCDAgentClusters for every agent-connected cluster, managed or autonomous) and the principal's destination-rewrite to that named secret (confirmed correct in argocd-agent's principal/event.go) are both wired correctly - but versions of ArgoCD prior to argo-cd#26442 (merged 2026-02-19, reported as argo-cd#26425) only had skip-reconcile gate the sync/reconcile-action trigger, not the periodic status-refresh/comparison path - so the hub's own app-controller still attempts to resolve the mirrored app's spec.project (rewritten by the principal to <agent>-<project>, which only ever exists on the spoke for autonomous mode) and errors on every attempt. Fix is an ArgoCD version bump to one that includes that PR - not a hub app-controller toggle or a code change here. Confirm which ArgoCD version you're running via the application-controller pod's startup log (level=info msg="ArgoCD Application Controller is starting" ... version=vX.Y.Z).

Architecture

Hub Controller ↔ Addon Agent

The gitopscluster controller (hub) creates: AddOnDeploymentConfig (env vars per cluster), ManagedClusterAddOn (triggers addon deployment), AddOnTemplate (agent manifests), ArgoCD Policy (ArgoCD CR enforcement), PlacementBinding, cluster secrets (with agentName for agent routing).

The gitopsaddon agent (spoke) detects cluster type (OCP/non-OCP/hub), installs the GitOps operator, waits for Policy-deployed ArgoCD CR, patches ServiceAccounts with pull secrets (non-OCP only).

OCP vs Non-OCP Operator Version Gap (Critical)

OCP and non-OCP managed clusters run different operator versions:

OCP: gitopsaddon creates an OLM Subscription (latest channel, redhat-operators catalog). OLM resolves to whatever version is current in the catalog. All component images (operator, ArgoCD, redis, dex) are determined by the OLM CSV. The hub controller has zero control over these. When Red Hat publishes a new version, OCP clusters auto-upgrade.
Kind/EKS (non-OCP): gitopsaddon deploys the embedded Helm chart (gitopsaddon/charts/openshift-gitops-operator/). The version is in Chart.yaml. All component images are hardcoded SHA digests in pkg/utils/config.go DefaultOperatorImages. These never change until someone runs the Manifest Update Procedure.
Agent image is the exception: On both OCP and Kind, the argocd-agent image comes from the hub controller's drift heal — it reads the principal pod's image and writes it into each managed cluster's AddOnDeploymentConfig ARGOCD_AGENT_IMAGE (not the shared ArgoCD Policy). So the agent image is always consistent with the hub principal regardless of spoke operator version, and still flows through per-cluster ManagedClusterImageRegistry mirroring. Annotate the GitOpsCluster with skip-agent-version-heal: "true" to opt out; a user-set Policy spec.argoCDAgent.agent.image is never overwritten either way.
The version gap means: the embedded operator on Kind/EKS may lack features, CRD fields, or RBAC rules added in newer OCP versions. When bumping, update: Chart.yaml version, CRDs (extract from hub), ClusterRole rules (from CSV), image SHAs in config.go. See Manifest Update Procedure.
Manifest update lessons: (1) ClusterRole rules extracted from CSV contain * wildcards that MUST be quoted as "*" in YAML — unquoted * is a YAML alias and causes silent parse failures (the ClusterRole is never created, operator crashes with informer cache sync timeouts). (2) Route and Monitoring CRD stubs are NOT needed — the operator has proper API discovery (argoutil.VerifyAPI() / InspectCluster()) that gracefully skips Route and Monitoring watches when the APIs don't exist on non-OCP. (3) Run test-cycle.sh for at least 2 cycles to catch inter-cycle state issues. (4) Do NOT modify anything from the source of truth — if the CSV has duplicate rules, empty apiGroups, OLM labels on CRDs, or other quirks, keep them in the chart exactly as extracted. The chart must match the source of truth (hub cluster, orb bundle, upstream repos). Fixing upstream issues in the chart causes it to go out of sync and makes future verification impossible. The same applies to CRD metadata labels — if the hub CRDs have operators.coreos.com labels, keep them. The gitopsaddon label (apps.open-cluster-management.io/gitopsaddon: "true") is added at deploy time by templateAndApplyChart, not in the chart YAML itself.

Template Modes

Mode	Template Name	When Used
Static	`gitops-addon`	`gitopsAddon.enabled=true`, `argoCDAgent.enabled=false`
Dynamic	`gitops-addon-{ns}-{name}`	`argoCDAgent.enabled=true` (includes `RegistrationSpec` for client cert provisioning)

Agent Modes

Aspect	Managed	Autonomous
Source of truth	Hub (principal)	Spoke
App creation	Hub; principal dispatches	Spoke (Policy/Git/kubectl)
Hub capability	Full control	Read-only mirror
AppProject in Policy	Not included (principal propagates)	Included
Best for	Centralized control	App of Apps, GitOps-first

Hybrid Mode (hub app-controller + argocd-agent principal, same instance)

The hub runs a single ArgoCD instance in openshift-gitops that hosts both the argocd-agent principal (dispatching to managed clusters) and its own application controller (controller.enabled: true) reconciling anything targeting local-cluster/in-cluster directly. There is no separate ArgoCD instance or namespace for the hub — that legacy design (a dedicated local-cluster namespace with its own agent-connected ArgoCD instance) has been removed.

Isolation mechanism: every agent-routed cluster secret (cluster-<name> for real spokes) carries argocd.argoproj.io/skip-reconcile: "true", so the hub's app-controller ignores anything dispatched to an agent and lets the principal own it exclusively. local-cluster's own cluster secret (cluster-local-cluster, created by ensureLocalClusterSecret in argocd_agent_clusters.go) carries no skip-reconcile and no agent-name label — it's a plain cluster secret pointing at https://kubernetes.default.svc, so the hub app-controller reconciles it like any other in-cluster ArgoCD destination.
local-cluster is never an addon-install target: it already has GitOps operator + ArgoCD installed independently (a prerequisite, not addon-managed). The GitOpsCluster's own placementRef must always exclude local-cluster — the addon-install loop (CreateAddOnDeploymentConfig/EnsureManagedClusterAddon) and the ArgoCD-CR-enforcing Policy both skip/must never select it (IsLocalCluster guards this in gitopscluster_controller.go/argocd_agent_clusters.go; a warning is logged if the Placement ever resolves to include it, since the Policy would otherwise deploy a second, conflicting ArgoCD CR into openshift-gitops).
ApplicationSets targeting local-cluster: use a Placement separate from the GitOpsCluster's own placementRef — this one is free to include local-cluster alongside real managed clusters. The generated Application resolves destination: {name: 'local-cluster'} against the plain cluster-local-cluster secret and gets created in openshift-gitops, reconciled by the hub's app-controller.
RBAC: the operator's default ClusterRole for the app-controller is read-only outside openshift-gitops itself. Any Application targeting local-cluster needs the hub app-controller's own ServiceAccount (openshift-gitops-argocd-application-controller) granted elevated RBAC (e.g. cluster-admin, matching the same pattern already used for spoke app-controllers) in whatever destination namespaces those apps create — do this before creating the AppSet, or the first sync attempt exhausts ArgoCD's 5-retry budget and self-heal won't retry a revision that already failed that many times.

Basic Pull Model (Classic ManifestWork-Based Delivery)

The original (2023) integration, predating argocd-agent and still shipping by default alongside it — the two are intentionally coexisting alternatives, not a deprecated/replacement pair. Nothing in the code or git history marks the pull model itself as deprecated (only the MulticlusterApplicationSetReport v1alpha1 CRD carries a Kubernetes-convention deprecation warning — a data-model sunset note, not a statement that the mechanism is going away).

Key distinction from argocd-agent: no new agent process, no gRPC, no principal/cert machinery. It reuses the klusterlet work-agent every managed cluster already has (the same channel used for Policies and classic Subscriptions) via the standard ManifestWork API.

Data flow:

Hub creates a blank/dummy cluster secret per target cluster (createBlankClusterSecrets, forced true whenever gitopsAddon is enabled) — server: https://<cluster>-control-plane, a deliberately fake/unreachable URL. Its only purpose is making ArgoCD's ApplicationSet clusterDecisionResource generator treat the cluster as a valid destination; nothing ever actually connects to that URL.
The propagation controller (propagation-controller/application/, part of the multicluster-integrations deployment) watches Application objects cluster-wide for label apps.open-cluster-management.io/pull-to-ocm-managed-cluster: "true" plus annotation apps.open-cluster-management.io/ocm-managed-cluster: <cluster-name>. On match, it wraps the Application (destination rewritten to https://kubernetes.default.svc, i.e. "reconcile in-cluster" — meaning in the spoke's own cluster once delivered) plus its Namespace and AppProject (skipped if it's just the default default/openshift-gitops pair, since every ArgoCD install already has that) into a ManifestWork in the target cluster's hub namespace, with FeedbackRules requesting health/sync/operation status back.
The spoke's own existing klusterlet work-agent pulls that ManifestWork down and applies it — no new network channel, no addon-specific delivery path.
The spoke's own ArgoCD (installed by gitopsaddon, same as agent mode, just without the argoCDAgent block) reconciles the delivered Application against its local API server.
Status returns two ways: ManifestWork.status.resourceStatus feedback is consumed by multiclusterstatusaggregation, merged with fine-grained detail polled from ACM's search-search-api by gitopssyncresc (communicated between the two sidecar containers via a shared emptyDir, not an API), into a MulticlusterApplicationSetReport CR — and separately, application_status_controller (same propagation-controller binary) watches that report and reflects health/sync/operation state directly onto the hub's own placeholder Application object, plus a synthetic AdditionalStatusReport condition pointing users at the report and the spoke's own Application for full detail.
Only Applications generated by an ApplicationSet (i.e. carrying an ApplicationSet ownerReference) get the apps.open-cluster-management.io/application-set label on their ManifestWork — a hand-created, non-ApplicationSet-owned Application still gets pulled and reconciled, but produces no MulticlusterApplicationSetReport (multiclusterstatusaggregation only lists ManifestWorks carrying that label).

Coexistence with hybrid mode / argocd-agent: if the hub's own app-controller is enabled (hybrid mode, controller.enabled: true) and the pull-model placeholder Application's destination.server is the in-cluster address, the hub app-controller would otherwise try to reconcile that same placeholder locally, on top of the propagation controller delivering it to the spoke. Set argocd.argoproj.io/skip-reconcile: "true" directly on the pull-model Application itself (not just on cluster secrets — this is the same generic upstream ArgoCD annotation, applied at a different level) to keep the hub's copy inert.

OCP-only: this integration is supported only for OCP managed clusters. Both OCP and non-OCP spokes get a real ArgoCD instance from gitopsaddon, but the pull model has not been validated against the non-OCP embedded-chart path.

local-cluster support is currently dormant, not actively gated: an earlier iteration supported pulling Applications onto local-cluster itself, via a dedicated gitopsaddon-deployed ArgoCD instance in a local-cluster namespace. That dedicated instance was removed when hybrid mode replaced it with the plain in-cluster cluster-local-cluster secret (see "Hybrid Mode" above), and the e2e coverage for pull-model-on-local-cluster was deleted at the same time. No explicit code-level guard currently blocks targeting local-cluster with a pull-model Application (unlike the Maestro variant, which explicitly skips it), but the mechanism it used to depend on no longer exists in the same shape — treat this path as unverified.

Package Structure

pkg/controller/gitopscluster/ — Hub controller (reconciler, policy gen, certs, addon mgmt, agent clusters)
gitopsaddon/ — Addon agent (install logic, controller, cleanup, secret controller, embedded Helm chart)
gitopsaddon/charts/openshift-gitops-operator/ — Embedded operator for non-OCP
deploy/crds/ — Hub-side CRDs generated by controller-gen (GitOpsCluster, etc.) + some legacy operator CRDs. NOT used for embedded chart deployment — only gitopsaddon/charts/.../templates/crds/ matters for spoke operator install.
test/e2e/gitopsaddon/ — Ginkgo e2e tests
Reference copies of upstream repos (argocd-agent, gitops-operator, argocd-operator) are useful for cross-referencing during manifest updates — clone them fresh with git clone when needed. Clone them OUTSIDE the repo tree (e.g. /tmp), never into the repo root. Each has its own go.mod, but make manifests (a dependency of every test-e2e-gitopsaddon-*/test-local-e2e-gitopsaddon-* target) runs controller-gen crd webhook paths="./...", which does NOT respect nested-module boundaries the way go build ./... does — if these repos are present under the repo root, controller-gen scans their vendored API types too and silently regenerates unrelated/bogus CRDs into deploy/crds/ (e.g. argoproj.io_argocds.yaml, a new argoproj.io_argocdexports.yaml). Always check git status for unexpected deploy/crds/ changes after running any e2e target if reference repos were ever cloned into the tree, even temporarily. Do not edit these reference copies.

Key Annotations on GitOpsCluster

skip-agent-version-heal: "true" — Disables writing the hub principal image into each cluster's AddOnDeploymentConfig ARGOCD_AGENT_IMAGE, and omits that variable from the managed ADC set entirely, so a user-set Policy spec.argoCDAgent.agent.image or a manually-set ADC value is not overwritten. Does not touch the ArgoCD Policy (the controller never writes or clears Policy agent.image).
skip-argocd-policy: "true" — Prevents Policy recreation. Independent of agent-image heal (heal writes ADC, not Policy).
skip-hub-ca-propagation: "true" — Skips Hub CA via ManifestWork

Key Annotations on ManagedClusterImageRegistry

apps.open-cluster-management.io/gitops-addon-image-mirroring: "true" — Opt-in. Without this, the gitops-addon image-registry controller ignores the CR (ACM's own cluster-image-registry-controller still uses it for klusterlet images). Existing disconnected-registry setups are unchanged until a user explicitly annotates the CR. Removing the annotation reverts any gitops-addon ADC mirroring this controller previously applied.

Key Features (Deep Reference)

Agent Version Drift Heal

Controller watches the principal Deployment (app.kubernetes.io/component=principal) for image changes. When the image changes (operator upgrade), it writes the live principal image into each managed cluster's own AddOnDeploymentConfig ARGOCD_AGENT_IMAGE — never the shared ArgoCD Policy. That ADC value then flows through the same per-cluster ManagedClusterImageRegistry mirroring pipeline as every other addon image. findContainerImage(containers, deploymentName) checks: (1) container named "principal", (2) container matching deployment name, (3) containers[0].

skip-agent-version-heal: "true" on the GitOpsCluster opts this out: HealAgentVersionDrift returns empty, and ExtractVariablesFromGitOpsCluster drops ARGOCD_AGENT_IMAGE from the managed set so CreateAddOnDeploymentConfig will not overwrite a user-set ADC value. reconcileArgoCDPolicyAgentSpec never writes or clears spec.argoCDAgent.agent.image, so users who want to pin the agent image via Policy can do so whether or not the skip annotation is set.

Disconnected Image Mirroring (ManagedClusterImageRegistry)

A dedicated controller (imageregistry_controller.go) rewrites gitops-addon AddOnDeploymentConfig image variables to disconnected-registry mirrors. It is opt-in: only a ManagedClusterImageRegistry annotated apps.open-cluster-management.io/gitops-addon-image-mirroring: "true" is watched or reconciled. CreateAddOnDeploymentConfig preserves already-mirrored values when the recorded original still matches the desired source, so the two controllers do not fight over the ADC on the steady-state path; a genuine source change (hub image upgrade / principal heal) is written through so this controller can re-mirror it.

Cert Rotation & Autorefresh

argocd-agent-client-tls has 24-hour validity. OCM ClientCertController rotates at ~80% lifetime. SecretReconciler propagates via: source watch (immediate), target deletion watch (instant re-copy), periodic requeue (5 min safety net). On actual cert data change, rolling-restarts all agent Deployments (app.kubernetes.io/part-of=argocd-agent).

Resource Proxy TLS SAN

argocd-agent-resource-proxy-tls cert must include the resource-proxy service DNS name. getResourceProxyHostNames calls FindArgoCDAgentResourceProxyService directly. deleteSecretIfSANsDrifted detects missing SANs and deletes the secret for re-issuance. Workaround: kubectl delete secret argocd-agent-resource-proxy-tls -n openshift-gitops.

Cluster Secret Server URL

Preferred: resource proxy URL https://...-resource-proxy.<ns>.svc:9090?agentName=<cluster>. Fallback: NodePort/LoadBalancer address. Routing by argocd-agent.argoproj-labs.io/agent-name label.

Skip-Reconcile Annotation on Agent Cluster Secrets

Agent cluster secrets are stamped with argocd.argoproj.io/skip-reconcile: "true". This tells the ArgoCD application controller on the hub to skip reconciliation for applications targeting that cluster — the agent handles reconciliation instead. Required for hybrid mode where a traditional app-controller coexists with the argocd-agent principal on the same hub. Without it, both the hub app-controller and the agent try to reconcile the same apps, causing conflicts. Set automatically by CreateArgoCDAgentClusters — no manual action needed.

Non-Agent → Agent Mode Transition

Creating agent mode after non-agent creates duplicate cluster names. CreateArgoCDAgentClusters calls deleteBlankClusterSecretsForCluster to remove old blank pull-model secrets.

OLM Subscription Details

Always in openshift-gitops-operator ns. OperatorGroup must be AllNamespaces (not OwnNamespace). patchArgoCDCRDConversionWebhookIfNotReady patches CRD conversion to None before subscription. recoverFromFailedWebhookInstallPlan handles failed InstallPlans from webhook timeout.

Addon RBAC

gitops-addon ClusterRole (fine-grained, not cluster-admin). Defined in both addonTemplates.yaml (static) and addon_template_management.go (dynamic). Separate gitops-addon-cleanup ClusterRole annotated addon-pre-delete for cleanup Job RBAC. Includes escalate/bind for operator leader-election Role.

No-Op-Update Guards on Hub-Owned Objects (AddOnTemplate, AddOnDeploymentConfig)

Both EnsureAddOnTemplate and the AddOnDeploymentConfig path in addon_management.go rebuild their desired object from scratch every reconcile, then must compare against what's already stored before calling Update() — an unconditional Update() bumps resourceVersion (and, for spec changes, .metadata.generation) even when nothing meaningfully changed, and the OCM addon framework treats a generation bump on an AddOnTemplate as "the workload changed," re-rendering and rolling every managed cluster's addon Deployment in response. Confirmed on a real hub: an AddOnTemplate left running for ~6 days accumulated generation 4157 (~2 min/update) purely from this pattern, with zero real config change during that window.

How to design the comparison, not just "add one": don't reflexively deep-Equal the whole built object — trace which of its inputs are actually capable of changing across reconciles of the same parent resource. Identity fields (name/namespace-derived) and anything hardcoded are permanently constant once the object exists; only genuinely external, mutable inputs (a controller's own image, a user-editable CR field) are worth comparing, and usually there are only one or two. Comparing just those is cheaper, isn't fooled by incidental formatting/ordering noise in unrelated parts of the object, and can't produce false "changed" reports that bring back the same churn under a different guise. customizedVariablesEqual (order-insensitive map comparison) is the existing pattern for AddOnDeploymentConfig; addOnTemplateNeedsUpdate (direct comparison of just the image and CA namespace, extracted with a narrow targeted JSON decode rather than a full manifest unmarshal) is the pattern for AddOnTemplate.

A stale map-iteration order can independently defeat any such guard. Building an ordered []corev1.EnvVar (or any positional/ordered output) by ranging directly over a map[string]string picks up Go's intentionally-randomized iteration order — the same content can serialize in a different order on every single call, which a naive equality check (or even a correct one, if it compares raw bytes/ordered slices instead of decoded/set content) will see as "changed" forever. Sort the keys before iterating whenever the output must be stable across calls, independent of whether anything is currently comparing it — otherwise a future correctness fix elsewhere can silently stop working the moment it starts relying on this output being stable.

Verifying a no-op-update fix on a live, leader-elected controller: kubectl rollout status reporting a deployment fully rolled out does NOT mean the new pod has taken over reconciling - leader election makes the new pod wait out the previous holder's full lease duration (leaseDuration, minutes) before "Successfully acquired lease" appears in its logs, regardless of how quickly the old pod terminated. Check for that log line (or an actual "Process <resource>" line) before concluding a just-applied config change didn't take effect - the more likely explanation is the new pod simply hasn't started reconciling yet.

Cleanup Rules

NEVER list-and-delete blindly. Only delete: (1) gitopsaddon-labeled ArgoCD CRs (waits for operator finalizer), (2) OLM Subscription/CSV on OCP, (3) gitopsaddon-labeled operator resources on embedded. ArgoCD workloads are the operator's responsibility. Never force-strip ArgoCD CR finalizer. On OCP, skip deleteOperatorResources — OLM handles it. templateAndApplyChart labels all resources with apps.open-cluster-management.io/gitopsaddon: "true".

Custom ArgoCD Namespace

spec.argoServer.argoNamespace overrides the namespace. GetEffectiveArgoNamespace() must be used for all operations (cert gen, CA propagation, service discovery).

Image Pull Secret Chain (Non-OCP)

Hub multiclusterhub-operator-pull-secret (in multicluster-engine + open-cluster-management) → managedcluster-import-controller bakes into {cluster}-import secret → klusterlet copies as open-cluster-management-image-pull-credentials → addon copies to ArgoCD namespaces → patchServiceAccountsWithImagePullSecrets patches all SAs. installViaEmbeddedManifests patches operator SA immediately after chart apply, deletes ImagePullBackOff pods.

Build / Push / Redeploy Workflow

# Build and push
docker build -t quay.io/stolostron/multicloud-integrations:latest -f build/Dockerfile .
docker push quay.io/stolostron/multicloud-integrations:latest

# Redeploy on hub
kubectl -n open-cluster-management set image deployment/multicluster-operators-application \
  multicluster-operators-gitopscluster=quay.io/stolostron/multicloud-integrations:latest
kubectl -n open-cluster-management rollout restart deployment multicluster-operators-application

# Prevent MCH operator revert
kubectl -n open-cluster-management scale deploy multiclusterhub-operator --replicas=0

# Set CONTROLLER_IMAGE for dynamic AddOnTemplates
kubectl -n open-cluster-management set env deployment/multicluster-operators-application \
  -c multicluster-operators-gitopscluster CONTROLLER_IMAGE=quay.io/stolostron/multicloud-integrations:latest

# Hub controller imagePullPolicy (for :latest tag)
kubectl get deploy multicluster-operators-application -n open-cluster-management -o json | \
  python3 -c "import json,sys; d=json.load(sys.stdin); [c.__setitem__('imagePullPolicy','Always') for c in d['spec']['template']['spec']['containers'] if c['name']=='multicluster-operators-gitopscluster']; json.dump(d,sys.stdout)" | kubectl apply -f -

Manifest Update Procedure (OpenShift GitOps Operator)

Hub cluster is the source of truth. Upstream repos (redhat-developer/gitops-operator, argoproj-labs/argocd-operator) are for cross-referencing — clone them fresh, OUTSIDE the repo tree (e.g. /tmp or ~/), each time you need them (see the deploy/crds pollution hazard and the bundle/manifests staleness warning below). Use orb for independent bundle verification.

Verification direction: SOURCE FIRST, then chart

Always start from the source of truth (hub cluster / orb bundle / upstream repo), enumerate everything it has, then check whether the chart matches. Never start from the chart and ask "does this look right" — that misses resources that should exist but don't, and resources that should be deleted but still exist. The source tells you what SHOULD be there; the chart is what you're validating.

What to update (complete checklist)

CRDs, ClusterRole, images, and Deployment env vars are obvious. But also check: leader-election Role rules, ConfigMap content, ServiceAccount changes, new/removed Services, new env vars in the Deployment template, new ports, new volumes, changed args/probes. The OLM bundle restructures RBAC (hashes names, combines roles) — the chart keeps them separate with readable names. Don't blindly copy OLM structure.

"Content, not count" — a rule count or file-existence match is not verification. Two RBAC rule sets with the same count can differ by a rename/removal+addition pair that cancels out numerically. Every comparison in this procedure must be a normalized content diff (compare apiGroups/resources/verbs/resourceNames sets, or full YAML equality after stripping only genuinely-dynamic fields) with an explicit list of "only in old" / "only in new" — never just "44 rules on both sides, looks fine."

The upstream repo's checked-in bundle/manifests/ can be STALE — do not trust it for CSV/RBAC/resource content. When cloning gitops-operator for cross-referencing, checking out the release branch (e.g. v1.21) does NOT guarantee bundle/manifests/gitops-operator.clusterserviceversion.yaml reflects the current released z-stream — it can be a stale fixture left over from a much earlier version (observed: a v1.21 branch checkout's bundle CSV showing version: 1.8.0). The live hub cluster CSV (kubectl get csv <name> -n openshift-gitops-operator -o yaml) is the only trustworthy source for RBAC rules, Deployment spec, images, ConfigMap/Service content — always extract from there, never from a cloned repo's bundle/manifests/. Only use the cloned repo for things that are genuinely version-controlled Go source (e.g. controllers/util/util.go's API-discovery logic), where git blame/branch state is meaningful.

Complete file inventory of gitopsaddon/charts/openshift-gitops-operator/templates/ (verified individually against a live hub multiple times across different bumps — use this as the checklist for every bump, not just the first one):

File	Live hub equivalent	How it was verified
`crds/*.crd.yaml` (14 files)	`kubectl get crd <name> -o yaml`	Full content diff after stripping runtime metadata + OLM-injected `olm.managed`/`operatorframework.io/` labels (these are added by OLM's live reconciliation, NOT part of the source CRD — strip them; but do NOT strip `operators.coreos.com/` labels, `deprecated`/`deprecationWarning` version fields, or anything else that's part of the real schema)
`openshift-gitops-operator-manager-role.clusterrole.yaml`	CSV `spec.install.spec.clusterPermissions[0].rules`	Full rule-set diff (see "content, not count" above)
`openshift-gitops-operator-leader-election-role.role.yaml`	CSV `spec.install.spec.permissions[0].rules`	Full rule-set diff
`openshift-gitops-operator-controller-manager.deployment.yaml`	CSV `spec.install.spec.deployments[0].spec`	Full diff of args, env vars, ports, probes, volumes, securityContext against the intentional-differences list below
`openshift-gitops-operator-controller-manager.serviceaccount.yaml`	`kubectl get sa openshift-gitops-operator-controller-manager -n openshift-gitops-operator -o yaml`	Name match; chart's extra `imagePullSecrets: open-cluster-management-image-pull-credentials` is an intentional non-OCP addition (OLM instead auto-injects its own internal-registry pull secret via annotation — OCP/OLM-only mechanism, not needed on the chart)
`openshift-gitops-operator-manager-rolebinding.clusterrolebinding.yaml`	ClusterRoleBinding with `olm.owner: <csv-name>` label bound to the hashed manager ClusterRole	Confirms same roleRef target (by content, not name) + same subject SA
`openshift-gitops-operator-leader-election-rolebinding.rolebinding.yaml`	RoleBinding named `<csv-name>` in the operator namespace	Confirms same roleRef target + same subject SA
`openshift-gitops-operator-manager-config.configmap.yaml`	`kubectl get cm openshift-gitops-operator-manager-config -n openshift-gitops-operator -o yaml`	Byte-identical `data` content confirmed. Note: this file is scaffolding from `controller-runtime`'s default manager options and is NOT actually wired to the Deployment via a `--config` flag — the real config comes from the CLI args, which win. Harmless to keep in sync but functionally inert either way.
`openshift-gitops-operator-webhook-service.service.yaml`	`kubectl get svc openshift-gitops-operator-webhook-service -n openshift-gitops-operator -o yaml`	Port/targetPort/selector match exactly
`openshift-gitops-operator-metrics-service.service.yaml`	`kubectl get svc openshift-gitops-operator-metrics-service -n openshift-gitops-operator -o yaml`	Pre-existing, long-standing mismatch, NOT version-bump-related — do not "fix" during a routine bump: chart names it `openshift-gitops-operator-controller-manager-metrics-service` (source: `openshift-gitops-operator-metrics-service`), port name `https` vs source `metrics`, `targetPort: 8080` (numeric) vs source `targetPort: metrics` (named, resolves to the container's `containerPort: 8443, name: metrics`). None of this matters functionally: the manager process only binds `127.0.0.1:8080` (loopback), so no Service — regardless of exact shape — can actually reach it from outside the pod. This Service has been a functionally-inert stub across multiple bumps already, consistent with the `--metrics-bind-address=127.0.0.1:8080` exception below.
`openshift-gitops-operator-metrics-reader.clusterrole.yaml`	No live equivalent by content search	Fully inert: no `ClusterRoleBinding` anywhere in the chart references it, so it grants nothing to anyone. Its one rule (`nonResourceURLs: [/metrics], verbs: [get]`) is also already present verbatim inside the 44-rule manager ClusterRole. Vestigial kubebuilder scaffolding; harmless, pre-existing, not touched.
`openshift-gitops-operator-proxy-role.clusterrole.yaml` + `...-proxy-rolebinding.clusterrolebinding.yaml`	No live equivalent by content search	Active but fully redundant: correctly binds the manager SA, but both of its rules (`tokenreviews: create`, `subjectaccessreviews: create`) are already present verbatim inside the 44-rule manager ClusterRole via the same binding. Leftover from the classic kubebuilder "kube-rbac-proxy sidecar" scaffold pattern; the real Deployment has no such sidecar container. Harmless duplicate permissions, pre-existing, not touched.
`openshift-gitops-aggregate-admin.clusterrole.yaml`	No live equivalent by content search	Deliberate, narrower hand-maintained analog of OLM's own mechanism. OLM auto-generates admin/edit/view aggregated `ClusterRole`s per owned CRD (e.g. `experiments.argoproj.io-v1alpha1-admin`, labeled `rbac.authorization.k8s.io/aggregate-to-admin: "true"` + `olm.opgroup.permissions/aggregate-to-<hash>-admin: "true"`) — one triplet per CRD, ~42 ClusterRoles total for this operator's 14 CRDs. The chart instead hand-picks just 3 "core" end-user-facing types (`argocds`, `appprojects`, `applications`) with full `*` verbs under one ClusterRole. This is a pre-existing, intentional simplification (not every CRD type needs end-user self-service access) — expanding it to match OLM's full per-CRD generation would be a scope-creep feature change, not a version-bump fix.
`values.yaml`	N/A	Helm chart's own user-facing schema (image overrides, proxy config) — not derived from source, nothing to diff

Also worth knowing when comparing RBAC: on the live hub, OLM auto-promotes the namespaced permissions[0] Role into an additional, separately-hashed cluster-scoped ClusterRole (same 3 rules — configmaps, leases, events — plus one extra namespaces: get/list/watch rule), as part of wiring an AllNamespaces-mode operator. This is OLM's own internal machinery for that install mode, not a distinct RBAC item declared in the CSV's install.spec — don't chase it as something to replicate. In any case the chart's manager ClusterRole already grants a superset (namespaces with full CRUD verbs, not just get/list/watch), so there's no functional gap.

Verify target version: kubectl get csv -n openshift-gitops-operator | grep openshift-gitops
Extract CRDs: kubectl get crd <name> -o yaml, strip runtime metadata (uid, resourceVersion, creationTimestamp, generation, managedFields, OLM annotations/labels, status), disable conversion webhooks (strategy: None, remove webhook block), escape Go templates ({{ }} → {{ "{{" }}). Place in gitopsaddon/charts/openshift-gitops-operator/templates/crds/ ONLY. deploy/crds/ is unrelated.
Update manager ClusterRole: Extract from CSV spec.install.spec.clusterPermissions[0].rules. Quote all * wildcards as "*" — unquoted * is a YAML alias. Validate: python3 -c "import yaml; yaml.safe_load(open('file'))".
Update leader-election Role: Extract from CSV spec.install.spec.permissions[0].rules. Compare with templates/openshift-gitops-operator-leader-election-role.role.yaml.
Update image SHAs: Extract from CSV env vars (non-RELATED_IMAGE_ ones) and container image spec. Update pkg/utils/config.go DefaultOperatorImages. Also check for new/removed env var names.
Update Deployment template: Compare CSV deployment spec with templates/openshift-gitops-operator-controller-manager.deployment.yaml. Check for: new env vars, changed args, new ports, changed probes, new volumes/mounts. The chart should match the source except for these intentional differences:
- ENABLE_CONVERSION_WEBHOOK=false (source=true) — no OLM conversion webhook on non-OCP
- DISABLE_DEFAULT_ARGOCD_INSTANCE=true (source=false on OCP) — embedded chart disables default instance
- --metrics-bind-address=127.0.0.1:8080 (source=:8443) — kept as-is to avoid behavior change (metrics go from localhost-only to externally accessible); changing would also require updating the ConfigMap
- seccompProfile.type: RuntimeDefault (source doesn't have it) — extra security hardening on non-OCP, kept for defense-in-depth
- ARGOCD_CLUSTER_CONFIG_NAMESPACES="*" (source=openshift-gitops) — chart uses wildcard so all namespaces get cluster-scoped RBAC
- --metrics-secure=true + the metrics-certs volume/volumeMount (source has both, chart has neither) — this pairs with the source's --metrics-bind-address=:8443 to serve metrics over TLS using a cert from the OLM-provisioned openshift-gitops-operator-metrics-tls secret (backed by the service.beta.openshift.io/serving-cert-secret-name annotation on the source's metrics Service, an OCP/OLM-only mechanism). Since the chart already deliberately keeps metrics on 127.0.0.1:8080 (plaintext, loopback-only) instead of matching the source, adding --metrics-secure=true without that cert-issuance mechanism would break the operator (missing cert files) — so both are intentionally omitted together.
Two env vars are easy to misjudge as "changed" when they haven't:
- DISABLE_DEFAULT_ARGOCD_INSTANCE genuinely does not appear at all in the source CSV's env list on OCP — it's not "removed," it's simply never set there, which defaults to false (create the default instance). Its absence from a freshly-extracted CSV env dump is the expected, unchanged state, not evidence the flag was deprecated — confirmed still read by the operator's own source (common/common.go) as of this writing. The chart must keep explicitly forcing it to true.
- CLUSTER_SCOPED_ARGO_ROLLOUTS_NAMESPACES is a real source env var (unlike the two above, it is NOT in the intentional-differences list — the chart must match its source value verbatim, currently openshift-gitops). Check its value on every bump like any other non-image env var; don't assume because it's rollout-related that it follows the same wildcard treatment as ARGOCD_CLUSTER_CONFIG_NAMESPACES.
For CRDs and RBAC rule sets specifically, a scripted structural diff (parse both sides as YAML, walk them recursively, ignore key/list order, treat rules as sets of (apiGroups,resources,verbs,resourceNames) tuples) is far more reliable than eyeballing a diff on files that can run past 1-2MB — a manual read easily misses a single differing field buried in a huge schema. Un-escape the chart's {{ "{{" }}/{{ "}}" }} Go-template escaping back to {{/}} before parsing its copy as YAML.
Bump Chart.yaml version
Check new/removed resources: Go file-by-file through the complete file inventory table above — every template file against its live hub equivalent, bidirectionally (something new on the hub with no chart file = possible addition needed; a chart file with no live equivalent and no clear rationale = possible removal candidate). Don't stop at "does the chart have a plausibly-named file" — confirm by content, using kubectl get <kind> -n openshift-gitops-operator -o yaml (or cluster-scoped, unnamespaced for ClusterRoles/ClusterRoleBindings) as the source, since OLM hashes/renames things. The OLM bundle also carries monitoring resources (ServiceMonitor, prometheus Role/RoleBinding, bearer-token Secret) that are genuinely NOT needed in the embedded chart (no Prometheus scraping non-OCP clusters via this path).
Cross-reference with orb: Extract the bundle independently and verify CRD count, image SHAs, and ClusterRole rules match (see orb section below).
Cross-reference with source repos for Go-level behavior only: Check gitops-operator and argocd-operator source (e.g. controllers/util/util.go's InspectCluster/VerifyAPI) for API-discovery behavior changes (e.g., how Route/Monitoring APIs are handled on non-OCP). Do NOT use these repos' checked-in bundle/manifests/ for CSV/RBAC/resource content — see the staleness warning above; the live hub CSV is the only trustworthy source for that.
Build and test: make build && make test, then push a locally-built image to a registry you control, redeploy it on the hub (see "Build / Push / Redeploy Workflow"), and run test-cycle.sh for at least 10 cycles with both OCP and Kind clusters. Confirming a pure image-digest bump this way still matters even when the CRD/RBAC/Deployment diffs all come back clean — running test-cycle.sh against the OCP cluster alone is not sufficient coverage for a manifest bump: the non-OCP path installs via the embedded chart, a materially different code path (OLM install vs templateAndApplyChart) that can only be exercised with a Kind (or other non-OCP) cluster in the MANAGED_CLUSTERS list. Confirmed live: an OCP-only run can pass every cycle cleanly while the Kind path still needs its own dedicated pass to prove the new image digests are actually pullable and the operator reaches Available there. Also confirmed live: running OCP and Kind together in the same cycle exercises the principal harder (more concurrent agents/events) and makes the pre-existing "Principal event-relay can get transiently stuck on one stale event" timing race (see "Real-infra timing behaviors" below) noticeably more likely to surface within a single test-cycle.sh run than an OCP-only run does — this is the same already-documented, self-healing characteristic, not something a manifest bump introduces, but don't be surprised if a combined run hits it where a single-cluster run didn't.

orb Tool (Verification)

go install github.com/joelanford/orb/cmd/orb@latest. Requires pkg-config and libgpgme-dev system packages. Requires DOCKER_CONFIG with registry.redhat.io auth. Extract bundle: DOCKER_CONFIG=/tmp/orb-auth orb bundle convert plain docker://registry.redhat.io/openshift-gitops-1/gitops-operator-bundle:vX.Y.Z dir:/tmp/orb-output -n openshift-gitops-operator. The output manifests.yaml contains all CRDs and non-CRD resources. Compare CRD names, image SHAs, and ClusterRole rules against the chart. The OLM bundle uses hashed ClusterRole names and combines RBAC differently — compare rule CONTENT, not names.

This is optional, independent cross-verification on top of the live-hub-CSV comparison in step 1-10 above (which remains the primary, sufficient source of truth per "Verification direction" above) — skip it if orb isn't installed or registry.redhat.io credentials aren't configured in the current environment, rather than blocking the rest of the procedure on it. A bump has been fully verified end-to-end (all CRDs/RBAC/Deployment diffed against a live hub, images updated, make build && make test green, test-cycle.sh green on both OCP and Kind) without ever running orb.

CI Pipeline

GitHub Actions (.github/workflows/e2e.yml): e2e (cluster-import), e2e-gitopsaddon (embedded, embedded-agent, olm-override, embedded-autonomous). Local mirror: make test-local-e2e-gitopsaddon-*.

E2E Testing Gotchas

CI timing: ArgoCD sync is slow on CI runners. Don't assert Synced with tight timeouts.
test-local-* vs test-e2e-*: Local creates fresh Kind clusters. Always use local to mirror CI.
Principal container naming: Red Hat operator: "principal". Upstream: deployment name. findContainerImage handles both.
Helm {{ in CRDs: Escape as {{ "{{" }}. Check imageupdaters CRD.
Kubeconfig contamination: Unset KUBECONFIG before make test-local-e2e-* or Kind writes into hub kubeconfig.
appProjectYAML must include name: '*': Without it, principal skips AppProject propagation → agents have no AppProject → apps stuck at sync=Unknown.
sync=Unknown health=Healthy: Principal never dispatched. Check: (1) app on spoke? (2) AppProject exists? (3) agent pod logs for gRPC errors.
argocd-agent-resource-proxy-tls bootstrap: Principal FATAL if missing. Early cert generation block in controller generates before VerifyArgocdNamespace.
Principal TLS NodePort SAN: getPrincipalHostNames adds node IPs when no LoadBalancer ingress exists.
UID mismatch after cleanup: Rapid delete/recreate causes stale informer cache. isObjectGoneError() aborts reconcile. Fix: restart controller + delete lease.
OperatorGroup must be AllNamespaces: OwnNamespace causes CSV failure.
CSR accumulation: Rapid deploy/delete cycles exhaust CSR quota. Fix: kubectl get csr -o name | grep gitops-addon | xargs kubectl delete.
ACM 2.16→2.17 CRB roleRef immutability: deleteStaleClusterRoleBinding() self-heals at startup.
Addon agent race during cleanup: scaleDownAddonAgent() scales to 0 before cleanup Job runs.
local-cluster guestbook (hybrid mode): local-cluster is never agent-dispatched, so there is no hub-vs-spoke identity conflict to work around. Its Application is verified and finalized normally through the hub's own application controller, same as any other in-cluster ArgoCD destination - see "Deploy/Delete Cycle Testing" below.
A single momentary "Running" or "Synced" sample is not proof of health: a crash-looping container can genuinely report status.phase: Running for the brief window between container start and a fatal exit, long enough for a 5-10s poll to sample mid-flicker and report a false pass. Require 2 consecutive stable samples (same phase AND same restart count) before declaring a pod healthy, not a single Eventually(...).Should(Equal("Running")). The same logic applies to any other "did it work" snapshot check - prefer confirming it stayed true, not that it was observed true once.
Test the actual function, not a proxy for it: for anything that reports status hub-side after a spoke-side action (agent-dispatched apps, autonomous mirrors), the strongest and simplest check is "did the hub receive a real update from the spoke at all" (non-empty, not stuck at the default "Unknown"), not "does the hub's value exactly equal the spoke's value right now" and not "does a specific log string never appear." Exact-match chases a moving target - the spoke and hub progress independently, and a strict-equality check can fail forever if the spoke advances to a new state before the hub catches up, even though the reporting pipeline is working fine. A broken pipeline naturally fails the "got any real update" check on its own; you don't need a separate implementation-specific log-grep for a known error string to catch it, and log-grepping is more fragile (ties the test to exact wording) than asserting on the real observable behavior.
Don't gate a coarse readiness check on an unreachable end state: e.g. requiring health: Healthy on the demo guestbook app before considering a deploy "done" - it structurally can't reach Healthy on OCP (the demo image runs as non-root binding port 80, which restricted SCC rejects), so that gate would time out on every run regardless of whether the actual thing under test works. Gate on something the environment can actually achieve (e.g. "has a real, non-Unknown sync status"), consistent with how the same known limitation is already tolerated (WARNING, not failure) elsewhere in the same test.
Nudge instead of waiting out a timer: when something is stuck because a component's next periodic action (a resync, a refresh) will naturally clear it, actively triggering that action (e.g. the argocd.argoproj.io/refresh=normal annotation) resolves it far faster than passively waiting out the interval. Worth doing partway through a retry loop (e.g. attempt 3-6) rather than only at the very start or not at all.

Deploy/Delete Cycle Testing

gitopsaddon/test-cycle.sh — automated deploy/verify/delete cycling against real clusters (not the Kind-only make test-local-e2e-* targets). This is the primary way to validate hybrid mode end-to-end: it always includes local-cluster in the ApplicationSet target (via a dedicated cycle-agent-appset-placement, kept separate from the addon-install cycle-agent-placement which still excludes local-cluster), verifies the cluster-local-cluster secret shape and that the generated app reconciles via the hub's own application controller, and tears everything down with no manual finalizer stripping anywhere in the script: it deletes the ApplicationSet first (per the "Cleanup (Manual Deletion Order)" section above), while agents and the hub app-controller are still connected, and waits for every generated Application — spoke-managed and local-cluster alike — to finalize naturally before touching GitOpsCluster/Policy/MCAs.

HUB_KUBECONFIG=/path/to/hub \
MANAGED_CLUSTERS="eks-cluster1:/path/to/eks.kc,ocp-cluster-name:/path/to/ocp.kc" \
  bash gitopsaddon/test-cycle.sh 5

Optional env vars: FORCE_CLEAN_FIRST (true), WAIT_DEPLOY_SECS (600), WAIT_CLEANUP_SECS (600), HUB_CONTROLLER_NS (ocm).

Prerequisite (one-time, per hub): the hub's own ArgoCD app-controller ServiceAccount needs elevated RBAC to manage the local-cluster hybrid app's destination namespace — see "Hybrid Mode" above. test-cycle.sh creates this itself (ensure_hub_appcontroller_rbac) so no manual step is needed for the script specifically, but real (non-test) hybrid-mode setups must do this too.

Building/pushing images from a Flatpak-sandboxed shell (e.g. this coding agent's Bash tool)

If docker/podman/kind/go aren't visible in the shell but ARE installed on the real host, the shell is likely running inside a Flatpak sandbox layer (check cat /etc/os-release — a Freedesktop SDK ... (Flatpak runtime) result confirms it). Escape it with flatpak-spawn --host <command> to run the command on the real host instead — this reaches the host's actual podman, kind, and go installs, registry credentials, and any existing local Kind clusters. Plain kubectl against real remote clusters typically works fine without the escape hatch (outbound network is already shared); use flatpak-spawn --host for anything that needs the host's container/VM tooling specifically.

Real-infra timing behaviors worth knowing before tuning timeouts

Observed running many test-cycle.sh cycles against real (AWS-hosted OCP + non-OCP) clusters — these are genuine, self-healing characteristics of the live system, not bugs to "fix" by weakening a check, but also not things a short timeout can paper over:

Principal event-relay can get transiently stuck on one stale event. The argocd-agent principal's event-processor can retry one superseded status-update event against a resourceVersion the hub has already moved past, logging "the server rejected our request due to an error in our request" repeatedly with exponential backoff, and eventually "Event failed after N retries, giving up to unblock queue". It self-heals once the agent's own next periodic resync (ArgoCD default 180s) sends a fresh event, or its own retry budget exhausts and it moves on - either way this can take several minutes end-to-end. A short (~1-2 min) timeout on "did the hub get a status update" will false-fail here even though nothing is actually broken; budget ~400s and nudge (see above) to shorten the common case.
A freshly-started agent can drop its own managed-app state while still stabilizing. During the already-documented CA-secret-propagation crash-loop window (see "Principal CrashLoopBackOff"), an agent that's restarting can lose track of an app it was managing on a reconnect/resync ("Dropping app deletion event because the app is not managed" / "could not delete existing app: ... is not managed"), and that app is not automatically redispatched once the agent stabilizes - something has to re-trigger dispatch (e.g. a principal restart, or the next full reconcile). Waiting for the agent pod to be stable (2 consecutive Running+Ready samples with an unchanged restart count, not just "Running once") before checking application status avoids racing this window in the first place.
FORCE_CLEAN_FIRST-style cleanup needs a principal restart too, not just a controller restart. Restarting the hub controller clears its own informer cache, but the principal separately needs a restart to drop stale gRPC events queued from whatever was just torn down - otherwise the next cycle's fresh agents can receive stale delete/update events and reject them as "not managed" (same failure mode as the documented "between cycles" reset, just also needed before the first cycle after a forced clean).

Development Workflow

Go 1.26+ (see go.mod), Kubernetes cluster with OCM
make test before submitting
make generate after API type changes
make manifests after RBAC/CRD annotation changes
Update CLAUDE.md after learning new context
Ask before acting on ambiguous tasks

Never pin a dev-tool go install to @latest in the Makefile. setup-envtest (used by make test) was pinned to @latest and broke with requires go >= 1.26.0 (running go 1.25.11) the moment a newer release shipped upstream - with no code change on this repo's side. Pin to a specific version (SETUP_ENVTEST_VERSION in Makefile/Makefile.prow) compatible with this repo's own go.mod/controller-runtime version, and key the "is it already installed" check on the actual installed version ($(TOOL) version output), not just "does the binary file exist" - otherwise bumping the pinned version later silently keeps using a stale cached binary from the old one.

Known Limitations

Policy is user-owned: Controller recreates deleted Policies (unless skip-argocd-policy), and reconciles agent connection fields (address, port, mode, DBM, TLS) on the ArgoCD template. It never writes or clears argoCDAgent.agent.image — users who want to pin the agent image via Policy can do so. Agent-image auto-heal is ADC-only (ARGOCD_AGENT_IMAGE); disable it with skip-agent-version-heal.
Cleanup order matters: GitOpsCluster → Policy → MCAs (strict order or resources get recreated).
GitOpsCluster deletion is a no-op: No finalizer, no cascade.
local-cluster must stay out of the GitOpsCluster's own placementRef: it's never an addon-install target (hybrid mode) — it already has its own ArgoCD instance. A separate Placement, used only by ApplicationSets, may include it.
OLM mode OCP only: Non-OCP uses embedded Helm chart.
RBAC before first app sync: ArgoCD retry limit (5) means stuck apps if RBAC missing. Delete spoke app copies to recover. This applies to the hub's own app-controller too for any app targeting local-cluster.
Non-OCP pull secret chain: multiclusterhub-operator-pull-secret must have registry.redhat.io in BOTH namespaces.
OCP OLM timing: 30-120s. Use 600s timeouts.
DBM must match principal and agent: Mismatched = "Resource not found" in UI.
Agent SA view ClusterRole: Not auto-provisioned. Add manually or via Policy.
Basic pull model is OCP-only: not validated/supported against non-OCP (embedded Helm chart) managed clusters. See "Basic Pull Model" under Architecture.
Basic pull model on local-cluster is dormant: the dedicated local-cluster-namespace ArgoCD instance it depended on was removed by hybrid mode; no code-level guard blocks it, but it is unverified. See "Basic Pull Model" under Architecture.
Basic pull model status aggregation requires an ApplicationSet owner: a hand-created Application with the pull label/annotation still gets delivered and reconciled, but produces no MulticlusterApplicationSetReport — only ApplicationSet-generated Applications get the application-set label needed for multiclusterstatusaggregation to pick them up.
Never delete a pull-model ManifestWork for an app you want to keep: proven destructive in live testing regardless of updateStrategy: ReadOnly or an explicit orphaningRules entry — see "Migrating an App from Basic Pull Model to argocd-agent" above for the actual safe sequence.

Container Registry

Default: quay.io/stolostron. Configure via REGISTRY and VERSION make variables (make REGISTRY=... VERSION=... — these are simple = assignments in the Makefile, so exporting them as shell env vars alone won't override the default; pass them on the make command line).

FilesExpand file tree

CLAUDE.md

Latest commit

History

CLAUDE.md

File metadata and controls

CLAUDE.md

Quick Reference — Do These First or Waste Hours

deploy/crds/ Is NOT for Operator Version Bumps

Pull Secret Chain (Non-OCP clusters WILL fail without this)

Agent Mode Setup Order (skip a step = broken)

Kind Cluster Import Order

Expected Waits (Not Bugs)

Repository Overview

Common Commands

Agent Mode Setup — Full YAML

Basic Pull Model Setup — Full YAML

Disabling the Basic Pull Model (multicluster-integrations-config)

Migrating an App from Basic Pull Model to argocd-agent

The verified sequence: transform the existing ApplicationSet's template in place

Designs that were rejected (and why)

Other gotchas hit while building and testing this

Creating and Importing a Kind Managed Cluster

Cleanup (Manual Deletion Order)

Troubleshooting

Apps Stuck at OutOfSync/Missing After RBAC Fix

Image Pull Issues on Non-OCP Clusters

Principal CrashLoopBackOff

Policy Not Compliant

ManagedClusterAddOn Stuck Deleting Forever (addon-pre-delete finalizer)

Hub Controller UID Mismatch

ArgoCD UI Live Manifest "Resource not found"

Hybrid Mode: local-cluster's ArgoCD UI Resource-Tree/Manifest Drill-Down Is Broken (upstream argocd-agent bug, not fixed here yet)

Agent Not Connecting

Basic Pull Model: Blank Cluster Secret Creation Fails ("found existing non-ACM ArgoCD clusters secrets")

kubectl get application Silently Returns Nothing on an ACM Hub

Hub App-Controller Fights the Principal Over an Autonomous Mirror's Status

Architecture

Hub Controller ↔ Addon Agent

OCP vs Non-OCP Operator Version Gap (Critical)

Template Modes

Agent Modes

Hybrid Mode (hub app-controller + argocd-agent principal, same instance)

Basic Pull Model (Classic ManifestWork-Based Delivery)

Package Structure

Key Annotations on GitOpsCluster

Key Annotations on ManagedClusterImageRegistry

Key Features (Deep Reference)

Agent Version Drift Heal

Disconnected Image Mirroring (ManagedClusterImageRegistry)

Cert Rotation & Autorefresh

Resource Proxy TLS SAN

Cluster Secret Server URL

Skip-Reconcile Annotation on Agent Cluster Secrets

Non-Agent → Agent Mode Transition

OLM Subscription Details

Addon RBAC

No-Op-Update Guards on Hub-Owned Objects (AddOnTemplate, AddOnDeploymentConfig)

Cleanup Rules

Custom ArgoCD Namespace

Image Pull Secret Chain (Non-OCP)

Build / Push / Redeploy Workflow

Manifest Update Procedure (OpenShift GitOps Operator)

Verification direction: SOURCE FIRST, then chart

What to update (complete checklist)

orb Tool (Verification)

CI Pipeline

E2E Testing Gotchas

Deploy/Delete Cycle Testing

Building/pushing images from a Flatpak-sandboxed shell (e.g. this coding agent's Bash tool)

Real-infra timing behaviors worth knowing before tuning timeouts

Development Workflow

Known Limitations

Container Registry

`deploy/crds/` Is NOT for Operator Version Bumps

`kubectl get application` Silently Returns Nothing on an ACM Hub