argo-cd 2.3 upgrade and application-set CRD error

I upgraded argo-cd from v2.0 to v2.3 with helm

while checking deployment, I recognized that one of argo-cd pod is failing.

$ kubectl get pods -n argocd
argocd-applicationset-controller-7b98466f57-rgtx8   0/1     CrashLoopBackOff   37         4h3m
$ kubectl logs -n argocd argocd-applicationset-controller-7b98466f57-rgtx8
1.6487458483935566e+09 INFO setup ApplicationSet controller v0.4.1 using namespace 'argocd' {"namespace": "argocd", "COMMIT_ID": "a5712ba564afaff0b23697135d6281d413e60345"}
1.6487458493964987e+09 INFO controller-runtime.metrics Metrics server is starting to listen {"addr": ":8080"}
time="2022-03-31T16:57:29Z" level=info msg="Starting configmap/secret informers"
time="2022-03-31T16:57:29Z" level=info msg="Configmap/secret informer synced"
1.6487458496040509e+09 INFO setup Starting webhook server
1.648745849604233e+09 INFO setup Starting manager
1.6487458496045666e+09 INFO Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.6487458496048899e+09 INFO Starting server {"kind": "health probe", "addr": "[::]:8081"}
1.6487458497056317e+09 INFO controller.applicationset Starting EventSource {"reconciler group": "argoproj.io", "reconciler kind": "ApplicationSet", "source": "kind source: *v1alpha1.ApplicationSet"}
1.648745849705685e+09 INFO controller.applicationset Starting EventSource {"reconciler group": "argoproj.io", "reconciler kind": "ApplicationSet", "source": "kind source: *v1alpha1.Application"}
1.648745849705695e+09 INFO controller.applicationset Starting EventSource {"reconciler group": "argoproj.io", "reconciler kind": "ApplicationSet", "source": "kind source: *v1.Secret"}
1.6487458497057e+09 INFO controller.applicationset Starting Controller {"reconciler group": "argoproj.io", "reconciler kind": "ApplicationSet"}
I0331 16:57:30.744424       1 request.go:665] Waited for 1.03498312s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/node.k8s.io/v1?timeout=32s
1.6487458518972032e+09 ERROR controller-runtime.source if kind is a CRD, it should be installed before calling Start {"kind": "ApplicationSet.argoproj.io", "error": "failed to get restmapping: no matches for kind \"ApplicationSet\" in group \"argoproj.io\""}
sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/source/source.go:137
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext

The error:

ERROR controller-runtime.source if kind is a CRD, it should be installed before calling Start {"kind": "ApplicationSet.argoproj.io", "error": "failed to get restmapping: no matches for kind \"ApplicationSet\" in group \"argoproj.io\""}

The issue is here, argo-cd helm chart enables application set by default

applicationSet:  # -- Enable Application Set controller  enabled:       true

and it requires customresourcedefinition.apiextensions.k8s.io/applicationsets.argoproj.io

By design, helm does not update CRD by default. (see https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#some-caveats-and-explanations)

There is no support at this time for upgrading or deleting CRDs using Helm. This was an explicit decision after much community discussion due to the danger for unintentional data loss. Furthermore, there is currently no community consensus around how to handle CRDs and their lifecycle. As this evolves, Helm will add support for those use cases.

So we need to create CRD manually or somehow in your pipeline

$ curl https://raw.githubusercontent.com/argoproj/argo-helm/master/charts/argo-cd/crds/crd-applicationset.yaml | kubectl apply -f -

Ismail YENIGUL
Devops Engineer