Repository navigation
sync: incubation to stable - #229
Merged
Merged
Conversation
* feat(tls): add proxy profile argument resolver Signed-off-by: Rui Vieira <ruidevieira@googlemail.com> * feat(tls): wire proxy profile arguments into workloads Signed-off-by: Rui Vieira <ruidevieira@googlemail.com> * fix(tls): retain cipher suites above minimum version * docs(tls): explain adherence fallback * test(e2e): wait for TAS deployment * test(e2e): provide bound TAS test storage * test(e2e): avoid asserting synthetic TAS deployment --------- Signed-off-by: Rui Vieira <ruidevieira@googlemail.com>
…-explainability#943) * feat(evalhub): support system collection curation overrides Render typed curation order overrides from the EvalHub CR into per-instance system collection ConfigMaps.\n\nAssisted-by: Codex Signed-off-by: Lior Aronovich <lioraronpr@gmail.com> * fix(evalhub): validate collection overrides consistently Ensure invalid and tenant collection overrides cannot be silently accepted. Assisted-by: Codex Signed-off-by: Lior Aronovich <lioraronpr@gmail.com> * fix(evalhub): limit overrides to v1 Keep collection overrides out of the unsupported v1alpha1 API and conversion path. Assisted-by: Codex Signed-off-by: Lior Aronovich <lioraronpr@gmail.com> --------- Signed-off-by: Lior Aronovich <lioraronpr@gmail.com>
* feat(tls): wire proxy profile arguments into workloads Signed-off-by: Rui Vieira <ruidevieira@googlemail.com> * test(e2e): wait for TAS deployment * test(e2e): provide bound TAS test storage * test(e2e): avoid asserting synthetic TAS deployment * feat(tls): refresh workloads on profile adherence changes * fix(tls): retain proxy args on transient profile fetch failures Signed-off-by: Rui Vieira <ruidevieira@googlemail.com> --------- Signed-off-by: Rui Vieira <ruidevieira@googlemail.com>
* feat(tls): wire proxy profile arguments into workloads Signed-off-by: Rui Vieira <ruidevieira@googlemail.com> * test(e2e): wait for TAS deployment * test(e2e): provide bound TAS test storage * test(e2e): avoid asserting synthetic TAS deployment * fix(tls): read adherence and detect FIPS mode Signed-off-by: Rui Vieira <ruidevieira@googlemail.com> --------- Signed-off-by: Rui Vieira <ruidevieira@googlemail.com>
…evalhub-inspect-primary-score chore(evalhub): sync inspect provider changes
[pull] main from trustyai-explainability:main
The MCP Gateway integration created a dependency on Inference Payload Processing's (IPP) NeMo BBR plugins which will soon be replaced. If `MCPGateway` is defined on the `NemoGuardrailsPodTemplate` without a custom deployment of the BBR plugin, it will cause theNeMo Guardrails server deployment to fail. Fix: remove `controllers/nemo_guardrails/envoy_filter.go`, `controllers/nemo_guardrails/mcp_gateway.go`, `controllers/nemo_guardrails/mcp_gateway_test.go`, `controllers/nemo_guardrails/templates/envoy-filter.tmpl.yaml` and related changes on other files. This reverts commit 9b9abea.
…inability#969) Assisted-by: Codex Signed-off-by: Lior Aronovich <lioraronpr@gmail.com>
…explainability#971) * fix(evalhub): detect FailedMount as evaluation job failure * fix(evalhub): ignore stale FailedMount events Require the newest FailedMount/FailedAttachVolume Event to be within the scheduling grace period so a transient mount warning cannot fail a job that later succeeds during a slow image pull. Assisted-by: Cursor Signed-off-by: Sobha Cheruku <scheruku@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com> * fix(evalhub): bounded requeue for late FailedMount events Keep rechecking volume-mount-stuck pods until volumeMountWatchMaxAge so a FailedMount that arrives after the first post-grace check is still reported. Assisted-by: Cursor Signed-off-by: Sobha Cheruku <scheruku@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com> * fix(evalhub): keep events list in module RBAC markers Add list to the trustyai-operator-module kubebuilder events markers so make manifests-tom does not drop the permission added for FailedMount detection. Assisted-by: Cursor Signed-off-by: Sobha Cheruku <scheruku@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com> --------- Signed-off-by: Sobha Cheruku <scheruku@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
[pull] main from trustyai-explainability:main
…explainability#902) * chore(gorch): Remove GuardrailsOrchestrator ecosystem from operator Signed-off-by: Rob Geada <rob@geada.net> * fix: module sync Signed-off-by: Rob Geada <rob@geada.net> * fix: image substitution on module kustomize Signed-off-by: Rob Geada <rob@geada.net> * fix: Add handling of removed/legacy services from operator main.go Signed-off-by: Rob Geada <rob@geada.net> * fix: Fix legacy service support, revert erroneous manifest changes Signed-off-by: Rob Geada <rob@geada.net> --------- Signed-off-by: Rob Geada <rob@geada.net>
…xplainability#975) * bump(evalhub): update GuideLLM to v0.6.1 (RHOAIENG-95788) Signed-off-by: Jooyeon Mok <jmok@redhat.com> * fix(evalhub): sync GuideLLM image in module templates (RHOAIENG-95788) Signed-off-by: Jooyeon Mok <jmok@redhat.com> --------- Signed-off-by: Jooyeon Mok <jmok@redhat.com>
…lity#917) Signed-off-by: Jooyeon Mok <jmok@redhat.com>
Signed-off-by: Jooyeon Mok <jmok@redhat.com>
…explainability#902) * chore(gorch): Remove GuardrailsOrchestrator ecosystem from operator Signed-off-by: Rob Geada <rob@geada.net> * fix: module sync Signed-off-by: Rob Geada <rob@geada.net> * fix: image substitution on module kustomize Signed-off-by: Rob Geada <rob@geada.net> * fix: Add handling of removed/legacy services from operator main.go Signed-off-by: Rob Geada <rob@geada.net> * fix: Fix legacy service support, revert erroneous manifest changes Signed-off-by: Rob Geada <rob@geada.net> --------- Signed-off-by: Rob Geada <rob@geada.net>
…xplainability#975) * bump(evalhub): update GuideLLM to v0.6.1 (RHOAIENG-95788) Signed-off-by: Jooyeon Mok <jmok@redhat.com> * fix(evalhub): sync GuideLLM image in module templates (RHOAIENG-95788) Signed-off-by: Jooyeon Mok <jmok@redhat.com> --------- Signed-off-by: Jooyeon Mok <jmok@redhat.com>
…lity#917) Signed-off-by: Jooyeon Mok <jmok@redhat.com>
trustyai-explainability#991) Add the IFBench image parameter to the base kustomization. Update IFBench and LightEval images to v0.6.3 across base, ODH, and RHOAI configurations and mirrored manifest templates. Includes the IFBench non-root NLTK fix and LightEval per-example result export support. Validation: ODH and RHOAI kustomize builds and git diff --check passed. Signed-off-by: Jooyeon Mok <jmok@redhat.com>
trustyai-explainability#991) Add the IFBench image parameter to the base kustomization. Update IFBench and LightEval images to v0.6.3 across base, ODH, and RHOAI configurations and mirrored manifest templates. Includes the IFBench non-root NLTK fix and LightEval per-example result export support. Validation: ODH and RHOAI kustomize builds and git diff --check passed. Signed-off-by: Jooyeon Mok <jmok@redhat.com> (cherry picked from commit 2b7c396)
Signed-off-by: Julian Payne <julpayne@redhat.com>
…in-to-odh sync: main with trustyai-explainability:main
…explainability#981) * feat(networkpolicy): add shared operand policy foundations Add typed egress validation, collision-safe workload identity, authority and ownership guards, complete-set reconciliation/status reporting, and UID-precondition cleanup. Include fake-client and local API serialization/race tests and integration/evidence documentation. Refs trustyai-explainability#956 Signed-off-by: Rui Vieira <ruidevieira@googlemail.com> * fix(networkpolicy): accept older owner reference versions * feat(networkpolicy): authorize reviewed namespace-only ingress peers Signed-off-by: Rui Vieira <ruidevieira@googlemail.com> * fix(networkpolicy): accept positive Exists namespace selectors Signed-off-by: Rui Vieira <ruidevieira@googlemail.com> * docs(networkpolicy): clarify positive namespace selectors Signed-off-by: Rui Vieira <ruidevieira@googlemail.com> * fix(networkpolicy): require target selector to match identity * fix(networkpolicy): reject universal namespace selectors * fix(networkpolicy): resolve owner GVK from scheme * fix(networkpolicy): canonicalize namespace selectors * fix(networkpolicy): avoid mutating owner TypeMeta * docs(networkpolicy): attach reconcile godoc correctly --------- Signed-off-by: Rui Vieira <ruidevieira@googlemail.com>
…ute when auth==true Signed-off-by: Christina Xu <chrxu@redhat.com>
Signed-off-by: Jooyeon Mok <jmok@redhat.com>
Signed-off-by: Jooyeon Mok <jmok@redhat.com>
…lhub-collection-classification fix(evalhub): sync collection classification metadata
trustyai-explainability#995) * feat(nemo): segregate NeMo admin endpoints into separate auth provider Signed-off-by: Rob Geada <rob@geada.net> * feat: NeMo Admin auth address review comments Signed-off-by: Rob Geada <rob@geada.net> * feat: NeMo Admin auth address review comments Signed-off-by: Rob Geada <rob@geada.net> * fix kube-rbac owner service name Signed-off-by: Rob Geada <rob@geada.net> --------- Signed-off-by: Rob Geada <rob@geada.net>
…1006) Bumps golang.org/x/net from v0.58.0 to v0.60.0 in the main module and in the trustyai-operator-module subproject, so both resolve the same x/net version and the repo-wide fs security scan stays green. Only the transitive golang.org/x family moves with it (crypto, sys, term, text, mod, sync). No code changes.
… main (trustyai-explainability#1015) * ci(gitleaks): allowlist generic-api-key false positive in gorch tls.go The Gitleaks workflow scans full history and fails on every branch and PR with a single generic-api-key finding at controllers/gorch/tls.go:13 (introduced in f7cb3c7; the file was since removed from main in 7f6270c): utils.MountSecret(deployment, tlsSecretToMount, tlsSecretToMount+"-vol", "/etc/tls/"+tlsSecretToMount, []int{0}) tlsSecretToMount is a Go variable holding a Kubernetes Secret name, not a secret value, so this is a false positive. Since the file no longer exists at the tip, an inline gitleaks:allow comment cannot suppress the historical finding; instead add a root .gitleaks.toml that keeps the default rules ([extend] useDefault = true) and allowlists the path. Verified with the workflow's pinned gitleaks 8.28.0 against a fresh clone of this repo: baseline reports the tls.go finding; with the config, gitleaks reports no leaks. * test(lmes): budget watch-driven drift repair above suite default The "repairs policy deletion/drift and Pod identity drift for a retained completed job" spec flakes in CI: its Eventually assertions for Pod relabeling, policy spec repair and policy recreation share the suite's 10s defaultTimeout, but under runner load (five envtest suites share one machine) informer event delivery can lag direct-client calls by more than 10s. In the 2026-10-14 main run (job 116020699805) the last reconcile before the failure was at 18:15:04 and the next pair arrived at 18:15:14 on the 10s PodCheckingInterval requeue; the watch events never surfaced in between while the spec's own client polls kept succeeding, and the repair landed right at the 10.001s deadline. Budget those three assertions with a spec-scoped watchRepairTimeout of 30s, matching the re-run path's existing 30s budget in this suite. Direct-client waits keep the suite default. The controller itself repairs correctly; only the wait budget is too tight under load. Closes trustyai-explainability#1011 Signed-off-by: Rui Vieira <ruidevieira@googlemail.com> * ci(test): run controller test packages serially on the runner The widened 30s budget was not enough on its own: this PR's first CI run (38660769323) failed the same spec with zero reconciles for the job during the whole 30s window. With go test ./... defaulting to -p GOMAXPROCS (4 on the runner), four envtest suites - each with its own kube-apiserver, etcd and manager - run concurrently on the runner's shared vCPUs. In both observed failures the apiserver kept serving the spec's direct client reads while watch event delivery stalled: ~10s on the Oct 14 main run (job 116020699805, repair landed just past the deadline) and 30s+ on the PR run. Completed LMEvalJobs have no requeue backstop (handleComplete drops the pullingJobs key), so watch delivery is the spec's only repair trigger, and an informer stall means an unbounded wait whatever the Eventually budget. Run the test packages serially (-p 1) in the controller-tests workflow so each suite's apiserver gets the runner to itself; keep the 30s watch-repair budget from the previous commit as defense in depth for the runner's residual scheduling noise. Local serial run of the full suite: 16/16 packages pass in 2m53s. Signed-off-by: Rui Vieira <ruidevieira@googlemail.com> * ci(gitleaks): suppress the false positive by fingerprint, not path Review feedback on trustyai-explainability#1015: the .gitleaks.toml path allowlist for controllers/gorch/tls.go would also mask any future real finding in that path. Replace it with a .gitleaksignore entry pinned to the exact finding (commit f7cb3c7, file, rule, start line), so only the known historical false positive is suppressed. Note the fingerprint's last component is the finding's start line (13): the scan report redacts it to "***", so a fingerprint copy-pasted from the report never matches. Verified with the workflow-pinned gitleaks 8.28.0 against a fresh clone of main at 258b3a9: the false positive is suppressed, a new secret planted at the same path is still reported, and the ignore file itself is not flagged. Signed-off-by: Rui Vieira <ruidevieira@googlemail.com> * fix: Update .gitleaksignore Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> --------- Signed-off-by: Rui Vieira <ruidevieira@googlemail.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
… with trustworthiness domains - Introduced a new ConfigMap for the MTEB provider with comprehensive benchmark configurations. - Updated the Nemo Guardrails provider to include trustworthiness domains and evaluation targets for various safety tasks. - Enhanced the RAGAS provider by adding grounded document understanding domains and relevant tasks. - Modified the Ruler provider to incorporate grounded document understanding domains and long context understanding tasks across multiple metrics. Signed-off-by: Julian Payne <julpayne@redhat.com>
Signed-off-by: Julian Payne <julpayne@redhat.com>
Signed-off-by: Julian Payne <julpayne@redhat.com>
Signed-off-by: Julian Payne <julpayne@redhat.com>
Signed-off-by: Julian Payne <julpayne@redhat.com>
…e overlays Signed-off-by: Julian Payne <julpayne@redhat.com>
…uration Signed-off-by: Julian Payne <julpayne@redhat.com>
Signed-off-by: Julian Payne <julpayne@redhat.com>
- Increased curation_order for various collections to 10 for better organization. - Adjusted pass criteria thresholds for model validation and instruction following collections. - Updated primary metrics to reflect changes in evaluation standards, including renaming metrics for clarity. - Enhanced documentation within collection YAML files to provide clearer guidance on evaluation processes and expectations. - Consolidated implementation plan references to streamline documentation and reduce redundancy. Signed-off-by: Julian Payne <julpayne@redhat.com>
Signed-off-by: Julian Payne <julpayne@redhat.com>
Signed-off-by: Julian Payne <julpayne@redhat.com>
…ability/build/sync-providers-collections build: add MTEB provider configuration and enhance existing providers
Sync changes from upstream to midstream
…ai-explainability#1017) Upgrading from RHOAI 3.5 to 3.6 leaves the DataScienceCluster stuck: the 3.5 TrustyAI CR stored spec.eval.lmeval.permitOnline and permitCodeExecution as the strings "allow"/"deny", but the modularized 3.6 operator declares the same field paths as booleans. The updated CRD rejects the stored values, so every server-side apply against the live object fails with "expected boolean, got &{deny}". The operator cannot migrate the object itself because its informer decodes TrustyAI CRs into the typed Go struct, so the legacy strings break the cache before any reconciler could rewrite them. Restore the 3.5 schema (string enum allow/deny, default "deny") for both fields so stored 3.5 values remain valid. The internal contract is unchanged: buildDSCConfigMap still writes "true"/"false" into the DSC ConfigMap consumed by the lmes controller, derived as permit == "allow". Refs: https://redhat.atlassian.net/browse/RHAI-10010
The EvalHub API lists Kueue LocalQueues in each tenant namespace to resolve queue-backed hardware profiles for GET /api/v1/info, but no ClusterRole bound in tenant namespaces granted it, so the endpoint returned HTTP 500 (localqueues.kueue.x-k8s.io is forbidden) and queue-backed profiles never appeared. Grant exactly the list verb on kueue.x-k8s.io/localqueues in the evalhub-hardware-profiles-reader ClusterRole. The operator already binds that ClusterRole to the API service account in every tenant namespace, so the existing RoleBinding wiring picks up the new rule with no controller changes. - add (kueue.x-k8s.io, localqueues) to the conftest ClusterRole allowlist in policy/clusterrole.rego - add policy/evalhub_hardware_profiles_reader.rego enforcing LocalQueue list-only on that role, with unit tests - add a Go manifest test asserting minimal verbs in both manifest copies (config component and module manifests-template)
Signed-off-by: Jooyeon Mok <jmok@redhat.com>
…oaieng-100718-localqueues-rbac fix(evalhub): grant API LocalQueue list for tenant hardware profiles
* Add spec.allowedConsumers with from Same, Selector, or All. The CRD rejects an empty selector and a selector combined with Same or All.
…s don't publish routes Signed-off-by: Christina Xu <chrxu@redhat.com>
…spect-image-v0.6.11 fix(evalhub): bump Inspect adapter to v0.6.11
Sync changes from upstream to midstream
sync: main to incubation
ruivieira
approved these changes
Oct 15, 2026
christinaexyou
approved these changes
Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
13 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
sync-branches: New code has just landed in incubation, so let's bring stable up to speed!