Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,6 @@
/controllers/lmes/ @ruivieira
/api/lmes/ @ruivieira

# Guardrails Orchestrator
/controllers/gorch/ @RobGeada @christinaexyou @m-misiura
/api/gorch/ @RobGeada @christinaexyou @m-misiura

# NeMo Guardrails
/controllers/nemo_guardrails/ @RobGeada @m-misiura @christinaexyou
/api/nemo_guardrails/ @RobGeada @m-misiura @christinaexyou
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/operator-chaos.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,6 @@ jobs:
config/components/tas/crd
config/components/evalhub/crd
config/components/lmes/crd
config/components/gorch/crd
config/components/nemo-guardrails/crd
sparse-checkout-cone-mode: false

Expand Down
2 changes: 0 additions & 2 deletions .gitleaksignore
Original file line number Diff line number Diff line change
@@ -1,2 +0,0 @@
# Known false positive: generic-api-key detection on the TLS mount helper.
f7cb3c7ec00372121ab8c4d46a7424634f29d3b4:controllers/gorch/tls.go:generic-api-key:13
55 changes: 5 additions & 50 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

## Overview

The TrustyAI Service Operator is a Kubernetes operator that manages the lifecycle of AI/ML trust, safety, and evaluation services on OpenShift AI. It deploys and configures five distinct service types — model explainability (TAS), LLM evaluation (LMES), evaluation hub (EvalHub), guardrails orchestration (GORCH), and NeMo Guardrails — through a single operator binary with dynamic controller registration.
The TrustyAI Service Operator is a Kubernetes operator that manages the lifecycle of AI/ML trust, safety, and evaluation services on OpenShift AI. It deploys and configures fourgi distinct service types — model explainability (TAS), LLM evaluation (LMES), evaluation hub (EvalHub), and NeMo Guardrails — through a single operator binary with dynamic controller registration.

Built with kubebuilder v4 and controller-runtime v0.17.0 in Go 1.24, the operator uses a plugin-based architecture where controllers are selectively enabled via a startup flag (`--enable-services`). Each controller watches its own CRD and reconciles the desired state by creating Deployments, Services, Routes, ConfigMaps, RBAC resources, and monitoring infrastructure.

Expand Down Expand Up @@ -61,12 +61,6 @@ flowchart TB
evalhub_metrics["Metrics server\n(port 8081, plain HTTP)"]
end

subgraph gorch_pod["Guardrails Orchestrator Pod(s)"]
gorch_svc["Orchestrator\n(guardrail chains)"]
gorch_gw["Gateway sidecar\n(optional)"]
gorch_det["Built-in detectors\n(optional)"]
end

subgraph nemo_pod["NeMo Guardrails Pod(s)"]
nemo_svc["NeMo Guardrails Server\n(NVIDIA NeMo)"]
end
Expand All @@ -75,13 +69,11 @@ flowchart TB
manager -->|"creates/manages"| tas_pod
manager -->|"creates/manages"| lmes_pod
manager -->|"creates/manages"| evalhub_pod
manager -->|"creates/manages"| gorch_pod
manager -->|"creates/manages"| nemo_pod

tas_svc -->|"HTTP/gRPC"| model_endpoints["KServe InferenceServices"]
lmes_main -->|"evaluates"| model_endpoints
evalhub_svc -->|"submits LMEvalJobs"| k8s_api
gorch_svc -->|"intercepts inference"| model_endpoints
nemo_svc -->|"guards inference"| model_endpoints

evalhub_svc -->|"SQLite / PostgreSQL"| db["Database"]
Expand Down Expand Up @@ -123,11 +115,6 @@ flowchart LR
evalhub_roles["tenant_roles.go\n(single-mode RBAC)"]
end

subgraph gorch_ctrl["gorch/"]
gorch_rec["GuardrailsOrchestratorReconciler"]
gorch_autoconfig["auto_config.go\n(service discovery)"]
end

subgraph nemo_ctrl["nemo_guardrails/"]
nemo_rec["NemoGuardrailsReconciler"]
nemo_ca["ca_bundle.go\n(certificate aggregation)"]
Expand All @@ -147,7 +134,6 @@ flowchart LR
tas_types["tas/v1, v1alpha1\nTrustyAIService"]
lmes_types["lmes/v1alpha1\nLMEvalJob"]
evalhub_types["evalhub/v1, v1alpha1\nEvalHub"]
gorch_types["gorch/v1alpha1\nGuardrailsOrchestrator"]
nemo_types["nemo_guardrails/v1alpha1\nNemoGuardrails"]
common["common/\nCondition, helpers"]
end
Expand All @@ -156,7 +142,6 @@ flowchart LR
registry --> tas_rec
registry --> lmes_rec
registry --> evalhub_rec
registry --> gorch_rec
registry --> nemo_rec
registry --> job_mgr_rec
```
Expand Down Expand Up @@ -188,9 +173,9 @@ flowchart TB

| Mode | Enabled Services | Use Case | Overlay |
|------|-----------------|----------|---------|
| **ODH (full)** | TAS, LMES, GORCH, NEMO_GUARDRAILS, EVALHUB | Open Data Hub — all AI trust services | `config/overlays/odh/` |
| **RHOAI (full)** | TAS, LMES, GORCH, NEMO_GUARDRAILS, EVALHUB | Red Hat OpenShift AI — production | `config/overlays/rhoai/` |
| **ODH + Kueue** | TAS, LMES, GORCH, NEMO_GUARDRAILS, EVALHUB, JOB_MGR | Full + job queuing via Kueue | `config/overlays/odh-kueue/` |
| **ODH (full)** | TAS, LMES, NEMO_GUARDRAILS, EVALHUB | Open Data Hub — all AI trust services | `config/overlays/odh/` |
| **RHOAI (full)** | TAS, LMES, NEMO_GUARDRAILS, EVALHUB | Red Hat OpenShift AI — production | `config/overlays/rhoai/` |
| **ODH + Kueue** | TAS, LMES, NEMO_GUARDRAILS, EVALHUB, JOB_MGR | Full + job queuing via Kueue | `config/overlays/odh-kueue/` |
| **LMES only** | LMES | LLM evaluation workloads only | `config/overlays/lmes/` |
| **EvalHub only** | EVALHUB | Evaluation hub service only | `config/overlays/evalhub-only/` |
| **NeMo only** | NEMO_GUARDRAILS | NeMo Guardrails only | `config/overlays/mcp-guardrails/` |
Expand All @@ -202,7 +187,6 @@ flowchart TB
| **TAS** | Optional (3 modes) | -- | Yes (ServiceMonitor) | Yes | Optional (PVC or DB) |
| **LMES** | -- | Optional (via JOB_MGR) | -- | -- | -- |
| **EvalHub** | -- | Optional (workload monitoring) | Yes | Yes | Required (SQLite or PostgreSQL) |
| **GORCH** | Optional (auto-config) | -- | Yes | Yes | -- |
| **NeMo Guardrails** | -- | -- | -- | Yes | -- |
| **JOB_MGR** | -- | Required | -- | -- | -- |

Expand All @@ -211,7 +195,6 @@ flowchart TB
KServe is **not a mandatory dependency**. The operator behaves as follows:

- **TAS controller**: Watches InferenceService objects. If none exist in the namespace, reconciliation continues without error. When InferenceServices are present, TAS patches them with payload-processing endpoints based on the deployment mode annotation.
- **GORCH controller**: Uses InferenceServices for auto-configuration (discovering generator and detector services). Falls back to manual ConfigMap-based configuration if KServe is unavailable.
- **LMES, EvalHub, NeMo**: No KServe dependency.

## Data and Message Flow
Expand Down Expand Up @@ -349,31 +332,6 @@ sequenceDiagram
EH->>K8sAPI: Update status (Ready, URL, active providers)
```

### Guardrails Auto-Configuration

```mermaid
sequenceDiagram
participant User
participant K8sAPI as Kubernetes API
participant GORCH as GORCH Controller
participant ISvcs as InferenceServices

User->>K8sAPI: Create GuardrailsOrchestrator CR (autoConfig)
K8sAPI->>GORCH: Watch event

GORCH->>K8sAPI: List InferenceServices in namespace
GORCH->>ISvcs: Find generator (by name from spec)
GORCH->>ISvcs: Find detectors (by label match)

GORCH->>K8sAPI: Generate orchestrator ConfigMap (endpoints)
GORCH->>K8sAPI: Generate gateway ConfigMap (if enabled)

GORCH->>K8sAPI: Create Deployment (orchestrator + gateway + detectors)
GORCH->>K8sAPI: Create Service + Route

GORCH->>K8sAPI: Update autoConfigState in status
```

## Dependencies

### Mandatory
Expand All @@ -385,7 +343,7 @@ sequenceDiagram

### Optional

- **KServe** v0.12.1 — enables TAS inference service patching and GORCH auto-configuration
- **KServe** v0.12.1 — enables TAS inference service patching
- **Kueue** v0.6.2 — enables job queuing for LMEvalJobs (requires JOB_MGR service)
- **Istio** — enables DestinationRule/VirtualService creation for service mesh traffic management (TAS with RawDeployment mode)
- **OpenShift Routes** — enables external HTTP/HTTPS access via Routes (gracefully skipped on vanilla Kubernetes)
Expand Down Expand Up @@ -419,7 +377,6 @@ flowchart TB
tas_cr["TrustyAIService CR"] --> tas_deploy["TAS Deployment\n+ kube-rbac-proxy"]
lmes_cr["LMEvalJob CR"] --> lmes_pods["Eval Pod(s)\n+ driver sidecar"]
evalhub_cr["EvalHub CR"] --> evalhub_deploy["EvalHub Deployment"]
gorch_cr["GuardrailsOrchestrator CR"] --> gorch_deploy["Orchestrator Deployment\n+ gateway + detectors"]
nemo_cr["NemoGuardrails CR"] --> nemo_deploy["NeMo Deployment"]
end

Expand Down Expand Up @@ -468,14 +425,12 @@ trustyai-service-operator/
│ ├── lmes/v1alpha1/ # LMEvalJob types
│ ├── evalhub/v1/ # EvalHub v1 (storage version)
│ ├── evalhub/v1alpha1/ # EvalHub v1alpha1
│ ├── gorch/v1alpha1/ # GuardrailsOrchestrator types
│ └── nemo_guardrails/v1alpha1/ # NemoGuardrails types
├── controllers/
│ ├── controllers.go # Service registry (init-based registration)
│ ├── tas/ # TAS reconciler (KServe, TLS, storage)
│ ├── lmes/ # LMES reconciler (pod lifecycle, state machine)
│ ├── evalhub/ # EvalHub reconciler (providers, tenancy, DB)
│ ├── gorch/ # GORCH reconciler (auto-config, detectors)
│ ├── nemo_guardrails/ # NeMo reconciler (CA bundles)
│ ├── job_mgr/ # Kueue job manager
│ ├── utils/ # Shared reconciliation helpers
Expand Down
6 changes: 2 additions & 4 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,6 @@ func init() {
| `TAS` | `controllers/tas/` | `TrustyAIService` (v1alpha1, v1) |
| `LMES` | `controllers/lmes/` | `LMEvalJob` (v1alpha1) |
| `EVALHUB` | `controllers/evalhub/` | `EvalHub` (v1alpha1, v1) |
| `GORCH` | `controllers/gorch/` | `GuardrailsOrchestrator` (v1alpha1) |
| `NEMO_GUARDRAILS` | `controllers/nemo_guardrails/` | `NemoGuardrail` (v1alpha1) |
| `JOB_MGR` | `controllers/job_mgr/` | (watches external jobs) |

Expand All @@ -104,7 +103,6 @@ api/ # CRD type definitions (kubebuilder markers)
tas/v1alpha1/, tas/v1/ # TrustyAIService (with version conversion)
lmes/v1alpha1/ # LMEvalJob
evalhub/v1alpha1/, evalhub/v1/ # EvalHub (v1 is storage version)
gorch/v1alpha1/ # GuardrailsOrchestrator
nemo_guardrails/v1alpha1/ # NemoGuardrail
common/ # Shared types (Condition, CABundle)
controllers/
Expand Down Expand Up @@ -191,7 +189,7 @@ Tenant ConfigMap changes trigger reconciliation via a ConfigMap watch (`tenantCo

Registers all API groups in order:
- Kubernetes core (`clientgoscheme`)
- TrustyAI: `tasv1alpha1`, `tasv1`, `lmesv1alpha1`, `evalhubv1alpha1`, `evalhubv1`, `gorchv1alpha1`, `nemoguardrailsv1alpha1`
- TrustyAI: `tasv1alpha1`, `tasv1`, `lmesv1alpha1`, `evalhubv1alpha1`, `evalhubv1`, `nemoguardrailsv1alpha1`
- External: `monitoringv1` (Prometheus), `kservev1alpha1`/`v1beta1`, `routev1` (OpenShift), `apiextensionsv1`, `kueuev1beta1`

### Key Dependencies
Expand Down Expand Up @@ -263,7 +261,7 @@ make policy-check # Full check against all overlays

**Current policies:**
- `policy/rbac.rego` — closed allowlist of expected `ClusterRoleBinding` resources. Any CRB not in `expected_crbs` or binding the wrong `ClusterRole` is denied.
- `policy/clusterrole.rego` — inspects ClusterRole **contents**: (1) closed allowlist of permitted `(apiGroup, resource)` pairs, (2) denylist blocking wildcards, secrets write, privilege escalation verbs. Manager roles that legitimately need secrets or CRB write (TAS, GORCH, nemo-guardrails, evalhub) are exempt by name suffix — see `policy/README.md` for the full exemption list.
- `policy/clusterrole.rego` — inspects ClusterRole **contents**: (1) closed allowlist of permitted `(apiGroup, resource)` pairs, (2) denylist blocking wildcards, secrets write, privilege escalation verbs. Manager roles that legitimately need secrets or CRB write (TAS, nemo-guardrails, evalhub) are exempt by name suffix — see `policy/README.md` for the full exemption list.

**When adding RBAC resources:**
- If you add a new `ClusterRoleBinding` (e.g. in a component's `rbac/` directory), you must add its post-kustomize name and expected `ClusterRole` to `expected_crbs` in `policy/rbac.rego`.
Expand Down
84 changes: 0 additions & 84 deletions Dockerfile.orchestrator

This file was deleted.

9 changes: 0 additions & 9 deletions PROJECT
Original file line number Diff line number Diff line change
Expand Up @@ -42,15 +42,6 @@ resources:
defaulting: true
validation: true
webhookVersion: v1
- api:
crdVersion: v1
namespaced: true
controller: true
domain: opendatahub.io
group: trustyai
kind: GuardrailsOrchestrator
path: github.com/trustyai-explainability/trustyai-service-operator/api/gorch/v1alpha1
version: v1alpha1
- api:
crdVersion: v1
namespaced: true
Expand Down
1 change: 0 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
The TrustyAI Kubernetes Operator aims at simplifying the deployment and management of various TrustyAI Kubernetes components, such as:
- [TrustyAI Service](https://github.com/trustyai-explainability/trustyai-explainability): A service that deploys alongside KServe models and collects
inference data to enable model explainability, fairness monitoring, and drift tracking.
- [FMS-Guardrails](https://github.com/foundation-model-stack/fms-guardrails-orchestrator): A modular framework for guardrailing LLMs
- [LM-Eval](https://github.com/EleutherAI/lm-evaluation-harness/tree/main): A job-based architecture for deploying and managing LLM evaluations, based on EleutherAI's lm-evaluation-harness library.
- [EvalHub](https://github.com/eval-hub): A service for orchestrating AI/ML model evaluations, providing a unified API for running benchmarks across multiple evaluation providers and frameworks, with support for multi-tenant and single-tenant deployments.

Expand Down
43 changes: 0 additions & 43 deletions api/gorch/v1alpha1/groupversion_info.go

This file was deleted.

Loading
Loading