Repository navigation
Commit ce99c3a
feat(application): add 507 snapshot normalizer MQTT adapter and chart (#866)
# Pull Request
## Description
> **Depends on #867.** Merge #867 first: it makes the 507 inference
service subscribe to the pinned `edge-ai/v1/+/camera/+/snapshots` filter
this adapter publishes to.
Adds the component 507 **snapshot normalizer**: an MQTT adapter that
subscribes to a binary JPEG snapshot topic, such as a media connector
`snapshot-to-mqtt` stream, and republishes each accepted snapshot as an
`image_snapshot` v1 request for the 507 inference service. It's built on
the transport-free `snapshot-normalizer-core` library from #831 and
follows the custom workload messaging contract from #852.
* **New crate `services/snapshot-normalizer`.** It's a binary crate that
depends on the core library by path, so the core stays transport-free as
its README promises.
* **Versioned output topic.** The default is
`edge-ai/v1/snapshot-normalizer/camera/{camera-id}/snapshots`, following
the #852
`{domain}/{version}/{producer}/{resource-kind}/{resource-id}/{message-kind}`
grammar pinned to `v1`. It matches 507's pinned
`edge-ai/v1/+/camera/+/snapshots` filter, which #867 adds to the binary
default and the shipped manifests alongside the legacy 5-level filter;
#867 also rejects envelopes whose `schema_version` major isn't `1`. The
adapter refuses to start when its output topic matches its own input
filter, including through wildcards.
* **CloudEvents attributes as MQTTv5 user properties** come from the
SDK's validated `CloudEventBuilder`: `specversion`, `type`, `source`,
`id`, `time`, `subject`, `datacontenttype` (also the MQTTv5 Content
Type), an optional `dataschema`, and a propagated W3C `traceparent`.
* **Producer-keyed deduplication (FR-01).** The key is the input's
CloudEvents `id`, scoped to the source topic and `source`, and bounded
by both entry count and time window. Content is never a key. A key is
recorded only after the PUBACK. Output `id` and `correlation_id` are
random UUIDv4 values, per #852, never derived from producer identifiers.
* **Delivery (at least once).** Inputs are acknowledged manually after
handling. Publishes get up to `PUBLISH_ATTEMPTS` (1-10) attempts with
exponential backoff capped at 30 s; a broker failure reason is not
retried. Failed publishes are counted and dropped rather than left
unacknowledged, because the SDK delivers acks in receive order. Only an
input left unacknowledged by a restart is redelivered, and it can then
be published twice.
* **Session.** The adapter fails at startup when the SUBACK carries a
failure reason, so the pod crash-loops visibly. It advertises a receive
maximum of 8 and a maximum packet size of `MAX_JPEG_BYTES` plus 64 KiB,
and subscribes with retain handling "do not send" so retained snapshots
aren't replayed.
* **Payload-safe logging.** Logs contain counters, reasons, and lengths
only; no payload, Base64 data, or producer identifiers.
* **Helm chart `charts/snapshot-normalizer` (FR-05):**
* A single-replica StatefulSet, so `AIO_MQTT_CLIENT_ID` (prefix plus
`metadata.name` through the Downward API) is unique per release and
stable at `<fullname>-0` across restarts, and the persistent session
resumes.
* `AIO_MQTT_SESSION_EXPIRY` set explicitly from
`mqtt.sessionExpirySeconds` (3600 by default).
* A dedicated service account without an automounted API token, a
projected `aio-internal` SAT over TLS, and the
`aio-broker-auth/workload` attribute annotation for least-privilege
rules.
* No ports or probes; a read-only root filesystem with dropped
capabilities and `RuntimeDefault` seccomp.
* A placeholder image registry.
* **507 consumer fix.** `IncomingMessage::ImageSnapshot.metadata` now
defaults when absent. The `image_snapshot` v1 schema makes `metadata`
optional and the core omits it when empty, but the consumer rejected
such messages with `missing field 'metadata'`. A regression test
deserializes the schema examples and a normalizer-shaped envelope
without `metadata`.
* **Local compose.** The `snapshot-normalizer` service loads
`services/snapshot-normalizer/.env.template`, then an optional
git-ignored `.env`, instead of hard-coded values.
The inference chart hardening items (FR-02 model paths and readiness,
FR-03 probe-only listener, FR-04 metrics port) are left to a separate
follow-up PR.
## Related Issue
Follows up #831 (PCM-03 packaging) and applies the custom workload
messaging contract from #852 (#835).
## Type of Change
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Blueprint modification or addition
- [x] Component modification or addition
- [x] Documentation update
- [x] CI/CD pipeline change
- [ ] Other (please describe):
## Implementation Details
* Uses `azure_iot_operations_mqtt` 1.2.3: `recv_manual_ack`,
`TopicName::matches_topic_filter`, and `aio::cloud_event`. Other 507 and
50x crates are unchanged.
* `config.rs` reads configuration through an injected lookup. It
validates the camera ID as a lowercase URL-safe topic segment, the
topics, the CloudEvents `source` and `dataschema`, and positive numeric
bounds.
* `pipeline.rs` is pure: `prepare()` returns `Rejected`, `Duplicate`, or
`Publish` without I/O. `main.rs` only owns the session, acks, retries,
counters, and SIGTERM handling.
* The chart renders numeric values through `int64`, so large byte limits
aren't emitted in float notation (for example `4.194304e+06`).
* Registered the crate in both `rust-tests.yml` matrices. `codecov.yml`
already covers `507-ai-inference/**`.
* Made the existing root-absolute `CONTRIBUTING.md` link in the 507
README relative so the link checker resolves it.
## Testing Performed
- [ ] Terraform plan/apply
- [ ] Blueprint deployment test
- [x] Unit tests
- [x] Integration tests
- [x] Bug fix includes regression test (see [Test
Policy](docs/contributing/testing-validation.md))
- [x] Manual validation
- [x] Other: container image build and smoke test
* `cargo fmt --check`, `cargo clippy --all-targets -- -D warnings`, and
`cargo test` pass: 33 tests for the new crate, including SUBACK
rejection, retain handling, capped backoff, the `PUBLISH_ATTEMPTS`
bound, random v4 IDs, and the v1 topic and loop guard. The
`ai-edge-inference` crate passes `cargo clippy --all-targets -- -D
warnings` and its 8 tests with the `metadata` fix.
* A local merge of this branch onto #867 is conflict-free and passes the
combined `ai-edge-inference` tests, clippy, `docker compose config`, and
`kubectl kustomize`.
* **End-to-end against an MQTTv5 broker (Mosquitto):**
* 5 inputs produced 3 published requests, 1 duplicate (same `id`), 1
`unkeyed`, and 1 `not_jpeg` rejection.
* Output arrived on the inference service's versioned input filter with
the expected CloudEvents properties. This end-to-end pass predates the
review follow-up commit that moved the topic to
`edge-ai/v1/snapshot-normalizer/camera/{camera-id}/snapshots`.
* Producer `time` and `traceparent` were propagated.
* A standalone serde probe of the 507 `IncomingMessage` confirmed that
an envelope without `metadata` failed before the fix.
* The release image built from `services/` runs as uid 1000 on a
read-only root filesystem, published to the broker, and logged final
counters on SIGTERM.
* `helm lint --strict` and `helm template` with default and override
values pass and render a ServiceAccount and a StatefulSet.
* `Validate-RustCrateRegistration.ps1` passes (18 crates). Grype finds
no vulnerabilities in the new `Cargo.lock`. gitleaks finds no leaks.
* markdownlint, markdown-table-formatter, cspell, and
markdown-link-check pass on the changed Markdown.
## Validation Steps
1. `cd src/500-application/507-ai-inference/services/snapshot-normalizer
&& cargo test && cargo clippy --all-targets -- -D warnings`
2. `helm template t
src/500-application/507-ai-inference/charts/snapshot-normalizer` and
confirm a `StatefulSet` with `replicas: 1`, `AIO_MQTT_CLIENT_ID` using
`$(POD_NAME)`, and `AIO_MQTT_SESSION_EXPIRY: "3600"`.
3. Optionally run a local MQTTv5 broker and publish a JPEG with an `id`
user property to `INPUT_TOPIC`, as described in the crate README.
## Checklist
- [x] I have updated the documentation accordingly
- [x] I have added tests to cover my changes
- [x] All new and existing tests passed
- [ ] I have run `terraform fmt` on all Terraform code
- [ ] I have run `terraform validate` on all Terraform code
- [ ] I have run `az bicep format` on all Bicep code
- [ ] I have run `az bicep build` to validate all Bicep code
- [x] I have checked for any sensitive data/tokens that should not be
committed
- [x] Lint checks pass (run applicable linters for changed file types)
## Security Review
- [x] No credentials, secrets, or tokens are hardcoded or logged
- [x] RBAC and identity changes follow least-privilege principles
- [x] No new network exposure or public endpoints introduced without
justification
- [x] Dependency additions or updates have been reviewed for known
vulnerabilities
- [x] Container image changes use pinned digests or SHA references
## Additional Notes
* No image is published. The chart uses
`your-registry.azurecr.io/snapshot-normalizer` as a placeholder.
* Microsoft Learn doesn't document whether the media connector sets a
CloudEvents `id` on `snapshot-to-mqtt` messages. Without one, messages
pass through un-deduplicated and are counted as `unkeyed`.
* Deduplication state is in memory, which is why the chart runs a
single-replica StatefulSet. Scaling to shared subscriptions would need
shared dedup state, per the #852 guidance.
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Bill Berry <WilliamBerryiii@users.noreply.github.com>1 parent cc89446 commit ce99c3a
18 files changed
Lines changed: 3743 additions & 0 deletions
File tree
- .cspell
- .github/workflows
- src/500-application/507-ai-inference
- charts/snapshot-normalizer
- templates
- services
- ai-edge-inference/src
- snapshot-normalizer
- src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
| 39 | + | |
38 | 40 | | |
39 | 41 | | |
40 | 42 | | |
| |||
200 | 202 | | |
201 | 203 | | |
202 | 204 | | |
| 205 | + | |
203 | 206 | | |
204 | 207 | | |
205 | 208 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
65 | 65 | | |
66 | 66 | | |
67 | 67 | | |
| 68 | + | |
68 | 69 | | |
69 | 70 | | |
70 | 71 | | |
| 72 | + | |
71 | 73 | | |
72 | 74 | | |
73 | 75 | | |
| |||
130 | 132 | | |
131 | 133 | | |
132 | 134 | | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
133 | 153 | | |
134 | 154 | | |
135 | 155 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
43 | 60 | | |
44 | 61 | | |
45 | 62 | | |
| |||
0 commit comments