Skip to content

synchronization: log per-event partial delivery errors from PublishBatch - #22998

Draft
pkcll wants to merge 2 commits into
developfrom
chipingress-partial-delivery-cl-nodes
Draft

synchronization: log per-event partial delivery errors from PublishBatch#22998
pkcll wants to merge 2 commits into
developfrom
chipingress-partial-delivery-cl-nodes

Conversation

@pkcll

@pkcll pkcll commented Jul 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

  • `chip_ingress_batch_worker.Send` was discarding the `PublishBatch` response (`_, err := ...`).
  • Now captures `resp` and inspects `resp.GetResults()`: per-event errors are grouped by error code and emitted as one WARN log per batch (`dropped`, `by_code`) so a batch with many failures does not flood logs.
  • Adds a new `chip_ingress_partial_delivery_dropped{telemetry_type, error_code}` Prometheus counter so dashboards and alerts can track rejection types by error code without relying on logs.
  • Existing `telemetry_client_messages_dropped` counter still increments per dropped event (backward-compatible).
  • Implements AC Features/encode bytes32 #4 from INFOPLAT-13901: inspect other `PublishBatch` call sites and surface partial-delivery errors.

Jira: INFOPLAT-13901 | Epic: INFOPLAT-7177

Depends on chainlink-common PR #2210 merging first, then a `go.mod` bump before this PR is ready to merge.

Changes

  • `core/services/synchronization/metrics.go`: new `chip_ingress_partial_delivery_dropped` counter with `["telemetry_type", "error_code"]` labels
  • `core/services/synchronization/chip_ingress_batch_worker.go`: capture `resp`, group results by `error_code`, single WARN log per batch, increment both counters per dropped event
  • `core/services/synchronization/chip_ingress_batch_worker_test.go`: `TestChipIngressBatchWorker_Send_PartialDelivery` — two failed events produce exactly one grouped WARN log line

Test plan

  • `go test ./core/services/synchronization/... -count=1` passes
  • `TestChipIngressBatchWorker_Send_PartialDelivery` passes — single WARN log for 2 failed events
  • After go.mod bump: re-run to confirm no regressions from chainlink-common dep update
  • `chip_ingress_partial_delivery_dropped{telemetry_type="ocr", error_code="PUBLISH_ERROR_CODE_VALIDATION_FAILED"}` increments per dropped event

@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

✅ No conflicts with other open PRs targeting develop

@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

I see you updated files related to core. Please run make gocs in the root directory to add a changeset as well as in the text include at least one of the following tags:

  • #added For any new functionality added.
  • #breaking_change For any functionality that requires manual action for the node to boot.
  • #bugfix For bug fixes.
  • #changed For any change to the existing functionality.
  • #db_update For any feature that introduces updates to database schema.
  • #deprecation_notice For any upcoming deprecation functionality.
  • #internal For changesets that need to be excluded from the final changelog.
  • #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
  • #removed For any functionality/config that is removed.
  • #updated For any functionality that is updated.
  • #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

@trunk-io

trunk-io Bot commented Jul 1, 2026 •
edited
Loading

Copy link
Copy Markdown

Static Badge   Static Badge   Static Badge   Static Badge

View Full Report ↗︎ ⋅ Docs

@pkcll
pkcll force-pushed the chipingress-partial-delivery-cl-nodes branch 3 times, most recently from 32d3c31 to 094609f Compare July 2, 2026 22:18
@pkcll
pkcll force-pushed the chipingress-partial-delivery-cl-nodes branch from 094609f to 6cade0f Compare July 21, 2026 04:23
@pkcll
synchronization: remove partial delivery WARN log, keep metric only
68635a2 
Partial delivery is a per-event, often persistent server-side condition
(e.g. missing schema), so logging it would WARN on every send interval
(as low as 500ms) for as long as it lasts. ChipIngressPartialDeliveryDropped
(telemetry_type, error_code) already reports it with bounded cardinality
and is the intended signal for alerting/investigation.
@cl-sonarqube-production

cl-sonarqube-production Bot commented Jul 21, 2026

Copy link
Copy Markdown

Quality Gate passed Quality Gate passed

Issues
1 New issue
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@pkcll