Improve harvester log grouping and error reporting by TahaKhan998 · Pull Request #831 · CERNDocumentServer/cds-rdm

TahaKhan998 · 2026-06-26T10:18:55Z

Closes #813
This PR improves INSPIRE harvester run error reporting by turning errors and warning into clear grouped errors and warnings, making log output much more useful for curators, and ensuring the report highlights the real failure reasons instead of vague or repetitive raw lines. It adds structured grouping for repeated failures and warnings, improves the plain-text download output to match that grouped view, and updates harvester writer messages so record-level failures explain what went wrong in a more actionable way, including which records were affected and why they failed.

Screen.Recording.2026-06-26.at.12.00.16.1.mp4

This is what a downloaded file of harvester logs looks like now:

harvester_logs_4a7fc391-5b9a-45b3-873f-bd2d2a297aaa_20260626_120105.log

kpsherva · 2026-07-02T11:48:34Z

 def format_timestamp(value):
    """Format timestamps for display."""
-    if value is None or value == "":
+    if value in (None, ""):


Suggested change

if value in (None, ""):

if not value:

kpsherva · 2026-07-02T12:34:48Z

+
+def _compact(value):
+    """Collapse whitespace while keeping the original text readable."""
+    text = str(value or "").replace("\\n", " ").replace("\n", " ")


where are the whitespaces originally added?

kpsherva · 2026-07-02T12:35:48Z

+    return " ".join(text.split()).strip()
+
+
+def _remove_prefix(text, prefix):


which prefixes are removed?

kpsherva · 2026-07-02T12:37:27Z

+    return None
+
+
+def _strip_leading_bracket_prefixes(message):


there is a lot of things happening here which I need some narration from your side to understand. Please book (on calendar) some time for us to go through the code together

TahaKhan998 · 2026-07-08T08:38:08Z

+    return match.group("id") if match else None
+
+
+def _unwrap_skipped_entry_error(reason):


remove dictonary

kpsherva · 2026-07-08T08:42:15Z

+    )
+
+
+def _safe_strip_wrapper_prefixes(reason):


check if this can be fixed while logging in the first place - are the classes added in the harvester task? if yes we can reorganise there instead of parsing. If these classes are added in the jobs - let's discuss

kpsherva · 2026-07-08T08:45:27Z

+        return "Unknown error"
+
+    if _PLACEHOLDER_ONLY_TITLE.match(stripped):
+        return "Record validation failed." if stripped == "<payload>" else "Log message"


please check how to fix .. errors and display the error correctly instead of stripping the payload

kpsherva · 2026-07-08T08:47:08Z

+
+
+def _known_error_stem(reason):
+    """Normalize legacy and current variants of recurring harvester errors."""


is this normalisation needed?

kpsherva · 2026-07-10T13:04:44Z

+    return None
+
+
+def _peel_string_list(text):


what is the case for which we are returning a list in the log? maybe the log is incorrect?

Good question. The screenshot shows where this comes from, in invenio-vocabularies, when a record fails it logs result.errors directly, and that's a list. So what ends up in the log looks like ['error message here'] instead of plain text. _peel_string_list just strips that wrapper off so the report can group errors properly. For example, without it we'd group on something like skipped entry with errors: ['more than 1 doi was found.'], brackets, quotes and all. After peeling we group on just more than 1 doi was found., so records that failed for the same reason actually end up in the same group.

It would be nicer if invenio-vocabularies wrote it as plain text in the first place. I handled it in the report for now since past harvester runs already have logs saved this way, but if you'd rather fix it at the source I can do that instead.

kpsherva · 2026-07-10T13:09:20Z

    """INSPIRE writer."""

+    @staticmethod
+    def _format_validation_error(error):


these methods would fit better in the logger.py, as logging is not the responsibility of this class. What do you think?

You're right — moved them to logger.py.

kpsherva · 2026-07-10T13:11:57Z

@@ -16,7 +16,7 @@
 from cds_rdm.harvester_runs.logs import (


feedback not connected with this file but with the directory:
could you move harvester_download and harvester_runs into inspire_harvester/reports/download and inspire_harvester/reports/runs - to package the whole functionality of the harvester together

kpsherva · 2026-07-10T13:13:06Z

+            if search_after:
+                params["search_after"] = search_after
+
+            result = current_jobs_logs_service.search(system_identity, params=params)


.search function will return only 10k results max, even with the pagination

In invenio-jobs it’s 2000 per .search() by default (screenshot). Not sure about the 10k, maybe that’s in the CDS deployment config? Our loop just keeps fetching batches of 2000 with search_after until we’ve got all the logs or hit the page limit. The old code only fetched once.

kpsherva · 2026-07-10T13:23:04Z

+    return " ".join(text.split()).strip()
+
+
+def flatten_error_payload(value, prefix=""):


I think there is a function which does something very similar: search for validation_error_to_list_errors/_iter_errors_dict in invenio_records_resources.errors

Yep, switched to the invenio helper and dropped the duplicate.

Move harvester run/download reports under inspire_harvester/reports, stabilise writer and transformer error messages for grouping, and add lean grouped error reporting in the admin harvester run view.

kpsherva reviewed Jul 2, 2026

View reviewed changes

TahaKhan998 force-pushed the feat/issue-813-harvester-log-grouping branch 18 times, most recently from 1e2134c to 47666b0 Compare July 8, 2026 07:27

TahaKhan998 commented Jul 8, 2026

View reviewed changes

kpsherva reviewed Jul 8, 2026

View reviewed changes

TahaKhan998 force-pushed the feat/issue-813-harvester-log-grouping branch 4 times, most recently from f07326b to 40ede92 Compare July 8, 2026 15:29

TahaKhan998 force-pushed the feat/issue-813-harvester-log-grouping branch 5 times, most recently from 6851e19 to 52262d7 Compare July 9, 2026 08:48

kpsherva reviewed Jul 10, 2026

View reviewed changes

TahaKhan998 force-pushed the feat/issue-813-harvester-log-grouping branch 3 times, most recently from e3a6f16 to 3548734 Compare July 10, 2026 15:12

feat(harvester): improve error messages and lean log grouping

167c385

Move harvester run/download reports under inspire_harvester/reports, stabilise writer and transformer error messages for grouping, and add lean grouped error reporting in the admin harvester run view.

TahaKhan998 force-pushed the feat/issue-813-harvester-log-grouping branch from 5f504a8 to 167c385 Compare July 10, 2026 15:27

		return " ".join(text.split()).strip()


		def _remove_prefix(text, prefix):

		return match.group("id") if match else None


		def _unwrap_skipped_entry_error(reason):



		def _known_error_stem(reason):
		"""Normalize legacy and current variants of recurring harvester errors."""

		return " ".join(text.split()).strip()


		def flatten_error_payload(value, prefix=""):

Uh oh!

Conversation

TahaKhan998 commented Jun 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kpsherva Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kpsherva Jul 8, 2026 •

edited

Loading