Skip to content

fix(medcat-trainer): Fix issue with metrics failing due to missing meta values#563

Merged
mart-r merged 2 commits into
mainfrom
fix/medcat-trainer/fix-issue-with-meta-values-not-existing-during-metrics
Jun 24, 2026
Merged

fix(medcat-trainer): Fix issue with metrics failing due to missing meta values#563
mart-r merged 2 commits into
mainfrom
fix/medcat-trainer/fix-issue-with-meta-values-not-existing-during-metrics

Conversation

@mart-r

@mart-r mart-r commented Jun 24, 2026

Copy link
Copy Markdown
Collaborator

There would - after hours of running - be errors such as this:

ERROR 2026-06-24 01:05:39,455 process_tasks.py l:26:/home/api/api/metrics.py:303: UserWarning: The meta_model Presence does not exist in this MedCATtrainer export.
  warnings.warn(f"The meta_model {category_name} does not exist in this MedCATtrainer export.", UserWarning)

ERROR 2026-06-24 01:05:39,455 tasks.py l:57:Rescheduling metrics-58-2026-06-23 15:03:19
Traceback (most recent call last):
  File "/home/.venv/lib/python3.12/site-packages/background_task/tasks.py", line 43, in bg_runner
    func(*args, **kwargs)
  File "/home/api/api/metrics.py", line 61, in calculate_metrics
    report = metrics.generate_report(meta_ann=loaded_model_pack)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/api/api/metrics.py", line 429, in generate_report
    anno_df = self.full_annotation_df()
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/api/api/metrics.py", line 340, in full_annotation_df
    meta_values = {v: k for k, v in meta_results['meta_values'].items()}
                                    ~~~~~~~~~~~~^^^^^^^^^^^^^^^
KeyError: 'meta_values'
WARNING 2026-06-24 01:05:39,618 models.py l:254:Marking task metrics-58-2026-06-23 15:03:19 as failed

Which is why metrics seem to fail.

So this PR tries to fix that. So at least things should (hopefully) run to the end.

EDIT:
I looked at this in a bit more detail. And while the task had been scheduled for hours by the time this happened, it failed more or less as soon as it ran!
So this may not be the ultimate fix. I'll need to look at this closer.

EDIT2: Look at comment below. This should fix the issue, I was fixated on something else.

@mart-r mart-r marked this pull request as draft June 24, 2026 09:38
@mart-r mart-r marked this pull request as ready for review June 24, 2026 13:02
@mart-r

mart-r commented Jun 24, 2026

Copy link
Copy Markdown
Collaborator Author

Just as an FYI:

The reason I was hesitant in the above:
I ran metrics yesterday around 3PM (according to logs anyway).
However, they only failed at around 1AM. And it looks like while it was trying to load a model pack a few times, it never explicitly failed (or at least the failure wasn't logged). So I wasn't sure what was going on.

But, of course, the logs I provided clearly showed:

The meta_model Presence does not exist in this MedCATtrainer export

So I think the expectation that this only fails if/when the meta model doesn't exist in the export. So the return value will be:

return {category_name: f"{category_name} does not exist"}

as per the line after the warning.

So this PR should fix the underlying issue for cases where the expected and given meta tasks/models match.

@tomolopolis tomolopolis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@mart-r mart-r merged commit a68cc0a into main Jun 24, 2026
11 checks passed
@mart-r mart-r deleted the fix/medcat-trainer/fix-issue-with-meta-values-not-existing-during-metrics branch June 24, 2026 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants