[pull] master from scrapinghub:master - #96
Merged
Conversation
* Update CLDR data to 44.1.0 and add hour/duration support - Upgrade CLDR source data from version 31.0.1 to 44.1.0: 288 new locales added, all existing locale JSONs updated in dateparser_data/cldr_language_data/date_translation_data/ - Add hour/duration unit to relative-type data for all locales - Regenerate all dateparser/data/date_translation_data/*.py from the new source data, keeping master's possessive-quantifier optimization in relative-type-regex patterns - Modernize dateparser_scripts/ to use pathlib.Path and unified cldr-json repository layout (utils.py, write_complete_data.py, get_cldr_data.py, order_languages.py) - Restore Ukrainian words removed by the CLDR upgrade (uk.yaml) - Update docs/supported_locales.rst and languages_info.py - Update 34 test inputs in test_languages.py to reflect CLDR 44.1.0 abbreviation/name changes (bs-Latn, ce, kl, qu, so, sr, sw, zu, am, as, brx, hy, ig, kok, mr, nn, de, eu, gu, mk, chr, bs-Cyrl) - Add /cldr-json/ to .gitignore Finishes PR #1216 (Gallaecio:cldr-update). Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * Fix 7 regressions from CLDR 44.1.0 update - en.yaml: restore 'in {0} weeks time' and 'in {0} weeks\' time' patterns for 'in \1 week' that CLDR 44 dropped; regenerate en.py - af.yaml: restore 'sek' (second abbreviation) dropped by CLDR 44; regenerate af.py - en-US: CLDR 44 has no en-US.json (US settings merged into base en); add minimal en-US.json, regenerate en-US.py, add en-US back to languages_info.py locale list for 'en' - tests/test_freshness_date_parser.py: update Cherokee (chr) test input from uppercase to lowercase Cherokee encoding used by CLDR 44 patterns - tests/test_search.py: update Danish detection test to text that includes 'tirsdag' and 'januar' (distinctly Danish vs Swedish), since CLDR 44 sv.py changes caused the old text to be ambiguous Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * Fix en-US locale name: add to en.yaml locale_specific section The en-US locale is loaded via en.py's locale_specific["en-US"] section, not as a standalone language file. Adding it to en.yaml ensures the locale_specific entry gets the correct name ("en-US") and date_order (MDY). Remove the standalone en-US.json that was added by mistake. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * Finalize CLDR 44 update: keep supported locale set and address review Regenerating the data files for CLDR 44 also produced ~144 new low-resource locales, but languages_info.py (the locale registry) was never regenerated, so those files were shipped unused (parse(locales=['ceb']) raised ValueError). Rather than expand the auto-detected language set in this data-refresh PR, keep the supported set identical to master and defer the new locales to a dedicated change. - order_languages.py: exclude the 144 new CLDR 44 locales via avoid_languages so generation stays limited to the already-supported set. - Regenerate data: drop the 144 unused locale files and their intermediate JSON; the 205 supported locales are byte-identical. - Restore ff-CM/ff-GN/ff-MR (dropped by CLDR 44, moved under ff-Latn): keep them registered and add supplementary ff.yaml so they resolve to the base `ff` data, preserving backwards compatibility. - languages_info.py and docs/supported_locales.rst: revert to master now that the supported set is unchanged (en-US stays in sorted order, ff keeps its locales). - Drop the unnecessary OrderedDict usage from the generation scripts: dict keeps insertion order on all supported Python versions and the output is sorted, so the generated data is byte-identical. - Add tests for the new hour/duration parsing (ca "2 hores") and ff-CM/GN/MR backwards compatibility. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Restore vocabulary dropped by CLDR 44 so original tests pass unchanged The CLDR 31->44 refresh silently dropped ~690 previously-recognized words/patterns across 87 languages (e.g. ru "5 д назад", de "nachm.", pt "há 5 mins", hi "5 मि पहले", mk "минатата година", da "tir"/"tor"). Restore all of them through supplementary_language_data YAMLs - the mechanism already used for uk/en/af/ff - so the update becomes purely additive: every form that parsed before still parses, plus ~1,800 new CLDR 44 forms. Excluded from the restore are 8 forms that CLDR corrected deliberately (dsb/hsb "now" is a November abbreviation, mk "недела" moved from week to sunday, hi "से" moved from second to skip, bg "с" moved from second to week, af "oor N minuut" was wrongly mapped to in-N-days/months, and ms "dalam N saat" was wrongly mapped to in-N-years). Revert all test-expectation edits back to master, keeping only the additive tests (ff-CM/ff-GN/ff-MR locales, ca standalone hour unit). Add "den" as a Danish skip word: CLDR 44 gave Swedish "den här veckan" style phrases, which made Danish text win-by-tie-break as "sv" in language detection. Danish dates routinely use "den" ("den 8. maj 1945"), so this restores correct detection and makes such strings parseable; the two Danish search_dates spans now include "den", like "de" in Spanish. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
The new strategy tokenizes the text on characters that never occur inside a date expression and tries to parse token n-grams, longest first, so that the most complete date is found. Compared to the default translation-based "split" strategy it produces more predictable results on noisy real-world text. The strategy is exposed through the new "strategy" parameter of search_dates and defaults to the existing behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )