Skip to content

fix(extract): honor HTML base href for relative URLs - #43

Open
Yoomitech wants to merge 1 commit into
karust:mainfrom
Yoomitech:fix/extract-base-href
Open

fix(extract): honor HTML base href for relative URLs#43
Yoomitech wants to merge 1 commit into
karust:mainfrom
Yoomitech:fix/extract-base-href

Conversation

@Yoomitech

Copy link
Copy Markdown

Closes #42

Summary

Honor the first valid HTML <base href> element when resolving relative URLs during extraction.

Previously, relative references were always resolved against the fetched document URL. For pages using a base element such as:

<base href="/">

this could produce duplicated path components such as:

https://example.com/de/de/article.html

instead of:

https://example.com/de/article.html

Changes

  • determine the effective URL base from the first valid <base href>
  • use the effective base for extracted links and canonical URLs
  • use the effective base for Markdown links and relative image URLs
  • retain the fetched document URL as Trafilatura's OriginalURL
  • fall back to the fetched document URL when no usable HTTP or HTTPS base is present
  • add regression tests for metadata, canonical, Markdown link, and image resolution

Testing

  • make fmt
  • make lint
  • make test
  • real-world extraction test against a page using <base href="/">

All package tests and lint checks pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: URL extraction ignores HTML <base href> when resolving relative URLs

1 participant