Add offline HTML parsing module to LyngIO
This commit is contained in:
1 parent
66ab7db3d8
commit
26a1ca9710
9 files changed
+240
-1
No files matched your search
@@ -98,6 +98,7 @@ Requires installing `lyngio` into the import manager from host code.
|
||||
- `import lyng.io.ws` (WebSocket client API; currently supported on JVM, capability-gated elsewhere)
|
||||
- `import lyng.io.net` (TCP/UDP transport API; currently supported on JVM, capability-gated elsewhere)
|
||||
- `import lyng.io.html` (pure Lyng HTML builder DSL: `html { body { h3 { +"text" } } }`)
|
||||
- `import lyng.io.html.parse` (offline HTML5 parsing: `Html.parse(source, baseUrl = "")`, document title/text, CSS selectors, element text/attributes/absolute URLs; see [HTML parsing](lyng.io.html.parse.md))
|
||||
- Shared network value-type packages are also available when installed by host code:
|
||||
- `import lyng.io.http.types` (`HttpHeaders`)
|
||||
- `import lyng.io.ws.types` (`WsMessage`)
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# lyng.io.html
|
||||
|
||||
For reading existing HTML, see [lyng.io.html.parse](lyng.io.html.parse.md).
|
||||
|
||||
`lyng.io.html` provides a pure Lyng HTML builder DSL. It uses Lyng context
|
||||
receiver extensions, so text can be appended with `+"text"` inside tag blocks
|
||||
without global builder state.
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
# lyng.io.html.parse
|
||||
|
||||
Offline HTML5 parsing backed by the core [Ksoup](https://github.com/fleeksoft/ksoup)
|
||||
parser (no network extensions). Installed from `lyngio`:
|
||||
|
||||
```kotlin
|
||||
createHtmlParseModule(scope.importManager)
|
||||
```
|
||||
|
||||
```lyng
|
||||
import lyng.io.html.parse
|
||||
|
||||
val doc = Html.parse("<title>News</title><main><p>Hello & welcome</p><a href='/next'>Next</a></main>", "https://example.com/")
|
||||
println(doc.title)
|
||||
println(doc.text())
|
||||
for (raw in doc.select("a[href]")) {
|
||||
val link = raw as HtmlElement
|
||||
println(link.text() + " => " + link.absUrl("href"))
|
||||
}
|
||||
```
|
||||
|
||||
`Html.parse(source, baseUrl = "")` accepts documents, fragments and malformed HTML.
|
||||
It returns `HtmlDocument`, with `title: String`, `text(): String` and
|
||||
`select(css): List<HtmlElement>`. Elements have `text()`, `attr(name)`,
|
||||
`absUrl(name)` and `select(css)`. Missing selections return an empty list;
|
||||
missing attributes or unresolvable absolute URLs return an empty string.
|
||||
Invalid CSS selectors raise an error. `<base href>` participates in URL resolution.
|
||||
Use the final fetched URL as the base when available.
|
||||
|
||||
Text extraction decodes entities, normalizes whitespace, preserves block and
|
||||
`br` boundaries as newlines, and excludes `script`, `style`, `template`, `noscript`
|
||||
and `[hidden]` subtrees. Selection retains the original DOM. This is readable
|
||||
text extraction, not CSS rendering or article detection; navigation and cookie
|
||||
banners may remain. It does not preserve preformatted whitespace.
|
||||
|
||||
Parsing never fetches resources or executes JavaScript. `absUrl` only resolves
|
||||
URLs; it does not filter schemes. Fetch through the host's authorized HTTP API,
|
||||
check status and content type, and choose bounded excerpts to return to an AI.
|
||||
This API parses HTML only; XML parsing is separate future work.
|
||||
|
||||
Canonical declarations: `lyngio/stdlib/lyng/io/html_parse.lyng`.
|
||||
Reference in new issue
Block a user