如何在Golang中仅从HTML中提取文本？

Question

为了从HTML中提取文本，我使用了一个完整的HTML5-compliant tokenizer and parser，如下所示

    s := `
<p>Links:</p><ul><li><a href="foo">Foo</a><li>
<a href="/bar/baz">BarBaz</a></ul><span>TEXT <b>I</b> WANT</span>
<script type='text/javascript'>
/* <![CDATA[ */
var post_notif_widget_ajax_obj = {"ajax_url":"http:\/\/site.com\/wp-admin\/admin-ajax.php","nonce":"9b8270e2ef","processing_msg":"Processing..."};
/* ]]> */
</script>`

    domDocTest := html.NewTokenizer(strings.NewReader(s))
    for tokenType := domDocTest.Next(); tokenType != html.ErrorToken; {
        if tokenType != html.TextToken {
            tokenType = domDocTest.Next()
            continue
        }
        TxtContent := strings.TrimSpace(html.UnescapeString(string(domDocTest.Text())))
        if len(TxtContent) > 0 {
            fmt.Printf("%s\n", TxtContent)
        }
        tokenType = domDocTest.Next()
    }

但是我得到了这样的结果

Links:
Foo
BarBaz
TEXT
I
WANT
/* <![CDATA[ */
var post_notif_widget_ajax_obj = {"ajax_url":"http:\/\/site.com\/wp-admin\/admin-ajax.php","nonce":"9b8270e2ef","processing_msg":"Processing..."};
/* ]]> */

我不想要CDATA内容。一些想法，如何只获取文本内容？

Answer 1

正如@Eric Pauley所指出的，我关注的是TextTokens & StartTagTokens。以下是我的解决方案

    s := `
<p>Links:</p><ul><li><a href="foo">Foo</a><li>
<a href="/bar/baz">BarBaz</a></ul><span>TEXT <b>I</b> WANT</span>
<script type='text/javascript'>
/* <![CDATA[ */
var post_notif_widget_ajax_obj = {"ajax_url":"http:\/\/site.com\/wp-admin\/admin-ajax.php","nonce":"9b8270e2ef","processing_msg":"Processing..."};
/* ]]> */
</script>`

    domDocTest := html.NewTokenizer(strings.NewReader(s))
    previousStartTokenTest := domDocTest.Token()
loopDomTest:
    for {
        tt := domDocTest.Next()
        switch {
        case tt == html.ErrorToken:
            break loopDomTest // End of the document,  done
        case tt == html.StartTagToken:
            previousStartTokenTest = domDocTest.Token()
        case tt == html.TextToken:
            if previousStartTokenTest.Data == "script" {
                continue
            }
            TxtContent := strings.TrimSpace(html.UnescapeString(string(domDocTest.Text())))
            if len(TxtContent) > 0 {
                fmt.Printf("%s\n", TxtContent)
            }
        }
    }

Answer 2

如果你使用github.com/PuerkitoBio/goquery，很容易实现你想要的东西。

• 首先需要使用document.Find()来标识要删除的元素，在本例中为scripts，因此您需要使用element.Remove()
• Finally，将其从文档中删除使用document.Text()

打印/获取文本

所以，最终的代码应该是

package main

import (
  "fmt"
  "strings"
  "github.com/PuerkitoBio/goquery"
)

func main(){
  s := `<p>Links:</p><ul><li><a href="foo">Foo</a><li><a href="/bar/baz">BarBaz</a></ul><span>TEXT <b>I</b> WANT</span><script type='text/javascript'>/* <![CDATA[ */var post_notif_widget_ajax_obj = {"ajax_url":"http:\/\/site.com\/wp-admin\/admin-ajax.php","nonce":"9b8270e2ef","processing_msg":"Processing..."};/* ]]> */</script>`

  p := strings.NewReader(s)
  doc, _ := goquery.NewDocumentFromReader(p)

  doc.Find("script").Each(func(i int, el *goquery.Selection) {
      el.Remove()
  })

  fmt.Println(doc.Text()) // Links:FooBarBazTEXT I WANT

}