基于Enlive和Clojure的MLA引用的HTML提取

Question

我的目标是从网页中提取和解析一系列参考书目，以便以后进入数据库。参考资料均采用MLA格式。这应该是一个通用的解决方案，对于所有的MLA格式的书目，并且应该工作的不仅仅是下面所示的网页。

下面是我的尝试代码，它不起作用：

(use '[net.cgrand.enlive-html])

(def ^:dynamic *base-url* "https://www.impacttest.com/research/?Clinical-Research-Database-4")
(def ^:dynamic *ref-selector*     [:div#content_1 :ul :li])


(defn fetch-url [url]
  (html-resource (java.net.URL. url)))

(defn references []
  (select (fetch-url *base-url*) *ref-selector*))

(def ^:dynamic *ref-regex*    #"\s([A-Z]{1}[\w|\s]+)[,|\.]")
(def ^:dynamic *ref-modifier* `(remove :content))

(defmacro extract-re [node re modifier]
  `(doseq [seqs (map :content (node))]
    (re-find re (apply str (modifier seqs)))))

(extract-re references *ref-regex* *ref-modifier*)

(macroexpand-1 '(extract-re references *ref-regex* *ref-modifier*))

我希望宏extract-re创建一个在所有活动节点上运行正则匹配器( regex，re-find)的doseq。有两个变量需要更改:一个是regex本身，另一个是修饰符，它在处理enlive节点之前对其进行修改。没有修饰符，正则表达式将匹配作者和一些标题。我试着编写一个函数，但在一般情况下无法让它工作，所以我认为宏是可行的。

在MLA引用上，我认为在enlive节点上使用修饰符比使用regex进行所有提取要容易，尽管我可能错了。我想不出如何做一个只与标题或作者匹配的正则表达式。

因此，如何将修饰符传递给宏并使其正确执行？我不完全理解宏的引用细节，所以我可能对我如何开始编写宏，或者即使需要宏也不太了解。

Answer 1

为了说明..。

注意:我将以html作为前缀。

(require '[net.cgrand.enlive-html :as html])

(references)的输出是单个引用元素的序列，如

(def data-sample 
  '{:tag :li, :attrs nil, 
    :content 
    ("\n\t\t\t\t\t\t\t\t\t\t\t\t\t" 
      {:tag :strong, :attrs nil, 
       :content 
       ("AMPAR peptide values *snip*.")}
      " Dambinova SA, Shikuev, Weissman JD, Mullins, JD. "
      {:tag :em, :attrs nil, :content ("Military Medicine.")}
      " 2013, 178 (3):285-290.\t\t\t\t\t\t\t\t\t\t\t\t")})

您会注意到文章标题是粗体，期刊是斜体，所以我们至少可以使用选择器来提取这些内容。但是，由于格式的更改用于提供组件的可视化分离，因此它们也将在数据中提供分离。

(defn trimmed-text-only [html-data] 
  (as-> html-data x
    (html/select x [html/text-node])
    (map clojure.string/trim x)
    (remove empty? x)))

(trimmed-text-only data-sample)
;=> 
("AMPAR peptide values *snip*." 
 "Dambinova SA, Shikuev, Weissman JD, Mullins, JD." 
 "Military Medicine." 
 "2013, 178 (3):285-290.")

这已经使组件变得明显，但是请注意，每个组件都由一个句点分隔，而不是在组件中使用句点。因此，我们也可以完全忽略格式设置，并按句点进行拆分，并增加删除这些句点的好处。

(defn extract-major-reference-components
  [html-data]
  (as-> html-data x
    (trimmed-text-only x)
    (apply str x)
    (clojure.string/split x #"\.")
    (zipmap [:title :authors :journal :issue-ref] x)))

(extract-major-reference-components data-sample)
;=> 
{:title "AMPAR peptide values *snip*"
 :authors "Dambinova SA, Shikuev, Weissman JD, Mullins, JD",
 :journal "Military Medicine",
 :issue-ref "2013, 178 (3):285-290"}

现在，您可以将此提取函数映射到引用序列上。使用输出映射，您可以使用update和regexp进行进一步的转换，例如将单个作者或年份、发行号和页面与问题-引用分离。

Answer 2

这段代码有很多问题。

'(use [net.cgrand.enlive-html])

这不会引入一个库，它会创建一个文字列表，并且不会对它做任何事情：

user> (class '(use [net.cgrand.enlive-html]))
clojure.lang.PersistentList

这实际上是一种禁止行动。

(def ^:dynamic *ref-modifier* `(remove :content))

这将创建一个两个元素列表，而不是任何类型的“修饰符”。

(defmacro extract-re [node re modifier]
  `(doseq [seqs (map :content (node))]
    (re-find re (apply str (modifier seqs)))))

在这里，您使用语法引号，但您从不取消引用里面的任何内容。宏不以任何方式使用它的任何参数。

您似乎希望像应用函数一样应用modifier (这甚至还没有开始发生，请参阅上面的引用问题)，但是正如我们在实际调用中看到的那样，modifier是一个两个元素列表，如果调用将导致错误。

最后，doseq只对副作用起作用，并且总是返回零。doseq块不使用re-find生成的值，因此doseq体实际上是一个无操作。

此外，我还看到了对将作为显式函数参数提供的var使用动态var声明的可疑效用。

随着所有这些问题的解决，我认为我们更接近于一些行之有效的东西：

(use 'net.cgrand.enlive-html)

(def ^:dynamic *base-url*
  "https://www.impacttest.com/research/?Clinical-Research-Database-4")

(def ^:dynamic *ref-selector* [:div#content_1 :ul :li])


(defn fetch-url [url]
  (html-resource (java.net.URL. url)))

(defn references []
  (select (fetch-url *base-url*) *ref-selector*))

(def ^:dynamic *ref-regex* #"\s([A-Z]{1}[\w|\s]+)[,|\.]")

(def ^:dynamic *ref-modifier* (partial remove :content))

(defn extract-re [node re modifier]
  (doall
    (for [sq (map :content (node))]
      (re-find re (apply str (modifier sq))))))

在行动中：

user> (extract-re references *ref-regex* *ref-modifier*)

/*
* 提示：该行代码过长，系统自动注释不进行高亮。一键复制会移除系统注释 
* ([" Dambinova SA," "Dambinova SA"] [" Zuckerman SL," "Zuckerman SL"] [" Conklin HM," "Conklin HM"] [" Covassin T," "Covassin T"] [" Maerlender A," "Maerlender A"] [" Fedor A," "Fedor A"] [" Resch J," "Resch J"] [" Elbin RJ," "Elbin RJ"] [" Rabinowitz AR," "Rabinowitz AR"] [" Kinnaman KA," "Kinnaman KA"] [" Tsushima WT," "Tsushima WT"] [" Amonette WE," "Amonette WE"] [" Lovell MR," "Lovell MR"] [" Schatz P," "Schatz P"] [" McGrath N," "McGrath N"] [" Kontos AP," "Kontos AP"] [" AB," "AB"] [" Meehan WP," "Meehan WP"] [" Rieger BP," "Rieger BP"] [" Solomon GS," "Solomon GS"] [" Sandel NK," "Sandel NK"] [" Schatz P," "Schatz P"] [" Schatz P," "Schatz P"] [" Lebrun CM," "Lebrun CM"] [" Brooks B," "Brooks B"] [" Meehan WP," "Meehan WP"] [" Fakhran S," "Fakhran S"] [" Cole WR," "Cole WR"] [" Tsushima M," "Tsushima M"] [" Zuckerman SL," "Zuckerman SL"] [" JK," "JK"] [" Covassin T," "Covassin T"] [" Moser RS," "Moser RS"] [" Mayers LB," "Mayers LB"] [" McAllister TW," "McAllister TW"] [" Meehan WP 3rd," "Meehan WP 3rd"] [" Neal MT," "Neal MT"] [" Lau BC," "Lau BC"] [" Kontos AP," "Kontos AP"] [" Gardner A," "Gardner A"] [" Elbin RJ," "Elbin RJ"] [" Wolf EG," "Wolf EG"] [" Reddy CC," "Reddy CC"] [" Moser RS," "Moser RS"] [" Guerriero RM," "Guerriero RM"] [" Deibert E," "Deibert E"] [" Wiebe DJ," "Wiebe DJ"] [" Baillargeon A," "Baillargeon A"] [" Erdal K." "Erdal K"] [" Maugans TA," "Maugans TA"] [" Iverson GL," "Iverson GL"] [" Ponsford J," "Ponsford J"] [" Schatz P," "Schatz P"] [" Mulligan I," "Mulligan I"] [" Echlin PS," "Echlin PS"] [" McLeod TC," "McLeod TC"] [" Zuckerman SL," "Zuckerman SL"] [" Kontos AP," "Kontos AP"] [" Zuckerman SL," "Zuckerman SL"] [" Schatz P," "Schatz P"] [" Kontos AP," "Kontos AP"] [" Covassin T," "Covassin T"] [" Covassin T," "Covassin T"] [" Duhaime AC," "Duhaime AC"] [" Echemendia RJ," "Echemendia RJ"] [" Ramanathan DM," "Ramanathan DM"] [" Meehan WP 3rd," "Meehan WP 3rd"] [" Krol AL," "Krol AL"] [" Turgeon C," "Turgeon C"] [" Randolph C." "Randolph C"] [" Barlow M," "Barlow M"] [" Schatz P," "Schatz P"] [" Moser RS," "Moser RS"] [" Broglio SP," "Broglio SP"] [" Thomas DG," "Thomas DG"] [" Allen BJ," "Allen BJ"] [" Solomon GS," "Solomon GS"] [" Ponsford J," "Ponsford J"] [" Johnson EW," "Johnson EW"] [" Randolph C," "Randolph C"] [" Elbin RJ," "Elbin RJ"] [" Broglio SP," "Broglio SP"] [" Kontos AP," "Kontos AP"] [" Lau BC," "Lau BC"] [" Lau BC," "Lau BC"] [" Hettich T," "Hettich T"] [" Elbin T," "Elbin T"] [" Maerlender A," "Maerlender A"] [" Kontos AP," "Kontos AP"] [" Talavage TM," "Talavage TM"] [" Meehan WP 3rd," "Meehan WP 3rd"] [" Lange RT," "Lange RT"] [" Covassin T," "Covassin T"] [" Schatz P." "Schatz P"] [" Lange RT," "Lange RT"] [" Pardini JE," "Pardini JE"] [" Echlin PS," "Echlin PS"] [" Schatz P," "Schatz P"] [" Echlin PS," "Echlin PS"] [" Keightley ML," "Keightley ML"] [" McGrath N." "McGrath N"] [" Covassin T," "Covassin T"] [" Pontifex MB," "Pontifex MB"] [" AB," "AB"] [" Casson IR," "Casson IR"] [" McCrory P," "McCrory P"] [" Covassin T," "Covassin T"] [" Bruce JM," "Bruce JM"] [" Covassin T," "Covassin T"] [" Lovell M." "Lovell M"] [" Lau B," "Lau B"] [" Nance ML," "Nance ML"] [" Peterson SE," "Peterson SE"] [" Lovell M." "Lovell M"] [" Broglio SP," "Broglio SP"] [" Broglio SP," "Broglio SP"] [" Colvin AC," "Colvin AC"] [" Reddy CC," "Reddy CC"] [" Solomon GS," "Solomon GS"] [" Covassin T," "Covassin T"] [" Majerske CW," "Majerske CW"] [" Lovell MR," "Lovell MR"] [" AB," "AB"] [" Tsushima WT," "Tsushima WT"] [" Miller JR," "Miller JR"] [" Slobounov S," "Slobounov S"] [" Mihalik JP," "Mihalik JP"] [" Covassin T," "Covassin T"] [" Lovell MR," "Lovell MR"] [" Stoller KP." "Stoller KP"] [" Broglio SP," "Broglio SP"] [" Moser RS," "Moser RS"] [" Iverson G." "Iverson G"] [" Fazio VC," "Fazio VC"] [" Swanik CB," "Swanik CB"] [" Broglio SP," "Broglio SP"] [" Covassin T," "Covassin T"] [" Broglio SP," "Broglio SP"] [" Chen JK," "Chen JK"] [" Van Kampen DA," "Van Kampen DA"] [" Broglio SP," "Broglio SP"] [" Pellman EJ," "Pellman EJ"] [" Pellman EJ," "Pellman EJ"] [" Schatz P," "Schatz P"] [" Biasca N," "Biasca N"] [" Collins M," "Collins M"] [" Lovell MR," "Lovell MR"] [" Lovell MR," "Lovell MR"] [" Iverson GL," "Iverson GL"] [" Cantu RC," "Cantu RC"] [" McClincy MP," "McClincy MP"] [" Schatz P," "Schatz P"] [" Iverson GL," "Iverson GL"] [" Van Kampen DA," "Van Kampen DA"] [" Lovell M," "Lovell M"] [" Mihalik JP," "Mihalik JP"] [" Moser RS," "Moser RS"] [" Broshek DK," "Broshek DK"] [" Grove R," "Grove R"] [" McCrea M," "McCrea M"] [" McCrory P," "McCrory P"] [" Iverson GL," "Iverson GL"] [" Lovell MR," "Lovell MR"] [" Bruce JM," "Bruce JM"] [" Pellman EJ," "Pellman EJ"] [" Iverson GL," "Iverson GL"] [" Lovell MR," "Lovell MR"] [" Kontos A," "Kontos A"] [" Collins MW," "Collins MW"] [" Iverson GL," "Iverson GL"] [" Lovell M," "Lovell M"] [" Field M," "Field M"] [" Covassin T," "Covassin T"] [" Iverson GL," "Iverson GL"] [" Lovell MR," "Lovell MR"] [" Collins MW," "Collins MW"] [" Lovell MR," "Lovell MR"] [" Collins MW," "Collins MW"] [" Collins MW," "Collins MW"] [" Collins MW," "Collins MW"] [" Maroon JC," "Maroon JC"] [" Lovell MR," "Lovell MR"] [" Lovell MR." "Lovell MR"] [" Aubry M," "Aubry M"] [" Grindel SH," "Grindel SH"] [" Collins MW," "Collins MW"] [" Lovell MR," "Lovell MR"] [" Collins MW," "Collins MW"] [" Lovell MR," "Lovell MR"])
*/