使用clojure和flambo过滤RDDs

Question

我有一个包含表单索引的RDD：(:rdd xctx)

[[["1" "32" "44" "55" "14"] 0] [["21" "23" "24" "25" "24"] 1] [["41" "53" "54" "5" "24"] 2] [["11" "35" "34" "15" "64"] 3]]

我想过滤掉在向量中包含索引的RDDs，例如：

:row-list s[1 3] 

我试过了，但不知怎的，我犯了个错误：

(defn remove-index-rows
 "Function to catch the row(s) with the specific Row Number(s) in rows-list
  input = { :rows-list [ val(s)]}"
  [row input]
  (let [{:keys [ rows-list ]} input
    row-and-index (f/collect (f/filter #(= row (get % 0)) (:rdd xctx)))]
    (when-not (some #(= (get row-and-index 1) %) rows-list) row)))

期望的产出是：

 [ [["1" "32" "44" "55" "14"] 0] [["41" "53" "54" "5" "24"] 2] ]

谢谢你的帮助

Answer 1

对于星星，我会用set替换行列表。让我们将其定义如下

(set row-list)

在此之后，您可以像这样进行过滤：

(f/filter
 (:rdd xctx)
 (f/fn [row] (let [[v i] row] (not (contains? row-set i)))))