Clojure Cassandra驱动程序性能

Question

我在一个6节点的Cassandra集群上测试Clojure中Alia的性能。即使是在多线程的情况下，我也只能达到每秒400次写入。使用激动人心的Cassandra驱动程序和在Java中手动处理线程，我们能够在96个线程的情况下获得每秒5000次写入。

我在这里使用代理的过程中做错了什么吗？在运行它的机器上，CPU使用率只有大约25%，这看起来真的很低。

更新:在Alia的作者的建议下，使用准备好的语句而不是原始语句，以同步、单线程的方式实现了高达2500/秒的收益。我仍然需要通过使用Clojure的多线程和单独利用Alia/底层Java驱动程序中内置的异步函数来测试这一点，看看哪个更快。

更新2:通过额外利用驱动程序内置的异步功能，我现在看到了与下面的mpenet类似的结果。

(ns alia-perf-test.core
  (:gen-class)
  (:require [qbits.alia :as alia]
            [qbits.hayt :as hayt]))

(defn exec-query [session query]
  (alia/execute session (hayt/->raw query)))

(defmacro time-query
  [expr]
  `(let [start# (. System (nanoTime))
         ret# ~expr]
     (/ (double (- (. System (nanoTime)) start#)) 1000000.0)))

(defn write-entity
  [total-time session entity]
  (let [query (hayt/->raw (hayt/insert :entities (hayt/values entity) (hayt/using :timestamp 1234)))
        query-time (time-query (alia/execute session query))]
      (+ total-time query-time)))

(defn generate-entity []
  {:id (str (java.util.UUID/randomUUID)) :num 0})

(defn write-something
  [write-agent session]
  (send-off write-agent write-entity
        session 
        (generate-entity)))

(defn -main [& args]
  (let [cluster (alia/cluster ["server1"
                               "server2" 
                               "server3" 
                               "server4" 
                               "server5" 
                               "server6"]
                              :pooling-options {:core-connections-per-host [:local 16 :remote 16]
                                                :max-connections-per-host  [:local 1000 :remote 1000]
                                                :max-simultaneous-requests-per-connection [:local 32 :remote 32]
                                                :min-simultaneous-requests-per-connection [:local 16 :remote 16]})
        session (alia/connect cluster)]
    (alia/set-consistency! :any)
    (exec-query session (hayt/create-keyspace :aliaperftest
                                              (hayt/with {:replication
                                                          {:class "NetworkTopologyStrategy"
                                                           :dc1 3 :dc2 3}})))
    (exec-query session (hayt/use-keyspace :aliaperftest))
    (exec-query session (hayt/create-table :entities
                                           (hayt/column-definitions {:id :varchar
                                                                     :num :int
                                                                     :primary-key [:id]})))
    (let [num-entities 10000
          write-agent (agent 0)]
      (dotimes [n num-entities]
        (write-something write-agent session))
      (await write-agent)
      (println "Wrote" num-entities "entities in" @write-agent "ms -"
           (* (/ num-entities @write-agent) 1000.0) "ops/sec"))

    (exec-query session (hayt/drop-table :entities))
    (exec-query session (hayt/drop-keyspace :aliaperftest))
    (alia/shutdown session)
    (alia/shutdown cluster)
    (shutdown-agents)))

Answer 1

更新:我已经让它在异步模式下在大约8秒内完成100k请求(使用alia的成功处理程序，只有一个节点)，使用一个atom接收结果，并等待所有响应到达。

根据您的设置和批处理，也可以通过使用自定义执行器来获得更高的性能，我并没有走得那么远。请参阅：https://github.com/mpenet/alia/blob/master/docs/guide.md#executors