Norikra 試用紀錄

Norikra is a open source server software provides "Stream Processing" with SQL, written in JRuby, runs on JVM, licensed under GPLv2.

1. install JRuby (downloads)

# tar xzvf jruby-bin-1.7.15.tar.gz
# mv jruby-1.7.15 /opt
# echo "PATH=$PATH:/opt/jruby-1.7.15/bin" > ~/.bashrc
# jruby -v
jruby 1.7.15 (1.9.3p392) 2014-09-03 82b5cc3 on Java HotSpot(TM) 64-Bit Server VM 1.6.0_33-b04 +jit [linux-amd64]

2. install Norikra (要等個幾分鐘才會開始裝)

# gem install norikra

3. start Norikra (要等個幾分鐘才會啟動成功)

# norikra start

4. Norikra web UI ：http://IP:26578
5. 以 web service 的 log 為例，首先定義 target 的欄位 (minimal fields set for variations of 'www' events)，定義最基本的欄位就好，可再修改

# norikra-client target open www path:string status:integer referer:string agent:string userid:integer
# norikra-client target list
TARGET  AUTO_FIELD
www     true
1 targets found.

6. 加入 query 指令：查詢首頁(path="/")被存取的次數

# norikra-client query add www.toppage 'SELECT count(*) AS cnt FROM www WHERE path="/" AND status=200'

7. 加入 query 指令 with Views (Windows)。Views are used to define time windows or to control timing to fire events. With views, norikra controls the range of events for each queries.

# norikra-client query add www.toppageviews 'SELECT count(*) AS cnt FROM www.win:time_batch(3 sec) WHERE path="/" AND status=200'

8. 開始將資料送到 norikra
• script

#!/bin/sh
for i in `seq 1 1 3`;
do
  data1='{"path":"/", "status":200, "referer":"", "agent":"MSIE", "userid":3}'
  echo `date +"%Y/%m/%d-%H:%M:%S"` $data1
  echo $data1 | norikra-client event send www
  sleep 3
  #
  data2='{"path":"/login", "status":301, "referer":"/", "agent":"MSIE", "userid":3}'
  echo `date +"%Y/%m/%d-%H:%M:%S"` $data2
  echo $data2 | norikra-client event send www
  sleep 3
  #
done

• log

2014/09/17-23:07:58 {"path":"/", "status":200, "referer":"", "agent":"MSIE", "userid":3}
2014/09/17-23:08:13 {"path":"/login", "status":301, "referer":"/", "agent":"MSIE", "userid":3}
2014/09/17-23:08:29 {"path":"/", "status":200, "referer":"", "agent":"MSIE", "userid":3}
2014/09/17-23:08:45 {"path":"/login", "status":301, "referer":"/", "agent":"MSIE", "userid":3}
2014/09/17-23:09:00 {"path":"/", "status":200, "referer":"", "agent":"MSIE", "userid":3}
2014/09/17-23:09:16 {"path":"/login", "status":301, "referer":"/", "agent":"MSIE", "userid":3}

9. 查看 query 結果

# norikra-client event see www.toppage
{"time":"2014/09/17 23:08:10","cnt":1}
{"time":"2014/09/17 23:08:41","cnt":2}
{"time":"2014/09/17 23:09:13","cnt":3}

# norikra-client event see www.toppageviews
{"time":"2014/09/17 23:08:13","cnt":1}
{"time":"2014/09/17 23:08:16","cnt":0}
{"time":"2014/09/17 23:08:43","cnt":1}
{"time":"2014/09/17 23:08:46","cnt":0}
{"time":"2014/09/17 23:09:16","cnt":1}
{"time":"2014/09/17 23:09:19","cnt":0}

10. 取得 query 結果，取到的結果會被刪除，如沒有新的事件發生，再查看 query 結果是沒有資料的

# norikra-client event fetch www.toppageviews
{"time":"2014/09/17 23:08:13","cnt":1}
{"time":"2014/09/17 23:08:16","cnt":0}
{"time":"2014/09/17 23:08:43","cnt":1}
{"time":"2014/09/17 23:08:46","cnt":0}
{"time":"2014/09/17 23:09:16","cnt":1}
{"time":"2014/09/17 23:09:19","cnt":0}

# norikra-client event fetch www.toppageviews

心得：

• 易上手，定義 data set 後，寫少少的程式(SQL)就可開始做資料處理。
• 具備排程功能，資料處理的動作可以 run forever，可設定對全部的資料做處理或者一次處理一段時間內的資料，因資料皆存在 memory 中，對全部的資料做處理時要注意 out of memory 的問題。
• 安裝簡單，單機作業，無資料分散 or 運算分散的架構(ex: storm)。
• 目前無考慮 HA，作者說也不需要考慮。
• 想想適合用在哪裡？複雜多樣的小資料收集與前處理，定期 fetch 處理結果 or 一前處理完就往其他地方匯出，清空主機 memory。

參考資料：

• Norikra 官網
• Installing jRuby for Centos