sonots:io

Fluentd-server Released!

Fluentd Server, a Fluentd config distribution server, was released!

What is Fluentd Server

I heard as some users of Fluentd want something like chef-server for Fluentd, so I created the fluentd-server.

With Fluentd Server, you can manage fluentd configuration files centrally with erb.

For example, you may create a config whose name is worker as:

1
2
3
4
5
6
7
8

<source>
  type forward
  port <%= port %>
</source>

<match **>
  type stdout
</match>

Then you can download the config via an API whose uri is like /api/worker?port=24224 where its query parameters are replaced with variables in the erb. The downloaded contents should become as follows:

1
2
3
4
5
6
7
8

<source>
  type forward
  port 24224
</source>

<match **>
  type stdout
</match>

How To Use From Fluentd

The include directive of fluentd config supports http, so write just one line on your fluentd.conf as:

1
2

# /etc/fluentd.conf
include http://fqdn.to.fluentd-server/api/:name?port=24224

where :name is the name of your config post. Fluentd downloads the configuration from the Fluentd Server.

You can see the specification detail of API at API.md.

Demo

I deployed it on heroku, so you can see the demo at http://fluentd-server.herokuapp.com.

All the end

Fluentd server is yet under development. Patches are welcome!

Norikra vs. InfluxDB

Today, I give a comparison between Norikra and InfluxDB in terms of query features.

Introduction – What is Norikra, and What is InfluxDB

Norikra is a Schema-less Stream Processor based on Esper, which is a kind of CEP (Complex Event Processing) engine. With Norikra, we can use highly-functional SQL-like query for processing time series streaming data.

InfluxDB is a distributed time series database, which has a SQL-like query language designed for working with time series and analytics. It has a feature called “Continuous Query” which applies queries charged in advance, and stores the resulted data into a Series (which is like a “table” of RDBMS) progressively.

It looks both can do similar things.

So, I compared Norikra and InfluxDB in terms of query features. I felt that it would be nice to use InfluxDB if it has sufficent query features because it can also be used for data persistence.

TL; DR

Norikra won

Feature Catalog

I just write catalog tables here.

SQL Fuctions

Function Norikra InfluxDB NOTE
COUNT YES YES
MIN YES YES
MAX YES YES
AVG YES YES
MEAN YES YES
MODE NO YES
MEDIAN NO YES
DISTINCT YES YES
PERCENTILE YES YES Norikra implemented as a UDF
HISTOGRAM NO YES
DERIVATIVE NO YES
SUM YES YES
STDDEV NO YES
FIRST YES YES
LAST YES YES
MAXBY YES NO
MINBY YES NO

NOTE: Norikra supports UDF, so users can create functions by themselves if they want

SQL Features

Feature Norikra InfluxDB NOTE
Time batch window YES YES Of course, both support the feature to process periodically
Externally timed batch window YES NO A feature to process data based on the time field of messages
Sub query YES NO
JOIN YES YES
MERGE NO YES A feature of InfluxDB to merge results from multiple tables. One can use regular expressions to specify tables
Query group YES NO Make a group for queries
Program codes YES NO One can write Java codes on Norikra queries
Nested JSON YES NO One can specify fields like parrent.child when JSON is nested
GROUP BY YES YES
HAVING YES NO
ORDER BY YES NO InfluxDB has order, but it is only for the time field
UDF YES NO

Conclusion

Norikra won overwhelmingly. That said, because InfluxDB is a storage, but Norikra is an on-memory engine, the value of InfluxDB shall not be impaired in any way.

How to Use Norikra to Count HTTP Status With Fluentd

In this article, I exaplain how to use Norikra to count HTTP status with Fluentd. Previously, it was common to use fluent-plugin-datacounter for the purpose, but, today I will explain how to achieve the same feature with Norikra.

REMARK: This article is a short translated version of Japanese article written at http://blog.livedoor.jp/sonots/archives/37921050.html

What Not Covered

In this article, I will not cover, how to install Norikra, how to install Fluentd.

What is Norikra, and Why to Use Norikra

Norikra is a Schema-less Stream Processor based on Esper, which is a kind of CEP (Complex Event Processing) engine. With Norikra, we can use highly-functional SQL-like query for processing streaming data.

We can achieve similar things by using plugins like fluent-plugin-datacounter. However, a Fluentd process can utilize only one CPU core because Fluentd uses CRuby, in contrast, a Norikra process can utilize multiple CPU cores because Norikra uses JVM (Esper is wirtten with Java, and Norikra is written with JRuby). This is the 1st reason to use Norikra. This is attractive especially when we need to processs heavy log data where one CPU core is not sufficient to process it in real-time.

The 2nd reason to use Norikra is because it does not require to restart for adding or removing queries. Fluentd requires us to restart its processes when we change their configuration files to add or remove new settings. I need the graceful restart.

Data Counting using fluent-plugin-datacounter

Let me describe how to perform data counting using fluent-pugin-datacounter first. We will replace this to Norikra later.

Requirement

Assume there is a demand as followings:

  1. Want to count HTTP status codes in a specified time interval by seeing status fields written in logs.
  2. Want to count the status code for each host.
  3. Want to count the status code as an aggregation of all hosts.

Specification of Input Data (Log)

Assume that log data are sent from Fluentd agent with tags like “visualizer.logname.hostname” where logname stands for an arbitrary string which operation engineers name to distinguish logs.

Also, assume that messages contain time, status, reqtime, method, uri fields.

Example) visualizer.api_restful.host001

1

{"status":"403","reqtime":"0.123","method":"GET","uri":"http://example.com/v1/restful/people?id=1"}

Configuration

The status code counting can be achieved with the following configuration using fluent-plugin-datacounter.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

<source>
  type forward
  port 24224
</source>
<match visualizer.api_restful.*>
  type copy
  # for each host
  <store>
    type record_reformer
    enable_ruby false
    tag ${tag_prefix[-2]}.hosts.${tag_parts[-1]}
  </store>
  # for all
  <store>
    type record_reformer
    enable_ruby false
    tag ${tag_prefix[-2]}.all.all
  </store>
</match>
<match visualizer.api_restful.{hosts,all}.*>
  type datacounter
  count_interval 60
  aggregate tag
  tag_prefix status_count
  count_key status
  pattern1 2xx ^2..$
  pattern2 3xx ^3..$
  pattern3 400 ^400$
  pattern4 4xx ^4..$
  pattern5 503 ^503$
  pattern6 5xx ^5..$
  output_per_tag yes
</match>

The output messages become as followings (unnecessary fields are ommitted):

1
2

status_count.visualizer.api_restful.hosts.host001:
{"2xx_count":5,"3xx_count":0,"400_count":0,"4xx_count":0,"503_count":0,"5xx_count":0}

Let us think of replacing this with Norikra.

Data Counting using Norikra

Referring the document of Norikra, and README of fluent-plugin-norikra, we will try to replace fluent-plugin-datacounter with out_norikra, Norikra Query (accurately, EPL of Esper), and in_norikra.

in_norikra

Configure Fluentd to receive data, and transfer the data to Norikra.

This configuration sets a target of Norikra (which are like the table of RDBMS) to be a logname of the “visualizer.logname.hostname” tag. Also, this sets hostname to the host field of messages by extracting it from tags.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

<source>
  type forward
  port 24224
</source>

<match visualizer.api_restful.**>
  type record_reformer
  enable_ruby false
  tag reformed.${tag_prefix[-2]} # Remove the hostname from the tag
  <record>
    host ${tag_parts[-1]} # Set the hostname to `host` field to use in GROPU BY statement.
  </record>
</match>

<match reformed.visualizer.api_restful>
  type norikra
  norikra localhost:26571
  remove_tag_prefix reformed.visualizer
  target_map_tag true # target will be `api_restful`
  <default>
    auto_field false  # norikra includes fields only used in queries.
  </default>
</match>

Norikra Query

Surprisingly, we can write Java code in a Norikra query. So, we can realize same conditions with the case of fluent-plugin-datacounter by using String#matches. Also, we can take couting for each host using GROUP BY statement. Cool.

EDIT: I replaced COUNT(1, status.matches('^2..$')) to COUNT(1, status REGEXP '^2..$') because the master tagomoris, the author of Norikra, said the latter is better in performance. Thinking of performance, replacing REGEXP to LIKE, or converting status field to interger and using COUNT(1, status / 100 = 2) would achieve better performances.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

# creation of a target
norikra-client target open api_restful

# all. `AS 2xx_count` was NG, so I made `As count_2xx`.
norikra-client query add status_count.all.api_restful "$(cat <<EOF
SELECT \
COUNT(1, status REGEXP '^2..$') AS count_2xx, \
COUNT(1, status REGEXP '^3..$') AS count_3xx, \
COUNT(1, status REGEXP '^400$') AS count_400, \
COUNT(1, status REGEXP '^4(?!00)..$') AS count_4xx, \
COUNT(1, status REGEXP '^503$') AS count_503, \
COUNT(1, status REGEXP '^5(?!03)..$') AS count_5xx \
FROM api_restful.win:time_batch(60 sec)
EOF
)"
# each host
norikra-client query add status_count.host.api_restful "$(cat <<EOF
SELECT \
host, \
COUNT(1, status REGEXP '^2..$') AS count_2xx, \
COUNT(1, status REGEXP '^3..$') AS count_3xx, \
COUNT(1, status REGEXP '^400$') AS count_400, \
COUNT(1, status REGEXP '^4(?!00)..$') AS count_4xx, \
COUNT(1, status REGEXP '^503$') AS count_503, \
COUNT(1, status REGEXP '^5(?!03)..$') AS count_5xx \
FROM api_restful.win:time_batch(60 sec) \
GROUP BY host
EOF
)"

Notice that the query can be also constructed as followings:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# 2xx
$ norikra-client query add status_count.all.2xx.api_restful \
  "SELECT COUNT(status) AS count_2xx \
  FROM api_restful.win:time_batch(60 sec) \
  WHERE status REGEXP '^2..$'"
# 3xx
$ norikra-client query add status_count.all.3xx.api_restful \
  "SELECT COUNT(status) AS count_3xx \
  FROM api_restful.win:time_batch(60 sec) \
  WHERE status REGEXP '^3..$'"
# 400 
# 4xx
# 503
# 5xx

but, I felt preparing many queries make difficult to manage them. So, I prefered the former COUNT(1, status REGEXP '^2..$') way.

in_norikra

We configure Fluentd to retrieve results of Norikra in each 60 seconds. Here, we set tag query_name so that the tag of retrieved messages to be the query name which we specified on query add commands.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

<source>
  type norikra
  norikra localhost:26571
  <fetch>
    method event
    target status_count.all.api_restful # query name
    tag query_name
    # tag string api_restful.status_count.all
    # tag field field_name
    interval 60s
  </fetch>
  <fetch>
    method event
    target status_count.host.api_restful # query name
    tag query_name
    interval 60s
  </fetch>
</source>

Done!

The output wil be as belows:

1
2

status_count.host.api_restful:
{"host":"host001","count_2xx":5,"count_3xx":0,"count_400":0,"count_4xx":0,"count_503":0,"count_5xx":0}

Conclusion

I explained how to replace fluent-plugin-datacounter with Norikra. Try it out!