FLUME-1342 adding jmx metrics tables to docs
authorEndre Major <emajor@cloudera.com>
Fri, 23 Nov 2018 10:47:15 +0000 (11:47 +0100)
committerFerenc Szabo <szaboferee@apache.org>
Fri, 23 Nov 2018 10:47:15 +0000 (11:47 +0100)
This PR adds a few tables to the User Guide that describe the metrics
published by sorurces, sinks and channels.
I used simple unix tools to gather the data then I wrote a small utility to
convert it to csv.
Then I used an online converter https://www.tablesgenerator.com/ to generate
the rst tables and then a little manual editing.
I discovered some rst formatting problems in the FlumeUserGuide.rst,
corrected them, too.
It was rather painful process to gather the data and find a decent
representation.
So far this PR only contains the end result. I would be happy to share the
utilities, just don't know what would be the best way.

This closes #242

Reviewers: Denes Arvay, Ferenc Szabo

(Endre Major via Ferenc Szabo)

flume-ng-doc/sphinx/FlumeUserGuide.rst

index 492d90e..df4d5c2 100644 (file)
@@ -1993,7 +1993,7 @@ selector.type                      replicating or multiplexing
 selector.*            replicating  Depends on the selector.type value
 interceptors          --           Space-separated list of interceptors
 interceptors.*
-====================  ===========  =================================================================
+====================  ===========  ================================================================
 
 
 For example, a syslog UDP source for agent named a1:
@@ -2387,9 +2387,7 @@ hdfs.filePrefix         FlumeData     Name prefixed to files created by Flume in
 hdfs.fileSuffix         --            Suffix to append to file (eg ``.avro`` - *NOTE: period is not automatically added*)
 hdfs.inUsePrefix        --            Prefix that is used for temporal files that flume actively writes into
 hdfs.inUseSuffix        ``.tmp``      Suffix that is used for temporal files that flume actively writes into
-hdfs.emptyInUseSuffix   false         If ``false`` an ``hdfs.inUseSuffix`` is used while writing the output. After closing the output
-                                      ``hdfs.inUseSuffix`` is removed from the output file name.
-                                      If ``true`` the ``hdfs.inUseSuffix`` parameter is ignored an empty string is used instead.
+hdfs.emptyInUseSuffix   false         If ``false`` an ``hdfs.inUseSuffix`` is used while writing the output. After closing the output ``hdfs.inUseSuffix`` is removed from the output file name. If ``true`` the ``hdfs.inUseSuffix`` parameter is ignored an empty string is used instead.
 hdfs.rollInterval       30            Number of seconds to wait before rolling current file
                                       (0 = never roll based on time interval)
 hdfs.rollSize           1024          File size to trigger roll, in bytes (0: never roll based on file size)
@@ -4962,6 +4960,182 @@ Monitoring in Flume is still a work in progress. Changes can happen very often.
 Several Flume components report metrics to the JMX platform MBean server. These
 metrics can be queried using Jconsole.
 
+Available Component Metrics
+---------------------------
+
+The following tables show what metrics are available for components. Each component only maintains a
+set of metrics, indicated by an 'x', the unmaintained ones show default values, that is 0.
+These tables tell you where you can expect meaningful data.
+The name of the metrics should be descriptive enough, for more information you have to dig into the
+source code of the components.
+
+Sources 1
+~~~~~~~~~
+
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+|                          | Avro | Exec | HTTP | JMS | Kafka | MultiportSyslogTCP | Scribe |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| AppendAcceptedCount      | x    |      |      |     |       |                    |        |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| AppendBatchAcceptedCount | x    |      | x    | x   |       |                    |        |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| AppendBatchReceivedCount | x    |      | x    | x   |       |                    |        |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| AppendReceivedCount      | x    |      |      |     |       |                    |        |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| ChannelWriteFail         | x    |      | x    | x   | x     | x                  | x      |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| EventAcceptedCount       | x    | x    | x    | x   | x     | x                  | x      |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| EventReadFail            |      |      | x    | x   | x     | x                  | x      |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| EventReceivedCount       | x    | x    | x    | x   | x     | x                  | x      |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| GenericProcessingFail    |      |      | x    |     |       | x                  |        |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| KafkaCommitTimer         |      |      |      |     | x     |                    |        |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| KafkaEmptyCount          |      |      |      |     | x     |                    |        |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| KafkaEventGetTimer       |      |      |      |     | x     |                    |        |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| OpenConnectionCount      | x    |      |      |     |       |                    |        |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+
+Sources 2
+~~~~~~~~~
+
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+|                          | SequenceGenerator | SpoolDirectory | SyslogTcp | SyslogUDP | Taildir | Thrift |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| AppendAcceptedCount      |                   |                |           |           |         | x      |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| AppendBatchAcceptedCount | x                 | x              |           |           | x       | x      |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| AppendBatchReceivedCount |                   | x              |           |           | x       | x      |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| AppendReceivedCount      |                   |                |           |           |         | x      |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| ChannelWriteFail         | x                 | x              | x         | x         | x       | x      |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| EventAcceptedCount       | x                 | x              | x         | x         | x       | x      |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| EventReadFail            |                   | x              | x         | x         | x       |        |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| EventReceivedCount       |                   | x              | x         | x         | x       | x      |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| GenericProcessingFail    |                   | x              |           |           | x       |        |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| KafkaCommitTimer         |                   |                |           |           |         |        |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| KafkaEmptyCount          |                   |                |           |           |         |        |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| KafkaEventGetTimer       |                   |                |           |           |         |        |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| OpenConnectionCount      |                   |                |           |           |         |        |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+
+Sinks 1
+~~~~~~~
+
++------------------------+-------------+------------+---------------+-------+--------+
+|                        | Avro/Thrift | AsyncHBase | ElasticSearch | HBase | HBase2 |
++------------------------+-------------+------------+---------------+-------+--------+
+| BatchCompleteCount     | x           | x          | x             | x     | x      |
++------------------------+-------------+------------+---------------+-------+--------+
+| BatchEmptyCount        | x           | x          | x             | x     | x      |
++------------------------+-------------+------------+---------------+-------+--------+
+| BatchUnderflowCount    | x           | x          | x             | x     | x      |
++------------------------+-------------+------------+---------------+-------+--------+
+| ChannelReadFail        | x           |            |               |       | x      |
++------------------------+-------------+------------+---------------+-------+--------+
+| ConnectionClosedCount  | x           | x          | x             | x     | x      |
++------------------------+-------------+------------+---------------+-------+--------+
+| ConnectionCreatedCount | x           | x          | x             | x     | x      |
++------------------------+-------------+------------+---------------+-------+--------+
+| ConnectionFailedCount  | x           | x          | x             | x     | x      |
++------------------------+-------------+------------+---------------+-------+--------+
+| EventDrainAttemptCount | x           | x          | x             | x     | x      |
++------------------------+-------------+------------+---------------+-------+--------+
+| EventDrainSuccessCount | x           | x          | x             | x     | x      |
++------------------------+-------------+------------+---------------+-------+--------+
+| EventWriteFail         | x           |            |               |       | x      |
++------------------------+-------------+------------+---------------+-------+--------+
+| KafkaEventSendTimer    |             |            |               |       |        |
++------------------------+-------------+------------+---------------+-------+--------+
+| RollbackCount          |             |            |               |       |        |
++------------------------+-------------+------------+---------------+-------+--------+
+
+Sinks 2
+~~~~~~~
+
++------------------------+-----------+------+------+-------+-----------+-------------+
+|                        | HDFSEvent | Hive | Http | Kafka | Morphline | RollingFile |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| BatchCompleteCount     | x         | x    |      |       | x         |             |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| BatchEmptyCount        | x         | x    |      | x     | x         |             |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| BatchUnderflowCount    | x         | x    |      | x     | x         |             |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| ChannelReadFail        | x         | x    | x    | x     | x         | x           |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| ConnectionClosedCount  | x         | x    |      |       |           | x           |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| ConnectionCreatedCount | x         | x    |      |       |           | x           |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| ConnectionFailedCount  | x         | x    |      |       |           | x           |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| EventDrainAttemptCount | x         | x    | x    |       | x         | x           |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| EventDrainSuccessCount | x         | x    | x    | x     | x         | x           |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| EventWriteFail         | x         | x    | x    | x     | x         | x           |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| KafkaEventSendTimer    |           |      |      | x     |           |             |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| RollbackCount          |           |      |      | x     |           |             |
++------------------------+-----------+------+------+-------+-----------+-------------+
+
+Channels
+~~~~~~~~
+
++---------------------------------+------+-------+--------+-----------------+-----------------+
+|                                 | File | Kafka | Memory | PseudoTxnMemory | SpillableMemory |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| ChannelCapacity                 | x    |       | x      |                 | x               |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| ChannelSize                     | x    |       | x      | x               | x               |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| CheckpointBackupWriteErrorCount | x    |       |        |                 |                 |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| CheckpointWriteErrorCount       | x    |       |        |                 |                 |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| EventPutAttemptCount            | x    | x     | x      | x               | x               |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| EventPutErrorCount              | x    |       |        |                 |                 |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| EventPutSuccessCount            | x    | x     | x      | x               | x               |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| EventTakeAttemptCount           | x    | x     | x      | x               | x               |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| EventTakeErrorCount             | x    |       |        |                 |                 |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| EventTakeSuccessCount           | x    | x     | x      | x               | x               |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| KafkaCommitTimer                |      | x     |        |                 |                 |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| KafkaEventGetTimer              |      | x     |        |                 |                 |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| KafkaEventSendTimer             |      | x     |        |                 |                 |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| Open                            | x    |       |        |                 |                 |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| RollbackCounter                 |      | x     |        |                 |                 |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| Unhealthy                       | x    |       |        |                 |                 |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+
 JMX Reporting
 -------------