README.md in fluent-plugin-azure-queue-0.0.1.pre vs README.md in fluent-plugin-azure-queue-0.0.2.pre

- old
+ new

@@ -1,23 +1,26 @@ -# A [Fluentd](http://github.com/fluent/fluentd) plugin to read from azure queues +# A [Fluentd](http://github.com/fluent/fluentd) plugin to read from azure queues and event hubs +The azure queue input plugin performs at about 30 messages/second in my tests. If you need more throughput from event hubs, + I suggest using the event hub capture plugin. ## Dependencies fluentd v.12 -## Input: Configuration +## azure_queue Input Plugin +### Input: Configuration + <source> @type azure_queue tag queue_input storage_account_name my_storage_account storage_access_key my_storage_access_key queue_name my_storage_queue fetch_interval 5 - batch_size 10 - lease_time 30 + lease_duration 30 </source> **tag (required)** The tag for the input @@ -36,16 +39,80 @@ **message_key** The the record key to put the message data into. Default 'message' -**fetch_interval** +**lease_duration** -How long to wait between getting messages from the queue. Default 5 +The time to lease the messages for. Default 300 -**batch_size** +**max_fetch_threads** -The maximum number of messages to pull from the queue at once. Default 10. Max 32 +The maximum number of threads to fetch and delete queue messages with. Default 30 -**lease_time** +## Integration with Azure Event Hub -The time to lease the messages for. Default 30 +You can use an azure function to forward messages from event hubs to storage queues for easy ingestion by this gem. This is not recommended for high volumes, but should serve as a stop gap until a complete azure event hub gem is created. + +```c# +using System; +using Microsoft.WindowsAzure.Storage.Queue; + +public static void Run(string[] hubMessages, ICollector<string> outputQueue, TraceWriter log) +{ + foreach (string message in hubMessages) + { + int bytes = message.Length * sizeof(Char); + if (bytes < 64000) + { + outputQueue.Add(message); + } + else + { + log.Warning($"Message is larger than 64k with {bytes} bytes. Dropping message"); + } + } +} +``` +## azure_event_hub_capture Input Plugin +This plugin is designed to work with blobs stored to a container via [Azure Event Hubs Capture](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-capture-overview) + +### Input: Configuration + + <source> + @type azure_event_hub_capture + + tag event_hub_input + storage_account_name my_storage_account + storage_access_key my_storage_access_key + container_name my_capture_container + fetch_interval 30 + lease_duration 30 + </source> + +**tag (required)** + +The tag for the input + +**storage_account_name (required)** + +The storage account name + +**storage_access_key (required)** + +The storage account access key + +**container_name (required)** + +The capture container name + +**message_key** + +The the record key to put the message data into. Default 'message' + +**fetch_interval** + +The time in seconds to sleep between fetching the blob list. Default 30 + +**lease_duration** + +The time to lease the messages for. Default 60