Overview
Stream Management is management of a specific collection of data that is used as a log. The data is written to and read from the log stream in a specific format so that all the instances of a particular stream are diverted into a single segment.
Data in a log stream can be compressed, replicated, and retained for a specific time period. DNIF allows you to configure customized settings for each stream in the organizational network. It will help you to compress the data without affecting the query speed, and also store data for compliance purposes without occupying much disk space.
Retention
Retention settings can be used to specify how long log events are stored in DNIF. By default, the log data retention period is for 90 days. The log data retention is configurable as per organizational requirements and all the data older than the current retention configurations will be deleted automatically.
Data Compression
The growth of big data has created a demand for ever-increasing processing power and efficient storage. Compressing big data can help address these demands by reducing the amount of storage and bandwidth required for data sets.
DNIF uses compression algorithms to manage large amounts of data. Compression can also remove irrelevant or redundant data, making analysis and processing easier and faster. It typically reduces the size of a file, which reduces both cost and transfer time.
Replication
DNIF Data Replication is the asynchronous process of storing the same data in multiple locations to improve data availability and accessibility, and to improve system resilience and reliability. One common use of data replication is for disaster recovery, to ensure that an accurate backup exists at all times in case of a catastrophe, hardware failure, or a system breach where data is compromised. By default, the data will be replicated once.
How to configure Streams?
- Click Streams icon on the left navigation bar of the Home Screen, to view list of all the streams configured
The Stream Management screen displays the following details
- The top section of the screen displays the total consumption of the tenant (previously known as cluster ) such as total volume of raw logs, enriched logs, disk space, percentage of compression.
- The total consumption by an individual stream is displayed in the grid format as follows:
| Field | Description |
| Stream name | Displays the name of the stream |
| Raw Vol. | Displays the total volume of raw logs for that particular stream |
| Enriched Vol. | Displays the total volume of enriched logs for that particular stream. |
| Size on Disk | Displays the disk space occupied by each stream |
| Compression | Displays the rate at which the logs are compressed for that particular stream in percentage. |
You can also apply the following filters to the above grid view
| Filters | Description |
| Online | Use this to view all the streams that are currently online |
| Historic | Use this to view all the streams ever since DNIF deployment |
| Current Month | Use this to view all the streams for the current month. By default, the details are displayed for the current month. |
- Click Edit icon displayed against each stream to configure details.
- The Edit screen displays the following configurations:
| Field Name | Description |
| Compression | Enter the compression type. By default, the compression type used is Normal. You can also change the compression type to High as per your requirement. |
| Retention | Enter the retention period. By default, the retention period is 90 days. |
| Replication | Enter the replication factor. By default, the replication factor is 1. Note:
|
- Modify the required configuration details and Click Save, to save the configuration.
- You can click Collection Status to view more details about the source streams from where logs are collected. Refer Device Collection Status.