Data Explorer Log Writer

Overview

This log writer can be used to push log information to an Azure Data Explorer database, with log data is serialized to JSON.

Requirements

In order to use this log writer some setup steps are required to prepare the Data Explorer instance before configuring the log writer itself. Before following this process, ensure the following Azure resources have been created:

• Data Explorer cluster with a created database
• Azure AD application, in the same Azure instance as the Data Explorer database, that UNIFYConnect will use to authenticate as
• Secret key created in the Azure AD application

Table Preparation

Create a table in the Data Explorer database. The following command can be used in the Azure portals query editor, but the table can also be created using the portal's interactive wizard as well. Set tableName to a name for the table.

.create table <tableName> (TimeStamp: datetime, Source: string, Module: string, Severity: string, Message: string, Level: string, Data: dynamic)

Execute the following command in the Azure portal query editor to setup JSON mappings for the logs table. Set tableName to the name of the table, and mappingName to a unique name for the mapping.

.create-or-alter table <tableName> ingestion json mapping "<mappingName>"
'['
'   { "column": "TimeStamp", "datatype": "datetime", "Properties": { "Path": "$.timeStamp" }},'
'   { "column": "Source", "datatype": "string", "Properties": { "Path": "$.source" }},'
'   { "column": "Module", "datatype": "string", "Properties": { "Path": "$.module" }},'
'   { "column": "Severity", "datatype": "string", "Properties": { "Path": "$.severity" }},'
'   { "column": "Message", "datatype": "string", "Properties": { "Path": "$.message" }},'
'   { "column": "Level", "datatype": "string", "Properties": { "Path": "$.level" }},'
'   { "column": "Data", "datatype": "dynamic", "Properties": { "Path": "$.data" }},'
']'

If streaming ingestion mode is going to be used (see below), run this command in the Azure portal query editor to enable it for the logs table. Set tableName to the name of the table.

.alter table <tableName> policy streamingingestion enable

Run the following commands in the Azure portal query editor to give the Entra ID application the required permissions to the database and table. Set databaseName to the name of the database, tableName to the name of the table, appId to the UUID identifier of the Entra ID application, and aadDomain to the domain name associated with the Entra instance.

.add database <databaseName> viewers ('aadapp=<appId>;<aadDomain>') 'UNIFYConnect Logging Viewer Entra App'
.add table <tableName> ingestors ('aadapp=<appId>;<aadDomain>') 'UNIFYConnect Logging Ingestor Entra App'

Configuration

The Data Explorer log writer requires the following by way of configuration.

Attribute	Description
Name	The display name of the log writer which is used for identification throughout UNIFYConnect.
Ingestion Mode	Specifies the method in which data is ingested into Data Explorer. See below for the pros and cons of the available modes.
Cluster URL	The URL to the Data Explorer cluster. This will be listed in the overview of the cluster in Azure portal.
Cluster Ingestion URL	The ingestion URL to the Data Explorer cluster when Queued Ingestion mode is used. This will be listed in the overview of the cluster in Azure portal.
Application ID	The UUID identifier for the Entra application UNIFYConnect will use to authenticate as.
Application Secret Key	The secrete key for the Entra application.
Application Tennant ID	The UUID identifier for the Azure AD instance the application belongs to.
Database Name	The name of the Data Explorer database.
Table Name	The name of the tabled create for UNIFYConnect logs.
Mapping name	The name of the JSON mapping created in the logs table.

Ingestion Mode

Azure Data Explorer provides different APIs for ingesting data, with different behaviours and performance costs. The ingestion mode of the log writer should be selected based on this information. The following gives a brief overview of the modes; for more details, refer to the Microsoft documentation.

Queued Ingestion

Ingested data are added to a queue before being periodically added to the Data Explorer table. This method is more efficient in terms of cluster compute, however results in a delay between the client uploading the data, and it being available in query results.

Streaming Ingestion

In this mode, uploaded data is streamed directly into the Data Explorer table, with virtually no latency between upload and availability for the query. This comes at the cost of a higher compute load (and therefore monetary cost) placed on the cluster.