Closed
Description
Component(s)
pkg/translator/azurelogs
Is your feature request related to a problem? Please describe.
Parent issue is #39186.
Describe the solution you'd like
When the record attributes are part of the body, it is not possible to query the record by these attributes. To achieve that, we need to add explicit support for the current categories.
This issue approaches the category AzureCdnAccessLog (the goal is to handle all the other categories currently upstream as well).
This is an example of a log of this category.
{
"records": [
{
"time": "2025-04-23T09:17:17.0000000Z",
"resourceId": "/SUBSCRIPTIONS/123CA/RESOURCEGROUPS/OPENTELEMETRY/PROVIDERS/MICROSOFT.CDN/PROFILES/OPENTELEMETRY-CDN-PROFILE",
"category": "AzureCdnAccessLog",
"operationName": "Microsoft.Cdn/Profiles/AccessLog/Write",
"properties": {
"trackingReference": "20250423T091717Z-r1b8867f7999cdtwhP1CO1urpg00000000mg000000001s5s",
"httpMethod": "GET",
"httpVersion": "1.1.0.0",
"requestUri": "https://opentelemetry-cdn-endpoint.azureedge.net:443/favicon.ico",
"sni": "N/A",
"requestBytes": "1572",
"responseBytes": "2027",
"userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
"clientIp": "13.83.128.90",
"clientPort": "37908",
"socketIp": "147.243.241.243",
"timeToFirstByte": "0.154",
"timeTaken": "0.154",
"requestProtocol": "HTTPS",
"securityProtocol": "TLS 1.3",
"rulesEngineMatchNames": [],
"httpStatusCode": "404",
"httpStatusDetails": "404",
"pop": "CO",
"cacheStatus": "MISS",
"errorInfo": "NoError",
"ErrorInfo": "NoError",
"result": "N/A",
"endpoint": "opentelemetry-cdn-endpoint.azureedge.net",
"isReceivedFromClient": false,
"backendHostname": "example.com:443"
}
}
]
}The proposed log attributes are:
| Original Field (JSON) | Log Record Attribute |
|---|---|
trackingReference |
azure.ref |
httpMethod |
http.request.method |
httpVersion |
network.protocol.version |
requestUri |
url.orginalAlso parses it to get fields: 1. url.scheme2. url.fragment3. url.query4. url.path5. url.port |
sni |
tls.server.name |
requestBytes |
http.request.size |
responseBytes |
http.response.size |
userAgent |
user_agent.original |
clientIp |
client.address |
clientPort |
client.port |
socketIp |
source.address |
timeToFirstByte |
azure.time_to_first_byte |
timeTaken |
duration |
requestProtocol |
network.protocol.name |
securityProtocol |
1. tls.protocol.name2. tls.protocol.version |
rulesEngineMatchNames |
Ignore for now |
httpStatusCode |
http.response.status_code |
httpStatusDetails |
Redundant, details can be found in table |
pop |
azure.pop |
cacheStatus |
azure.cache_status |
errorInfo |
exception.type |
ErrorInfo |
Same as errorInfo |
result |
Redundant, use tls.server.name (sni) and url.original to see if there is mismatch |
endpoint |
Either: 1. destination.address if it is equal to backendHostname2. network.peer.address otherwise. |
isReceivedFromClient |
network.io.direction- If true, receive- Else, transmit |
backendHostname |
1. destination.address 2. destination.port, if any |
This means that the example of the log I put above, will end up as (assuming #39571 is merged):
resourceLogs:
- resource:
attributes:
- key: cloud.provider
value:
stringValue: azure
- key: cloud.resource_id
value:
stringValue: /SUBSCRIPTIONS/123CA/RESOURCEGROUPS/OPENTELEMETRY/PROVIDERS/MICROSOFT.CDN/PROFILES/OPENTELEMETRY-CDN-PROFILE
- key: event.name
value:
stringValue: az.resource.log
scopeLogs:
- logRecords:
- attributes:
- key: http.request.size
value:
intValue: "1572"
- key: http.response.size
value:
intValue: "1572"
- key: client.port
value:
intValue: "37908"
- key: http.response.status_code
value:
intValue: "404"
- key: azure.time_to_first_byte
value:
intValue: "154"
- key: duration
value:
intValue: "154"
- key: url.original
value:
stringValue: https://opentelemetry-cdn-endpoint.azureedge.net:443/favicon.ico
- key: url.port
value:
intValue: "443"
- key: url.scheme
value:
stringValue: https
- key: url.path
value:
stringValue: /favicon.ico
- key: tls.protocol.name
value:
stringValue: TLS
- key: tls.protocol.version
value:
stringValue: "1.3"
- key: destination.address
value:
stringValue: example.com
- key: destination.port
value:
intValue: "443"
- key: network.peer.address
value:
stringValue: opentelemetry-cdn-endpoint.azureedge.net
- key: azure.ref
value:
stringValue: 20250423T091717Z-r1b8867f7999cdtwhP1CO1urpg00000000mg000000001s5s
- key: http.request.method
value:
stringValue: GET
- key: network.protocol.version
value:
stringValue: 1.1.0.0
- key: network.protocol.name
value:
stringValue: HTTPS
- key: user_agent.original
value:
stringValue: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
- key: client.address
value:
stringValue: 13.83.128.90
- key: source.address
value:
stringValue: 147.243.241.243
- key: azure.pop
value:
stringValue: CO
- key: azure.cache_status
value:
stringValue: MISS
- key: network.io.direction
value:
stringValue: transmit
body: {}
spanId: ""
timeUnixNano: "1745399837000000000"
traceId: ""
scope:
name: otelcol/azureresourcelogs
version: 1.2.3For comparision, if needed, this is the current result for such log.
resourceLogs:
- resource:
attributes:
- key: cloud.provider
value:
stringValue: azure
- key: cloud.resource_id
value:
stringValue: /SUBSCRIPTIONS/123CA/RESOURCEGROUPS/OPENTELEMETRY/PROVIDERS/MICROSOFT.CDN/PROFILES/OPENTELEMETRY-CDN-PROFILE
- key: event.name
value:
stringValue: az.resource.log
scopeLogs:
- logRecords:
- body:
kvlistValue:
values:
- key: category
value:
stringValue: AzureCdnAccessLog
- key: operation.name
value:
stringValue: Microsoft.Cdn/Profiles/AccessLog/Write
- key: error.type
value:
stringValue: NoError
- key: properties
value:
kvlistValue:
values:
- key: requestUri
value:
stringValue: https://opentelemetry-cdn-endpoint.azureedge.net:443/favicon.ico
- key: clientIp
value:
stringValue: 13.83.128.90
- key: httpStatusCode
value:
stringValue: "404"
- key: timeToFirstByte
value:
stringValue: "0.154"
- key: pop
value:
stringValue: CO
- key: socketIp
value:
stringValue: 147.243.241.243
- key: userAgent
value:
stringValue: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
- key: securityProtocol
value:
stringValue: TLS 1.3
- key: result
value:
stringValue: N/A
- key: httpVersion
value:
stringValue: 1.1.0.0
- key: httpMethod
value:
stringValue: GET
- key: sni
value:
stringValue: N/A
- key: requestBytes
value:
stringValue: "1572"
- key: rulesEngineMatchNames
value:
arrayValue: {}
- key: cacheStatus
value:
stringValue: MISS
- key: endpoint
value:
stringValue: opentelemetry-cdn-endpoint.azureedge.net
- key: isReceivedFromClient
value:
boolValue: false
- key: requestProtocol
value:
stringValue: HTTPS
- key: responseBytes
value:
stringValue: "2027"
- key: errorInfo
value:
stringValue: NoError
- key: trackingReference
value:
stringValue: 20250423T091717Z-r1b8867f7999cdtwhP1CO1urpg00000000mg000000001s5s
- key: timeTaken
value:
stringValue: "0.154"
- key: backendHostname
value:
stringValue: example.com:443
- key: clientPort
value:
stringValue: "37908"
- key: httpStatusDetails
value:
stringValue: "404"
spanId: ""
timeUnixNano: "1745399837000000000"
traceId: ""
scope:
name: otelcol/azureresourcelogs
version: 1.2.3Describe alternatives you've considered
N/A
Additional context
Internally (at Elastic), we are using ingest pipelines to parse this body and put everything as attributes. This would not be necessary if we fix this upstream.
See, for example, the AWS logs encoding extension upstream that parses a log and puts the field as log record attributes.