Our team has concluded our sanity check and all Log Collected data is now displaying correctly and is valid.
This incident is considered resolved.
Posted May 18, 2022 - 20:05 UTC
Update
April data is now fully available and valid for all our Log Collection clients. The team is doing a sanity check on the data collected for May to make sure the data is valid. Depending on the result fo this verification, certain dates might need to be reprocessed. More information to follow.
Posted May 17, 2022 - 14:47 UTC
Update
Processing has progressed and almost all of the data is now accessible and correct in WCM for Log Collection clients. The only exception is April 14th, which we are currently reprocessing. More information to follow.
Posted May 09, 2022 - 20:20 UTC
Identified
We would like to update our clients on this issue and clarify its impacts. On April 19th one of the servers responsible for routing collected log files to our various data processing mechanisms became unresponsive. The mechanisms responsible for fetching the logs were still active but the data was not able to be fully routed down the chain. This has caused incomplete data to show on our various UI (for example WCM and Streaming Explore).
The affected server has been rebuilt and data has been flowing normally to the various data processing mechanisms. However, large amounts of listener data still need to be processed for the various UI to show correct numbers.
Because of various factors (such as time zone normalization, normal delays in final logs being made available to Triton, etc), we are reprocessing data from April 14th onwards.
Our current ETA for processing of all logs is by end of day Friday May 6th.
More information to follow.
Posted Apr 26, 2022 - 19:59 UTC
Investigating
We are investigating an issue which is affecting a portion of our clients using Log Collection as a mechanism to populate Webcast Metrics data. The issue might be causing certain data not to be collected and therefore will cause WCM data to be inaccurate for these clients. More information to follow.