How can I create custom metrics in Cloud Monitoring using existing Logs entries?
Question:
I have a Python based microservice where Cloud Api Python SDK is used to create and record custom metrics, code is shown below.
from google.api import label_pb2 as ga_label
from google.api import metric_pb2 as ga_metric
from google.cloud import monitoring_v3
client = monitoring_v3.MetricServiceClient()
project_name = f"projects/{project_id}"
descriptor = ga_metric.MetricDescriptor()
descriptor.type = "custom.googleapis.com/my_metric" + str(uuid.uuid4())
descriptor.metric_kind = ga_metric.MetricDescriptor.MetricKind.GAUGE
descriptor.value_type = ga_metric.MetricDescriptor.ValueType.DOUBLE
descriptor.description = "Custom Metric recording specific code level events."
Example logs are shown below. Sensitive data has been redacted.
Can I ask how to create a log-based metric to count the number of log entries that match a given filter?
Current Google cloud documentation is complex and doesn’t clearly answer my application requirements. I’ve gone through this article published by Google but not being able to get the correct metrics created.
Answers:
Unfortunately only data received after user-defined metrics have been created will be included.
The data for a user-defined log-based metric comes only from log entries received after the metric is created. A metric isn’t retroactively populated with data from log entries that are already in Logging.
https://cloud.google.com/logging/docs/logs-based-metrics
For existing log entries, they should be kept for 30 days by default so you could look to import those into BigQuery and analyse with SQL. You could also setup a log sink for future log entries.
I have a Python based microservice where Cloud Api Python SDK is used to create and record custom metrics, code is shown below.
from google.api import label_pb2 as ga_label
from google.api import metric_pb2 as ga_metric
from google.cloud import monitoring_v3
client = monitoring_v3.MetricServiceClient()
project_name = f"projects/{project_id}"
descriptor = ga_metric.MetricDescriptor()
descriptor.type = "custom.googleapis.com/my_metric" + str(uuid.uuid4())
descriptor.metric_kind = ga_metric.MetricDescriptor.MetricKind.GAUGE
descriptor.value_type = ga_metric.MetricDescriptor.ValueType.DOUBLE
descriptor.description = "Custom Metric recording specific code level events."
Example logs are shown below. Sensitive data has been redacted.
Can I ask how to create a log-based metric to count the number of log entries that match a given filter?
Current Google cloud documentation is complex and doesn’t clearly answer my application requirements. I’ve gone through this article published by Google but not being able to get the correct metrics created.
Unfortunately only data received after user-defined metrics have been created will be included.
The data for a user-defined log-based metric comes only from log entries received after the metric is created. A metric isn’t retroactively populated with data from log entries that are already in Logging.
https://cloud.google.com/logging/docs/logs-based-metrics
For existing log entries, they should be kept for 30 days by default so you could look to import those into BigQuery and analyse with SQL. You could also setup a log sink for future log entries.