Skip to content

Observability

Grafana Alloy: Helpful Things

Debugging Configs

WebUI

visit the Alloy agent WebUI for me that means visiting <alloy-agent-ip>:12345/graph this will show you what configurations are having problems and sometimes even give you some helpful errors.

Oneliner to run local Alloy to test config

Run alloy as a container

Why aren't my relabels showing up?

Make sure you are taking your discovery.relabel.name.output is being used for your loki.source...targets

For Example:

discovery.relabel "kubernetes_pods" {
  targets = discovery.kubernetes.pods.targets

  rule {
      ...
  }

}

loki.source.kubernetes "pods" {
  targets    = discovery.relabel.kubernetes_pods.output
  forward_to = [loki.write.endpoint.receiver]
}

loki.write "endpoint" {
  endpoint {
      url = "http://<your-loki-hostname>:80/loki/api/v1/push"
      tenant_id = "local"
  }
}

Why am I seeing failed to create fsnotify watcher: too many open files

I can't speak to the exact issue you were facing but this helped me out: https://github.com/grafana/alloy/issues/1217#issuecomment-2236272320

My alloy config to get kubernetes pods looked like this:

discovery.kubernetes "pods" {
  role = "pod"
}

I needed to change it to:

discovery.kubernetes "pods" {
  role = "pod"

  selectors {
    role  = "pod"
    field = "spec.nodeName=" + coalesce(env("HOSTNAME"), constants.hostname)
  }
}

This fixed my problem, hope it help future me or you out.