Skip to content

Blog

directions to follow

https://apavankumar.medium.com/hashicvault-secrets-in-kubernetes-with-csi-driver-ec917d4a2672

  1. Deploy vault
$ kubectl exec -it vault-0 -- /bin/sh

$ vault login root 

$ vault secrets enable -version=1 kv

$ vault auth enable kubernetes

$ vault write auth/kubernetes/config token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

$ vault policy write kv_policy - <<EOF
path "kv/*" {
  capabilities = ["read"]
}
EOF

$ vault write auth/kubernetes/role/csi-kv \
bound_service_account_names=csi-sa \
bound_service_account_namespaces=default \
policies=kv_policy \
ttl=20m

## Put some Sample data 
vault kv put kv/db password=password
vault kv put kv/app user=admin

create a SecretProviderClass for our kv put values

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: vault-user-creds
spec:
  provider: vault
  parameters:
    roleName: 'csi-kv'
    vaultAddress: 'http://vault:8200' # use the service that is made by vault
    objects: |
      - objectName: "user"
        secretPath: "kv/app"
        secretKey: "user"
      - objectName: "password"
        secretPath: "kv/db"
        secretKey: "password"

https://tailscale.com/kb/1236/kubernetes-operator 2. Deploy csi storage secret 3. Deploy tailscale - this needs a specific secret client_id and client_secret, if you want it secure put it in vault and expose using a SecretProviderclass

  • in KeyCloak
  • 1 min read

KeyCloak

How to get user token

  • NOTE: do not use method for real environment, can expose password in cmd history
export CLIENT_ID=client_id
export REALM_NAME=realm_name
export REALM_USERNAME=realm_user
export REALM_PASSWORD=realm_password
export KEYCLOAK_HOST=http://keycloak_host
url -X POST -d "client_id=$CLIENT_ID" -d "username=$REALM_USERNAME" -d "password=$REALM_PASSWORD" -d "grant_type=password" "$KEYCLOAK_HOST/realms/$REALM_NAME/protocol/openid-connect/token"

Tempo

When using the tempo grafana helm chart you will want to add it as a data source to your grafana dashboard. The part I was getting confused by was what port to use. In my case you will want to use port 3100.

  • in Golang
  • 1 min read

Simple Cassandra Tips

Run cql on a cassandra pod

. Attach to a running pod using bash . cqlsh -u -p localhost 9042

What are the names of all my datacenters?

use system;
select data_center from local;

How do you use ORDER BY in Cassandra?

You will need to add the ORDER BY to your query.

Example Query

SELECT *
FROM chats
ORDER BY message_timestamp

If you use this query and get this error message: Error from server: code=2200 [Invalid query] message="Order by is currently only supported on the clustered columns of the PRIMARY KEY, got message_timestamp"

This means you need to make sure message_timestamp is part of your primary key or a clustered column.

if it is in your primary key and you are still getting this message, you probably have a primary key with multiple keys.

Which means you will need to add a WHERE clause.

SELECT *
FROM chats
WHERE CHAT_ID=123
ORDER BY message_timestamp

Grafana Alloy: Helpful Things

Debugging Configs

WebUI

visit the Alloy agent WebUI for me that means visiting <alloy-agent-ip>:12345/graph this will show you what configurations are having problems and sometimes even give you some helpful errors.

Oneliner to run local Alloy to test config

Run alloy as a container

Why aren't my relabels showing up?

Make sure you are taking your discovery.relabel.name.output is being used for your loki.source...targets

For Example:

discovery.relabel "kubernetes_pods" {
  targets = discovery.kubernetes.pods.targets

  rule {
      ...
  }

}

loki.source.kubernetes "pods" {
  targets    = discovery.relabel.kubernetes_pods.output
  forward_to = [loki.write.endpoint.receiver]
}

loki.write "endpoint" {
  endpoint {
      url = "http://<your-loki-hostname>:80/loki/api/v1/push"
      tenant_id = "local"
  }
}

Why am I seeing failed to create fsnotify watcher: too many open files

I can't speak to the exact issue you were facing but this helped me out: https://github.com/grafana/alloy/issues/1217#issuecomment-2236272320

My alloy config to get kubernetes pods looked like this:

discovery.kubernetes "pods" {
  role = "pod"
}

I needed to change it to:

discovery.kubernetes "pods" {
  role = "pod"

  selectors {
    role  = "pod"
    field = "spec.nodeName=" + coalesce(env("HOSTNAME"), constants.hostname)
  }
}

This fixed my problem, hope it help future me or you out.

Useful K8s Gateway (istio)

How do I pick an IP for my Gateway to use?

Add This to your kind: Gateway spec

  addresses:
  - value: <ip-i-want-to-use>
    type: IPAddress

Full example:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: mygatedway
spec:
  gatewayClassName: istio
  listeners:
  - hostname: "*.mydomain.com"
    name: http
    port: 80
    protocol: HTTP
  addresses:
  - value: 192.168.0.100
    type: IPAddress

Where did I get this info from: GH Discussion