Cant get data from Kubernetes Cluster

PBR

Recently added Kubernetes Cluster and can't get any data. Connection test works (API Token is OK) - however load makes a 404:

Fri Jan 6 15:25:50 2023: 404 page not found :

Fri Jan 6 15:25:50 2023: JSON decode response from API Error: garbage after JSON object, at character offset 4 (before "page not found\n") at /home/lpar2rrd/lpar2rrd/bin/Kubernetes.pm line 828.

 :

malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "(end of string)") at /home/lpar2rrd/lpar2rrd/bin/kubernetes-background.pl line 105.

[Fri Jan 6 15:25:51 2023] kubernetes-cleanup.pl: Use of uninitialized value $updated in subtraction (-) at /home/lpar2rrd/lpar2rrd/bin/kubernetes-cleanup.pl line 49.

Matej

Hi,

please send logs:

cd /home/lpar2rrd/lpar2rrd # or your LPAR2RRD working dir
tar cvhf logs.tar logs tmp/*txt tmp/*json data/Kubernetes/*json
gzip -9 logs.tar

 Send us logs.tar.gz via https://upload.lpar2rrd.com

Ricardo.Guidos

I have similar event:

Mon May 22 18:19:53 2023: 404 page not found :

Mon May 22 18:19:53 2023: JSON decode response from API Error [198.20.2.9:6443/apis/metrics.k8s.io/v1beta1/pods]: garbage after JSON object, at character offset 5 (before "age not found\n") at /home/lpar2rrd/lpar2rrd/bin/Kubernetes.pm line 844.

 :

Mon May 22 18:19:53 2023: 404 page not found :

Mon May 22 18:19:53 2023: JSON decode response from API Error [198.20.3.3:6443/apis/metrics.k8s.io/v1beta1/pods]: garbage after JSON object, at character offset 5 (before "age not found\n") at /home/lpar2rrd/lpar2rrd/bin/Kubernetes.pm line 844.

 :

malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "(end of string)") at /home/lpar2rrd/lpar2rrd/bin/kubernetes-background.pl line 105.

malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "(end of string)") at /home/lpar2rrd/lpar2rrd/bin/kubernetes-background.pl line 105.

[Mon May 22 18:19:54 2023] kubernetes-cleanup.pl: Use of uninitialized value $updated in subtraction (-) at /home/lpar2rrd/lpar2rrd/bin/kubernetes-cleanup.pl line 49.

date end all  : Mon May 22 18:19:54 CST 2023

Matej

Hi Ricardo,

please send the output:

export APISERVER=<k8s_cluster_adress>

export TOKEN=<token>

curl -X GET $APISERVER/apis/metrics.k8s.io/v1beta1/pods --header "Authorization: Bearer $TOKEN" --insecure

and check if metrics-server is running:

kubectl get pods -n kube-system

Ricardo.Guidos

is not running , how is my error ?

curl -X GET $APISERVER/apis/metrics.k8s.io/v1beta1/pods --header "Authorization: Bearer $TOKEN" --insecure

404 page not found

Matej

Hi Ricardo,

you need to have Kubernetes metrics-server installed - https://github.com/kubernetes-sigs/metrics-server

Look in the "Installation" section - https://github.com/kubernetes-sigs/metrics-server#installation

Ricardo.Guidos

I see my error, I am new on kubernets , after install the metrics, is begining to take the data, thanks

lmelcher

Im running into a similar error, where I can API auth is OK but won't load the namespaces. I followed this https://lpar2rrd.com/Kubernetes-performance-monitoring-install.php#Kubernetes and my metrics-server is running and reports back top pod and top node information. I ran curl -X GET $APISERVER/apis/metrics.k8s.io/v1beta1/pods --header "Authorization: Bearer $TOKEN" --insecure with my $APISERVER and $TOKEN info and this is the error that comes back.

# curl -X GET $APISERVER/apis/metrics.k8s.io/v1beta1/pods --header "Authorization: Bearer $TOKEN" --insecure

{

 "kind": "Status",

 "apiVersion": "v1",

 "metadata": {},

 "status": "Failure",

 "message": "pods.metrics.k8s.io is forbidden: User \"system:serviceaccount:kube-system:lpar2rrd\" cannot list resource \"pods\" in API group \"metrics.k8s.io\" at the cluster scope",

 "reason": "Forbidden",

 "details": {

  "group": "metrics.k8s.io",

  "kind": "pods"

 },

 "code": 403

clusterrole looks good.

# kubectl describe clusterrole lpar2rrd         Name:     lpar2rrd

Labels:    <none>

Annotations: <none>

PolicyRule:

 Resources          Non-Resource URLs Resource Names Verbs

 ---------          ----------------- -------------- -----

 endpoints          []         []       [get watch list]

 namespaces         []         []       [get watch list]

 nodes/proxy         []         []       [get watch list]

 nodes            []         []       [get watch list]

 pods            []         []       [get watch list]

 services          []         []       [get watch list]

 endpoints.metrics.k8s.io  []         []       [get watch list]

 namespaces.metrics.k8s.io  []         []       [get watch list]

 nodes.metrics.k8s.io/proxy []         []       [get watch list]

 nodes.metrics.k8s.io    []         []       [get watch list]

 pods.metrics.k8s.io     []         []       [get watch list]

 services.metrics.k8s.io   []         []       [get watch list]

               [/metrics]     []       [get]

lmelcher

I figured out my issue..... you may want to update your procedure to make sure when the serviceaccount is created, your current context is the default namespace. Mine was set to kube-system, so the service account was not working properly.