Trying to enable hubble ui in a cluster where cilium was installed with helm:

cilium hubble enable --ui --create-ca --relay-version v1.10.3

(The --relay-version is a workaround for https://github.com/cilium/cilium-cli/issues/456)

After port-forward, hubble status reports Max Flows 0/0 and all Nodes Unavailable even though running cilium status in each cilium pod shows Max Flows 4095/4095.

No known workaround.

Is this another case of cilium-cli being incompatible with a helm-installed Cilium? We wouldn't have to blaze that trail if cilium-cli were able to install Cilium chained to eks-vpc-cni.

When I port-forward the hubble-ui service and try to load the UI in a browser, the following happens:

• the web page remains stuck on the "The application is loading, please wait..." page.
• the logs of the hubble-ui pod show the following message:

{
  "name": "frontend",
  "hostname": "hubble-ui-79b6c7c67-z4bs5",
  "pid": 19,
  "req_id": "101ee530-14a9-4580-868a-66fed7c6fd49",
  "user": "admin@localhost",
  "level": 50,
  "err": {
    "message": "Can't fetch namespaces via k8s api: Error: getaddrinfo EAI_AGAIN $ENTER_AKS_CLUSTER_DOMAIN_NAME",
    "locations": [
      {
        "line": 4,
        "column": 7
      }
    ],
    "path": [
      "viewer",
      "clusters"
    ],
    "extensions": {
      "code": "INTERNAL_SERVER_ERROR"
    }
  },
  "msg": "",
  "time": "2020-03-08T18:09:56.167Z",
  "v": 0
}

Hi, when trying to follow the instructions that appears in this site: https://github.com/cilium/hubble/blob/master/Documentation/installation.md once you reach to hubble and try to run this cmd:

helm template hubble \
    --namespace kube-system \
    --set metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,http}" \
    > hubble.yaml

you will fail on :

Error: rendering template failed: runtime error: invalid memory address or nil pointer dereference

tried to install also without any metrics and also not working , it looks like the template that exist here not working . can you please update the guidelines if any thing is expected?

This adds support to Hubble CLI for filtering against endpoints workloads The server side of this was implemented in https://github.com/cilium/cilium/pull/21296

Flows and arrows are not visible in Hubble UI. Yet flows for "hubble" namespace are visible. Running in GKE.

Running procedure:

helm template cilium \
  --namespace cilium \
  --set global.nodeinit.enabled=true \
  --set nodeinit.reconfigureKubelet=true \
  --set nodeinit.removeCbrBridge=true \
  --set global.cni.binPath=/home/kubernetes/bin \
  --set global.tag=v1.7.0-rc1 \
  > cilium.yaml

helm template hubble \
    --namespace hubble \
    --set metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,http}" \
    --set ui.enabled=true \
    > hubble.yaml

I can confirm that flows are visible in "cilium monitor", "hubble observe", and "kubectl get cep".

Dear Hubble Community,

We are currently migrating to Cilium as our networking solution and are very excited to use Hubble for observability.

However, we miss one thing to be happy – OpenTelemetry (OpenTracing) support. I can see it was mentioned in the roadmap around Cilium 1.0 release:

h3. The Roadmap Ahead Integration with OpenTracing, Jaeger, and Zipkin: The minimal overhead of BPF makes it the ideal technology to provide tracing and telemetry functionality without imposing additional system load.

However, I haven't found any code/issues connected to it. I thought that might be Cilium Go Extensions is the right place to implement it. Then I checked Hubble, and it looks like all the data required is in place. I can potentially contribute to it if you give some guidance if Hubble Relay the right place for it.

I would like to ask how to clean up the cilium environment

I follow the official documentation

# install
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/v1.7.0/install/kubernetes/quick-install.yaml

# delete
kubectl delete -f https://raw.githubusercontent.com/cilium/cilium/v1.7.0/install/kubernetes/quick-install.yaml

After that, I found that all my pods cannot be created properly. about cilium crd,I have deleted. Do i need to delete anything？

Error message

# kubectl get pod  | grep httpd
httpd-596db6fdc4-4r22k                                 0/1     ContainerCreating   0          15m
httpd-596db6fdc4-5xldk                                 0/1     ContainerCreating   0          15m

# kubectl describe pod
Events:
  Type     Reason                  Age    From                             Message
  ----     ------                  ----   ----                             -------
  Normal   Scheduled               10m    default-scheduler                Successfully assigned default/httpd-596db6fdc4-5xldk to node001
  Warning  FailedCreatePodSandBox  9m17s  kubelet, node001  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "ffcd455f1ab5483a17f87cdad35beaea980e61317dbe35b788cac7953e72c95f" network for pod "httpd-596db6fdc4-5xldk": NetworkPlugin cni failed to set up pod "httpd-596db6fdc4-5xldk_default" network: unable to connect to Cilium daemon: failed to create cilium agent client after 30.000000 seconds timeout: Get http:///var/run/cilium/cilium.sock/v1/config: dial unix /var/run/cilium/cilium.sock: connect: no such file or directory
Is the agent running?

Dear Hubble community,

While logging traffic with Hubble:

hubble observe -f --server hubble-relay:80 -o json --tcp-flags ACK --not --tcp-flags SYN

Getting most events doubled in output: They have only difference in logging timestamp, ex:

{"time":"2021-10-18T11:30:20.830417817Z","verdict":"FORWARDED","ethernet":{"source":"66:54:11:3e:bd:de","destination":"12:7a:c7:e0:b1:28"},"IP":{"source":"10.0.2.75","destination":"10.45.80.193","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":49488,"destination_port":6443,"flags":{"ACK":true}}},"source":{"ID":140,"identity":6013,"namespace":"ingress-nginx","labels":["k8s:app.kubernetes.io/component=controller","k8s:app.kubernetes.io/instance=ingress-nginx","k8s:app.kubernetes.io/name=ingress-nginx","k8s:io.cilium.k8s.namespace.labels.field.cattle.io/projectId=p-8xjww","k8s:io.cilium.k8s.namespace.labels.name=ingress-nginx","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=ingress-nginx","k8s:io.kubernetes.pod.namespace=ingress-nginx"],"pod_name":"ingress-nginx-controller-db9d9c7f4-gjllb"},"destination":{"identity":6,"labels":["reserved:remote-node"]},"Type":"L3_L4","node_name":"dev-wg-app1","event_type":{"type":4,"sub_type":3},"traffic_direction":"EGRESS","trace_observation_point":"TO_STACK","is_reply":false,"Summary":"TCP Flags: ACK"}
{"time":"2021-10-18T11:30:26.853421611Z","verdict":"FORWARDED","ethernet":{"source":"66:54:11:3e:bd:de","destination":"12:7a:c7:e0:b1:28"},"IP":{"source":"10.0.2.75","destination":"10.45.80.193","ipVersion":"IPv4"},"l4":{"TCP":{"source_port":49488,"destination_port":6443,"flags":{"ACK":true}}},"source":{"ID":140,"identity":6013,"namespace":"ingress-nginx","labels":["k8s:app.kubernetes.io/component=controller","k8s:app.kubernetes.io/instance=ingress-nginx","k8s:app.kubernetes.io/name=ingress-nginx","k8s:io.cilium.k8s.namespace.labels.field.cattle.io/projectId=p-8xjww","k8s:io.cilium.k8s.namespace.labels.name=ingress-nginx","k8s:io.cilium.k8s.policy.cluster=default","k8s:io.cilium.k8s.policy.serviceaccount=ingress-nginx","k8s:io.kubernetes.pod.namespace=ingress-nginx"],"pod_name":"ingress-nginx-controller-db9d9c7f4-gjllb"},"destination":{"identity":6,"labels":["reserved:remote-node"]},"Type":"L3_L4","node_name":"dev-wg-app1","event_type":{"type":4,"sub_type":3},"traffic_direction":"EGRESS","trace_observation_point":"TO_STACK","is_reply":false,"Summary":"TCP Flags: ACK"}

How this could be explained and avoided? Thanks!

This PR aims to achieve the following:

• [x] Refactor, where applicable, to test output functions.
• [x] Add table driven inputs for invoking certain output functionality.

Signed-off-by: Simarpreet Singh [email protected]

Rename the current release make target to local-release, and update the release target to generate release artifacts from inside Docker.

Signed-off-by: Michi Mutsuzaki [email protected]

We cannot render the hubble-ui due to this below error message:

"message":"Can't fetch namespaces via k8s api: Error: unable to get issuer certificate","locations":[{"line":4,"column":7}],"path":["viewer","clusters"],"extensions":{"code":"INTERNAL_SERVER_ERROR"}}

{ name: 'inCluster', caFile: '/var/run/secrets/kubernetes.io/serviceaccount/ca.crt', server: 'https://10.110.121.43:443', skipTLSVerify: false }

Bumps actions/upload-artifact from 3.1.0 to 3.1.2.

You can trigger a rebase of this PR by commenting @dependabot rebase.

I am a green hand on Hubble. When I installed the Cilium and Hubble by helm, I found that the hubble-ui seems not secured by TLS, and have no more user identification mechanism. Thus any pods/worker nodes can access the hubble-ui-service-ip:80, and access all service maps in the whole cluster. Have I missed something? Looking forward to your reply!:)

Current Hubble 0.10.0 contains 16 GO related CVEs. Updating Hubble to use 1.18.9 will address these CVEs that have occurred since the June 2022 release of 0.10.0. I am requesting an incremental release of 0.10.x with this issue submission. Has there been any thought to aligning Hubble incremental release cadence with that of Cilium cadence (1.12.5 came out last week and updated to 1.18.9 GO) ?

Hi ,

I'm having some issue trying to display external services.

Here are the details :

I have an external service defined as follow :

kind: Service
apiVersion: v1
metadata:
  name: "searchmaster"
  labels:
    ressourcetype: service-solr-cd
    env: cd
spec:
  type: ExternalName
  externalName: searchmaster.mydomain.local 

Now , I have a pod that call this service , and also call a MysqL url ( which is not defined as a kubernetes service ). So basically configuration is :

searchMasterUrl: http://searchmaster:8080 mySqlUrl: np-mysql01.mydomain.local

Here is what I see in hubble :

I can see dns resolution works because i can see the the ip in the hubble log. But this is flag as "World"

Is there any way I can display my externalService name to identify theses flows ?

I might missunderstanding something maybe because this should work out of the box since I use a dns name and it should get caught by dns rule.

Cilium Version

v1.12.4

Kernel Version

5.10.0-14-amd64

Kubernetes Version

v1.25.4

Hubble version

v1.12.4

Thanks for your help !

Regards

HTTP flows contain headers but Hubble doesn't support filtering flows based on HTTP headers. Using the CLI, we can already filter based on HTTP status code, methods and paths but filtering on headers is still missing.