继续观光一下
【Hotplug】hotplugs.hotplug.kubecube.io v1
common 与 pivot-cluster 都打开
spec:
component:
-
name: audit
status: enabled
-
env: "address: elasticsearch-master-headless.elasticsearch.svc\n"
name: logseer
namespace: logseer
pkgName: logseer-v1.0.0.tgz
status: enabled
-
env: "clustername: \"{{.cluster}}\"\n"
name: logagent
namespace: logagent
pkgName: logagent-v1.0.0.tgz
status: enabled
-
name: elasticsearch
namespace: elasticsearch
pkgName: elasticsearch-7.8.1.tgz
status: enabled
-
env: "grafana:\n enabled: false\nprometheus:\n prometheusSpec:\n externalLabels:\n cluster: \"{{.cluster}}\"\n remoteWrite:\n - url: http://172.31.0.171:31291/api/v1/receive\n"
name: kubecube-monitoring
namespace: kubecube-monitoring
pkgName: kubecube-monitoring-15.4.12.tgz
status: enabled
-
name: kubecube-thanos
namespace: kubecube-monitoring
pkgName: thanos-3.18.0.tgz
status: enabled
spec:
component:
-
env: "address: elasticsearch-master.elasticsearch.svc \n"
name: logseer
status: enabled
-
env: "grafana:\n enabled: true\nprometheus:\n prometheusSpec:\n externalLabels:\n cluster: \"{{.cluster}}\"\n remoteWrite:\n - url: http://kubecube-thanos-receive:19291/api/v1/receive\n"
name: kubecube-monitoring
-
env: "receive:\n tsdbRetention: 7d\n replicaCount: 1\n replicationFactor: 1\n"
name: kubecube-thanos
status: enabled
默认配置，配置 elasticsearch-master-headless.elasticsearch.svc和elasticsearch-master.elasticsearch.svc 都配置过，理论上不会有什么影响，还是不行,然后进行调试
问题一：查询日志报错 “request elasticsearch fail”
问题二：操作审计无数据（经过调试已解决）
过程如下：
查看logseer运行pod的容器日志发现如下
2022-09-24 20:40:47.299 [http-nio-8080-exec-10] c.n.logseer.engine.impl.ElasticSearchEngineImpl:52 INFO - [getLogs] request to es, url: /*/_search?ignore_unavailable=true, requestBody: {
"size": 50,
"from": 0,
"query": {
"bool" : {
"filter" : [
{"term": {"cluster_name" : "pivot-cluster"}},
{"term": {"namespace" : "wordpress"}}
],
"must" : [
{
"query_string" : {
"default_field" : "message",
"query" : "elasticsearch-master.elasticsearch.svc:9200"
}
},
{
"range" : {
"@timestamp" : {
"gte" : 1664019350313,
"lte" : 1664022950313,
"format": "epoch_millis"
}
}
}
]
}
},
"aggs": {
"2": {
"date_histogram": {
"field": "@timestamp",
"interval": "1m",
"time_zone": "Asia/Shanghai",
"min_doc_count": 1
}
}
},
"highlight" : {
"fields" : {
"message" : {}
},
"fragment_size": 2147483647
},
"sort" : [
{ "@timestamp" : "asc"}
],
"_source" : {
"excludes": "tags"
},
"timeout": "30000ms"
}
2022-09-24 20:40:48.302 [http-nio-8080-exec-10] c.n.logseer.engine.impl.ElasticSearchEngineImpl:65 ERROR - request elasticsearch exception: {}
java.net.ConnectException: null
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:959)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233)
at com.netease.logseer.engine.impl.ElasticSearchEngineImpl.getLogs(ElasticSearchEngineImpl.java:53)
at com.netease.logseer.service.impl.LogSearchServiceImpl.commonSearch(LogSearchServiceImpl.java:154)
at com.netease.logseer.service.impl.LogSearchServiceImpl.searchLog(LogSearchServiceImpl.java:79)
at com.netease.logseer.api.controller.LogSearchController.searchLog(LogSearchController.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.netease.logseer.api.filter.FillWebContextHolderFilter.doFilter(FillWebContextHolderFilter.java:35)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.netease.logseer.api.filter.AuthFilter.doFilter(AuthFilter.java:92)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:115)
at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:59)
at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:90)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:108)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: null
at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:168)
at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:561)
at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:822)
at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:183)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:210)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:155)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
出现一个空指针异常
- 要么取出ES的数据为空
- 要么查找拼接的地址为空毕竟有一个没看到访问HOST地址的请求 request to es, url: /*/_search?ignore_unavailable=true
进入logseer容器
直接curl http://elasticsearch-master.elasticsearch.svc:9200//_search?ignore_unavailable=true 发现返回了一大堆数据，证明ES连通性是好的，不过毕竟没添加参数，curl不是很好加参数，加了参数可能就返回空报错了。
猜测是不是环境变量没有设置起，不停的调整环境变量格式，甚至configMap,内部配置文件 期望出现日志
equest to es, url: http://elasticsearch-master.elasticsearch.svc:9200//_search?ignore_unavailable=true。
是不是没有读取到address: elasticsearch-master.elasticsearch.svc 这个变量，最终放弃也许日志本来就是这么写的。
转入logagent 的filebeat 的configMap 发现
output.elasticsearch:
hosts: [elasticsearch-master.elasticsearch.svc:30435]
这个根本访问不到修改成
output.elasticsearch:
hosts: [elasticsearch-master.elasticsearch.svc:9200]
再试试，嗯一样的不通(好在的是filebeate不爆连接错误了 )
接着看了一下文档也没发现哪里不对，再修复下审计
我本来也安装的内部ES，还是当外部配置下吧
kubectl edit deploy audit -n kubecube-system
env:
- name: AUDIT_WEBHOOK_HOST
value: http://elasticsearch-master.elasticsearch:9200
- name: AUDIT_WEBHOOK_INDEX
value: audit
- name: AUDIT_WEBHOOK_TYPE
value: logs
审计可以了，
但是日志依然不通，看来只有想办法开放ES 9200端口出来用工具连连是没上传还是没查询到，
不过大体定位到如下可能的几个问题
- 可能是logseer没有读取到环境变量导致出错
- filebeate配置错误没有上传成功，导致查询空数据（这个数据内容空就报错应该不至于），但是确实直接安装的filebeat有一个connect 错误需要修复
费解
目前发现的问题猜测ripple和filebeat的配置感觉这里嫌疑最大，创建了新的日志抓取任务，也没看到/etc/filebeat/inputs.d 有什么文件改动
不过也建议修复下空报错的问题，让指示得更明显，只能去看看哪里有源码了
![[Bug]新版的ingress不能直接使用](https://avatars.githubusercontent.com/u/15830225?v=4)
![[Bug][Help]已有k8s 1.22.1集群中部署KubeCube不成功?](https://avatars.githubusercontent.com/u/1655233?v=4)
![[Feature]KubeCube适配接入用户自有监控后端](https://avatars.githubusercontent.com/u/41603640?v=4)
![[Bug] When add cluster, If my kubeconfig is wrong. kubecube container will panic.](https://avatars.githubusercontent.com/u/5100555?v=4)
![[Bug]节点添加脚本错误相关问题](https://avatars.githubusercontent.com/u/22486890?v=4)
![[Feature] 支持网络策略的配置或者支持不同租户下的网络隔离](https://avatars.githubusercontent.com/u/24875266?v=4)
