Elastic Stack을 운영중 data node 용량 문제로 아래와 같은 에러 로그를 발견. 해결책에 대하여 알아보자.
org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
1. Elasticsearch Error Logs
elastic master node log files
dz-secops-2021-04-01.log:org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
dz-secops-2021-04-01.log:Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
2. Elasticsearch Error 원인 (Cause)
elastic data node 환경 설정 (elasticsearch.yml)
thread_pool.index.queue_size: 1200
thread_pool.get.queue_size: 20000
thread_pool.search.queue_size: 20000
thread_pool.bulk.queue_size: 20000
indices.memory.index_buffer_size: 30%
indices.fielddata.cache.size : 82%
indices.breaker.fielddata.limit: 84%
indices.breaker.total.limit: 85%
indices.breaker.request.limit: 70%
#http.cors.enabled: true
cluster.routing.allocation.disk.threshold_enabled : false
cluster.routing.allocation.disk.watermark.low: 94%
cluster.routing.allocation.disk.watermark.high: 95%
data node 가 디스크 용량이 cluster.routing.allocation.disk.watermark.high: 95% 이상이면 forbidden 되며,
read only만 할수 있게됩니다.
3. 해결책 (Solution)
Kibana -> Dev Tools -> Console -> Query
PUT _all/_settings
{
"index": {
"blocks": {
"read_only_allow_delete": "false"
}
}
}
Linux Shell 에서 curl 을 사용하여 Elasticsearch API 포트로 명령어를 직접 날려도 됩니다.