Elasticsearch 403 FORBIDDEN Index read-only Error 처리

Elastic Stack을 운영중 data node 용량 문제로 아래와 같은 에러 로그를 발견. 해결책에 대하여 알아보자.

org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]

Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

1. Elasticsearch Error Logs

---

elastic master node log files

dz-secops-2021-04-01.log:org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
dz-secops-2021-04-01.log:Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

2. Elasticsearch Error 원인 (Cause)

---

elastic data node 환경 설정 (elasticsearch.yml)

thread_pool.index.queue_size: 1200
thread_pool.get.queue_size: 20000
thread_pool.search.queue_size: 20000
thread_pool.bulk.queue_size: 20000
indices.memory.index_buffer_size: 30%

indices.fielddata.cache.size : 82%
indices.breaker.fielddata.limit: 84%
indices.breaker.total.limit: 85%
indices.breaker.request.limit: 70%
#http.cors.enabled: true
cluster.routing.allocation.disk.threshold_enabled : false
cluster.routing.allocation.disk.watermark.low: 94%
cluster.routing.allocation.disk.watermark.high: 95%  

data node 가 디스크 용량이 cluster.routing.allocation.disk.watermark.high: 95%  이상이면 forbidden 되며,

read only만 할수 있게됩니다.

 

3. 해결책 (Solution)

---

Kibana -> Dev Tools -> Console -> Query 

PUT _all/_settings
{
    "index": {
        "blocks": {
        "read_only_allow_delete": "false"
        }
    }
}

Linux Shell 에서 curl 을 사용하여 Elasticsearch API 포트로 명령어를 직접 날려도 됩니다.

ElasticSearch 활용 장점 및 기능 사례 정리