python elasticsearch authenticationexception 401 error

python elasticsearch.exceptions.AuthenticationException error

 

Problem

nmap xml data not indexing(saved) as elasticsearch

nmap port scan result ->  VulntoES.py  -> elasticsearch indexing  

VulntoES.py ??

Vulnerability Data in ES

This small python script will ingest several different types of vulnerability/port scanners and import that data into Elasticsearch. Right now it supports:

• Nessus
• Nikto
• Nmap
• Openvas

The script creates a python dict from a vulnerability scanner output. It converts that to JSON and inserts that document into Elasticsearch.

python code download : https://github.com/ChrisRimondi/VulntoES

Error

Point : “AuthenticationException(401, u’security_exception’, u’missing authentication credentials for REST request”

status_code, error_message, additional_info
elasticsearch.exceptions.AuthenticationException: AuthenticationException(401, u'security_exception', u'missing authentication credentials for REST request [/ivre-vuln-2022/vuln]')
python /root/.../nmapelk/VulntoES/VulntoES.py -i /root/.../ivre/scans-2022-02/CLOUD/up/211/34/148/123.xml -e 1x.xx.1x7.x2 -p 9200 -r nmap -I ivre-vuln-2022 
Sending Nmap data to Elasticsearch
Traceback (most recent call last):
  File "/root/.../nmapelk/VulntoES/VulntoES.py", line 450, in <module>
    main()
  File "/root/.../nmapelk/VulntoES/VulntoES.py", line 441, in main
    np.toES()
  File "/root/.../nmapelk/VulntoES/VulntoES.py", line 241, in toES
    self.es.index(index=self.index_name,doc_type="vuln", body=json.dumps(dict_item))
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 370, in index
    "POST", _make_path(index, doc_type, id), params=params, body=body
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 358, in perform_request
    timeout=timeout,
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py", line 257, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py", line 182, in _raise_error
    status_code, error_message, additional_info
elasticsearch.exceptions.AuthenticationException: AuthenticationException(401, u'security_exception', u'missing authentication credentials for REST request [/ivre-vuln-2022/vuln]')

Cause

Elasticsearch connection authentication 401 error.

Solution

Elasticsearch is using authentication. You need to set the http_auth parameter set to (‘USERNAME’, ‘PASSWORD’) when configuring the connection to Elasticsearch

OS Ubuntu

python code fix (http_urllib3.py)

/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py

line 85 -> Modify Elasticsearch x-pack authentication credentials

http_auth=None, -> http_auth=('username', 'password'),

    def __init__(
        self,
        host="localhost",
        port=9200,
        http_auth=('username', 'password'),
        use_ssl=False,
        verify_certs=VERIFY_CERTS_DEFAULT,
        ssl_show_warn=True,
        ca_certs=None,
        client_cert=None,
        client_key=None,
        ssl_version=None,
        ssl_assert_hostname=None,
        ssl_assert_fingerprint=None,
        maxsize=10,
        headers=None,
        ssl_context=None,
        http_compress=False,
        cloud_id=None,
        api_key=None,
        **kwargs
    ):

Result

code fix -> nmap port scan result ->  VulntoES.py  -> elasticsearch indexing  

python /root/.../nmapelk/VulntoES/VulntoES.py -i /root/.../ivre/scans-2022-02/VIRTUAL/up/1/xx2/114/x0.xml -e x.xxx.xx.32 -p 9200 -r nmap -I ivre-vuln-2022 
Sending Nmap data to Elasticsearch                                                              
python /root/.../nmapelk/VulntoES/VulntoES.py -i /root/.../ivre/scans-2022-02/VIRTUAL/up/1/xx2/114/x2.xml -e x.xxx.xx.32 -p 9200 -r nmap -I ivre-vuln-2022 
Sending Nmap data to Elasticsearch                                                              
python /root/.../nmapelk/VulntoES/VulntoES.py -i /root/.../ivre/scans-2022-02/VIRTUAL/up/1/xx2/114/x3.xml -e x.xxx.xx.32 -p 9200 -r nmap -I ivre-vuln-2022 
Sending Nmap data to Elasticsearch                                                              
python /root/.../nmapelk/VulntoES/VulntoES.py -i /root/.../ivre/scans-2022-02/VIRTUAL/up/1/xx2/114/x4.xml -e x.xxx.xx.32 -p 9200 -r nmap -I ivre-vuln-2022 
Sending Nmap data to Elasticsearch                                                              
python /root/.../nmapelk/VulntoES/VulntoES.py -i /root/.../ivre/scans-2022-02/VIRTUAL/up/1/xx2/114/x5.xml -e x.xxx.xx.32 -p 9200 -r nmap -I ivre-vuln-2022 
Sending Nmap data to Elasticsearch                                                              
python /root/.../nmapelk/VulntoES/VulntoES.py -i /root/.../ivre/scans-2022-02/VIRTUAL/up/1/xx2/114/x7.xml -e x.xxx.xx.32 -p 9200 -r nmap -I ivre-vuln-2022 
Sending Nmap data to Elasticsearch                                                              
python /root/.../nmapelk/VulntoES/VulntoES.py -i /root/.../ivre/scans-2022-02/VIRTUAL/up/1/xx2/114/x8.xml -e x.xxx.xx.32 -p 9200 -r nmap -I ivre-vuln-2022 
Sending Nmap data to Elasticsearch                                                              
python /root/.../nmapelk/VulntoES/VulntoES.py -i /root/.../ivre/scans-2022-02/VIRTUAL/up/1/xx2/114/x9.xml -e x.xxx.xx.32 -p 9200 -r nmap -I ivre-vuln-2022 
Sending Nmap data to Elasticsearch

Elasticsearch Kibana index information after code modification 

Indexing (save) successful!!

ElasticSearch 활용 장점 및 기능 사례 정리