Elasticsearch Logstash Kibana (ELK) Authentication using Active Directory

By January 10, 2020 January 13th, 2020 Blogs

Written by Priyanka Sharma, DevOps Architect, Powerupcloud Technologies

This article covers how you can enable security features on ELK to communicate with AD to authenticate Users. The security features provide two realms to achieve the same: One is LDAP realm and the other one is the ActiveDirectory realm.

We have used the Active_directory realm in our configurations. active_directory realm uses an LDAP bind request so it is similar to the LDAP realm. The Active Directory realm authenticates users using an LDAP bind request. After authenticating the user, the realm then searches to find the user’s entry in Active Directory.

Setup:

  • Elasticsearch version: 7.2
  • Three Master Nodes in private subnets
  • Kibana EC2 standalone server in private subnet
  • Logstash running on the standalone application server in private subnet
  • One Internal ALB with host-based routing for Kibana and Elasticsearch Endpoints.
    • kibana.powerupcloud.com → Pointing to Kibana Server
    • elasticsearch.powerupcloud.com → Pointing to ES Masters
  • Active Directory with ESAdmins AD group and a few users added to it which requires Elasticsearch access. Ensure port TCP 389 and UDP 389 are allowed in the AD.

Elasticsearch Nodes Configuration

Ensure to activate X-Pack on the Elastic Stack. It is an Elastic Stack extension that provides security, alerting, monitoring, reporting, machine learning, and many other capabilities. By default, X-Pack is installed when you install the Elasticsearch.

Create the Certificate to be used by Transport Layer Security (TLS) in the Elastic Stack.

/usr/share/elasticsearch/bin/elasticsearch-certutil ca
/usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
mkdir -p  /etc/elasticsearch/certs/
cp -r elastic-certificates.p12 /etc/elasticsearch/certs/

Update the Certificate Path in the /etc/elasticsearch/elasticsearch.yml. Add the realm configuration in the same file. The final elasticsearch.yml looks like as shown below:

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

network.host: 0.0.0.0
cluster.name: puc_elasticsearch
node.name: master-1
http.port: 9200

node.master: true
node.data: true
plugin.mandatory: "discovery-ec2"
discovery.zen.hosts_provider: ec2
discovery.ec2.availability_zones: "us-east-1a, us-east-1b, us-east-1c, us-east-1d"
discovery.zen.minimum_master_nodes: 2
discovery.seed_hosts: ["172.31.xx.xx", "172.31.xx.xx", "172.31.xx.xx"]
cluster.initial_master_nodes: ["172.31.xx.xx", "172.31.xx.xx", "172.31.xx.xx"]

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.license.self_generated.type: trial
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/elastic-certificates.p12

xpack:
  security:
    authc:
      realms:
        active_directory:
          my_ad:
            order: 1
            domain_name: AD_DNS
            url: ldap://AD_DNS:389
            user_search:
              base_dn: "cn=users,dc=puc,dc=com"
            group_search:
              base_dn: "cn=users,dc=puc,dc=com"
            files:
              role_mapping: "/etc/elasticsearch/role_mapping.yml"

Replace AD_DNS with the domain name associated with the Active Directory.

The trial license of xpack is available only for 30 days. After that, it is mandatory to purchase a license and update the type of license in the configuration.

Update the role mapping file to map the AD group with an existing ES role.

vim /etc/elasticsearch/role_mapping.yml

superuser:
   - "CN=ESAdmins,CN=Users,DC=puc,DC=com"

ESAdmins is the AD Group Name. Replace it as required.

Superuser is an inbuilt available role in Elasticsearch. By this role_mapping, we are mapping the AD Group to the superuser ES role.

Upload the same certificate “ /etc/elasticsearch/certs/elastic-certificates.p12” in the other two nodes as well. You can use scp commands to achieve it.

Add the same xpack configurations in the other two nodes too.

Validate the ES authentication by executing the curl commands as shown in the screenshot below:

Logstash Configuration

Now that we have the AD authentication enabled on the ES nodes in the above section. Update the logstash configuration to authenticate ES with an AD user. Update the logstash.conf file to add the AD user credentials as highlighted in the below config:

vim /etc/logstash/conf.d/logstash.conf

input {
 file {
   path => ["/var/log/nginx/access.log", "/var/log/nginx/error.log"]
   type => "nginx"
 }
  beats {
    port => 5044
  }
}
filter {
 grok {
   match => [ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}"]
   overwrite => [ "message" ]
 }
 mutate {
   convert => ["response", "integer"]
   convert => ["bytes", "integer"]
   convert => ["responsetime", "float"]
 }
 geoip {
   source => "clientip"
   target => "geoip"
   add_tag => [ "nginx-geoip" ]
 }
 date {
   match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
   remove_field => [ "timestamp" ]
 }
 useragent {
   source => "agent"
 }
}
output {
 elasticsearch {
   hosts => ["http://elasticsearch.powerupcloud.com:80"]
   user => "esadmin@puc.com"
   password => "PASSWORD"
   index => "nginx-%{+YYYY.MM.dd}"
 }
 stdout { codec => rubydebug }
}

In the above configuration, Replace the ES Endpoint (elasticsearch.powerupcloud.com), AD user and password. The AD user must exist in the same AD group as specified in the role_mapping.yml.

Restart logstash: “service logstash restart”. Ensure to look at the logs after restart.

Kibana Configuration

Similar to Logstash, update the Kibana configuration to add the AD User Credentials for Elasticsearch endpoint.

vim /etc/kibana/kibana.yml

server.host: "0.0.0.0"
elasticsearch.hosts: ["http://elasticsearch.powerupcloud.com"]
elasticsearch.username: "esadmin@puc.com"
elasticsearch.password: "PASSWORD"

In the above configuration, Replace the ES Endpoint, elastic search.username and elasticsearch.password. The AD user must exist in the same AD group as specified in the role_mapping.yml.

Restart Kibana: service kibana restart

Hit Kibana Endpoint.

Enter the user credentials that exist in the AD group.

Appendix:

If the Xpack license is not activated earlier before enabling AD authentication, you can execute the below commands to start the trail after adding xpack configuration in elasticsearch.yml.

Create a local user:

/usr/share/elasticsearch/bin/elasticsearch-users useradd  priyanka -r superuser -p PASSWORD
curl -u priyanka http://localhost:9200/_xpack/license
curl -X POST -u priyanka "http://localhost:9200/_license/start_trial?acknowledge=true&pretty"

Validate with AD user:

curl -u esadmin@puc.com http://localhost:9200/_xpack/license

The above commands are executed only in one master server.

curl -u esadmin@puc.com http://elasticsearch.powerupcloud.com/_xpack/license

And that’s all. Hope you found it useful.

Leave a Reply