Elasticsearch in Docker: threat intelligence with filebeat

Goals:

• collect observables from supported feeds
• collect observables from unsupported feeds with elastic-tip

Setup elasticsearch and kibana for filebeat

We could use superuser elastic to setup filebeat but we are going to use a dedicated user with just the minimum permissions.

Open Kibana and go to Stack Management > Security > Roles. Click Create role and enter the following settings:

• Role name: filebeat_threatintel_setup
• Cluster privileges: monitor, manage_ilm, manage_ml
• Index privileges:
• Indices: filebeat-*
• Privileges: manage, write, read

Click Create role.

Go to Stack Management > Security > Users. Click Create user and enter the following settings:

• Username: filebeat_threatintel_setup
• Privileges: filebeat_threatintel_setup, kibana_admin, ingest_admin, machine_learning_admin

Click Create user.

Now let’s setup the index, index templates, dashboards & so on. We do that by running filebeat setup once. We attach it to elastic network, pass it root CA, username and password of the user we just created, and the index name and policy.

⚠️ One important thing to know is: when you run filebeat setup command, it will import ALL available dashboards, even those you do not care about, even if you specify --modules on the command line. You can find several posts and issues on the subject:

• October 2018: discuss.elastic.co/t/load-only-some-metricbeat-dashboards/152899
• April 2019: discuss.elastic.co/t/load-one-dashboard-instead-of-all-filebeat-default-dashboards/176281
• July 2019: discuss.elastic.co/t/load-one-dashboard-instead-of-all-filebeat-default-dashboards-follow-up/188455
• January 2021: github.com/elastic/beats/issues/23651

If you just want to load the threat intel dashboards, you need to make all the other dashboards unavailable to filebeat setup. You can either download the dashboards from github and save them in a directory named dashboards or run the following commands:

# in a first terminal
docker run -it --rm --name ti docker.elastic.co/beats/filebeat:7.16.3 bash

# in a second terminal
mkdir dashboards

for i in Filebeat-threatintel-abuse-url.json Filebeat-threatintel-anomali.json Filebeat-threatintel-malwarebazaar.json Filebeat-threatintel-overview.json Filebeat-threatintel-alienvault-otx.json Filebeat-threatintel-aubse-malware.json Filebeat-threatintel-misp.json Filebeat-threatintel-recordedfuture.json ; do docker cp ti:/usr/share/filebeat/kibana/7/dashboard/$i dashboards/ ; done

When you have copied the file, you can exit from the first container.

Now run the filebeat setup command:

docker run --rm \
-v elastic_certs:/certs:ro \
--network elastic_default \
docker.elastic.co/beats/filebeat:7.16.3 setup \
-E output.elasticsearch.hosts=["https://es:9200"] \
-E output.elasticsearch.ssl.certificate_authorities=/certs/ca.crt \
-E output.elasticsearch.username=filebeat_threatintel_setup \
-E output.elasticsearch.password=password \
-E setup.kibana.host=kibana:5601 \
-E setup.kibana.protocol=https \
-E setup.kibana.ssl.certificate_authorities=/certs/ca.crt \
-E setup.kibana.username=filebeat_threatintel_setup \
-E setup.kibana.password=password \
-E setup.ilm.policy_name="7-days-default"

Send data to Elasticsearch

Create another user with just enough permissions to send data to Elasticsearch. Open Kibana and go to Stack Management > Security > Roles. Click Create role and enter the following settings:

• Role name: filebeat_threatintel_writer
• Cluster privileges: monitor, read_ilm, read_pipeline, manage_ingest_pipelines, manage_index_templates
• Index privileges:
• Indices: filebeat-*
• Privileges: create_doc, view_index_metadata, create_index

Click Create role.

⚠️ the documentation says to only provide cluster privileges monitor, read_ilm and read_pipeline. However, if you do not provide manage_ingest_pipelines and manage_index_templates, you will encounter connection issues.

Go to Stack Management > Security > Users. Click Create user and enter the following settings:

• Username: filebeat_threatintel_writer
• Privileges: filebeat_threatintel_writer

Click Create user.

Create a file named filebeat.yml with the following content.

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

output.elasticsearch:
  hosts: ${ELASTICSEARCH_HOSTS}
  username: '${ELASTICSEARCH_USERNAME}'
  password: '${ELASTICSEARCH_PASSWORD}'
  ssl:
    certificate_authorities: ["/certs/ca.crt"]

setup.ilm.check_exists: false

monitoring:
  enabled: true
  elasticsearch:
    username: '${MONITORING_USERNAME}'
    password: '${MONITORING_PASSWORD}'

You can find the base file for Docker on github. The complete filebeat.yml reference is available on Elastic website.

Retrieve default threat intel configuration file

docker run --rm docker.elastic.co/beats/filebeat:7.16.3 cat modules.d/threatintel.yml.disabled > threatintel.yml

Edit the file newly created to enable/disable and customize the supported feeds.

Create docker-compose.yml file with the following content:

version: '3'

services:
  filebeat:
    image: docker.elastic.co/beats/filebeat:7.16.3
    restart: always
    env_file:
      - ./.env
    networks:
      - elastic_default
    volumes:
      - ./filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro
      - ./threatintel.yml:/usr/share/filebeat/modules.d/threatintel.yml:ro
      - elastic_certs:/certs:ro
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 1000M
    memswap_limit: 1000M

networks:
  elastic_default:
    external: true

volumes:
  elastic_certs:
    external: true

Start the container: docker compose up

You should not witness any error and if you go to Kibana, you should see documents in the filebeat index as well as on the various threat intel dashboards.