OpenShift v4.x – Apache Log4j2 (CVE-2021-44228) 보안 취약점 조치 방법

OpenShift v3.11, v4.x 버전에서 Apache Log4j2 (CVE-2021-44228)에 대한 보안 취약점 조치 방법에 대해서 정리한다.
이번 보안 취약점에 영향이 있는 컴포넌트는 OpenShift Logging(EFK Stack)의 ElasticSearch에 영향을 받는다.

ElasticSearch의 Deployment Config을 확인 한다.

[root@bastion ~]# for i in {es,es-ops}; do oc get dc -l component=$i --no-headers -n openshift-logging | awk '{print $1}'; done
logging-es-data-master-u1hrzwwd
logging-es-ops-data-master-bxc3eg5d

Deployment Config에 log4j2에 대한 NoLookups에 대한 변수를 추가 한다.

[root@bastion ~]# oc set env dc logging-es-data-master-u1hrzwwd ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true" -n openshift-logging
[root@bastion ~]# oc set env dc logging-es-ops-data-master-bxc3eg5d ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true" -n openshift-logging

Deployment Config에 추가한 변수를 Replication Controller에 반영하기 위해 Rollout 명령어를 통해 배포한다.

[root@bastion ~]# oc rollout latest dc/logging-es-data-master-u1hrzwwd -n openshift-logging
[root@bastion ~]# oc rollout latest dc/logging-es-ops-data-master-bxc3eg5d -n openshift-logging

[root@bastion ~]# for i in $(oc get pod -n openshift-logging | grep logging-es | awk '{print $1}'); do oc exec $i -c elasticsearch -n openshift-logging -- env | grep ES_JAVA_OPTS; done
ES_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true
ES_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true

결과값에서 -Dlog4j2.formatMsgNoLookups=true 값이 있으면 됨.

[root@bastion ~]# for i in $(oc get pod -n openshift-logging | grep logging-es | awk '{print $1}'); do \
oc exec -c elasticsearch -n openshift-logging $i -- grep -a log4j2.formatMsgNoLookups /proc/1/cmdline && echo; done
/bin/java-XX:+UseConcMarkSweepGC-XX:CMSInitiatingOccupancyFraction=75-XX:+UseCMSInitiatingOccupancyOnly-XX:+AlwaysPreTouch-server-Xss1m-Djava.awt.headless=true-Dfile.encoding=UTF-8-Djna.nosys=true-Djdk.io.permissionsUseCanonicalPath=true-Dio.netty.noUnsafe=true-Dio.netty.noKeySetOptimization=true-Dio.netty.recycler.maxCapacityPerThread=0-Dlog4j.shutdownHookEnabled=false-Dlog4j2.disable.jmx=true-Dlog4j.skipJansi=true-XX:+HeapDumpOnOutOfMemoryError-XX:+UnlockExperimentalVMOptions-XX:+UseCGroupMemoryLimitForHeap-XX:MaxRAMFraction=2-XX:InitialRAMFraction=2-XX:MinRAMFraction=2-Dlog4j2.formatMsgNoLookups=true-Xms3072m-Xmx3072m-XX:HeapDumpPath=/elasticsearch/persistent/heapdump.hprof-Dsg.display_lic_none=false-Dio.netty.recycler.maxCapacityPerThread=0-Dio.netty.allocator.type=unpooled-Des.path.home=/usr/share/elasticsearch-cp/usr/share/elasticsearch/lib/*org.elasticsearch.bootstrap.Elasticsearch-Epath.conf=/etc/elasticsearch/

Operator Hub에서 제공하는 Cluster Logging Operator와 ElasticSearch Operator의 Instance에서 Operator가 자동 관리 하지 않도록 비활성화 한다.

[root@bastion ~]# oc patch clusterlogging/instance -n openshift-logging \
--type=merge --patch '{"spec": {"managementState": "Unmanaged"}}'

[root@bastion ~]# oc patch elasticsearch/elasticsearch -n openshift-logging \
--type=merge --patch '{"spec": {"managementState": "Unmanaged"}}'

Deployment 변수 추가를 위해 배포를 활성화(paused: false) 한다.

[root@bastion ~]# oc get deployment -l component=elasticsearch --no-headers -n openshift-logging | awk '{print $1}'
elasticsearch-cdm-7jf12usg-1
elasticsearch-cdm-7jf12usg-2
elasticsearch-cdm-7jf12usg-3

[root@bastion ~]# oc patch deployment/elasticsearch-cdm-7jf12usg-1 -n openshift-logging --type=merge -p '{"spec":{"paused": false}}'
[root@bastion ~]# oc patch deployment/elasticsearch-cdm-7jf12usg-2 -n openshift-logging --type=merge -p '{"spec":{"paused": false}}'
[root@bastion ~]# oc patch deployment/elasticsearch-cdm-7jf12usg-3 -n openshift-logging --type=merge -p '{"spec":{"paused": false}}'

Deployment Config에 log4j2에 대한 NoLookups에 대한 변수를 추가 한다.

[root@bastion ~]# oc set env deployment/elasticsearch-cdm-7jf12usg-1 -c elasticsearch -n openshift-logging ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true"
[root@bastion ~]# oc set env deployment/elasticsearch-cdm-7jf12usg-2 -c elasticsearch -n openshift-logging ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true"
[root@bastion ~]# oc set env deployment/elasticsearch-cdm-7jf12usg-3 -c elasticsearch -n openshift-logging ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true"

[root@bastion ~]# for i in $(oc get pod -l component=elasticsearch --no-headers -n openshift-logging | awk '{print $1}'); do oc exec $i -c elasticsearch -n openshift-logging -- env | grep ES_JAVA_OPTS; done
ES_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true
ES_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true
ES_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true

결과값에서 -Dlog4j2.formatMsgNoLookups=true 값이 있으면 됨.

[root@bastion ~]# for i in $(oc get pod -l component=elasticsearch --no-headers -n openshift-logging | awk '{print $1}'); do \
oc exec -c elasticsearch -n openshift-logging $i -- grep -a log4j2.formatMsgNoLookups /proc/1/cmdline && echo ; done

/usr/lib/jvm/jre/bin/java-XX:+UseConcMarkSweepGC-XX:CMSInitiatingOccupancyFraction=75-XX:+UseCMSInitiatingOccupancyOnly-Des.networkaddress.cache.ttl=60-Des.networkaddress.cache.negative.ttl=10-XX:+AlwaysPreTouch-Xss1m-Djava.awt.headless=true-Dfile.encoding=UTF-8-Djna.nosys=true-XX:-OmitStackTraceInFastThrow-Dio.netty.noUnsafe=true-Dio.netty.noKeySetOptimization=true-Dio.netty.recycler.maxCapacityPerThread=0-Dlog4j.shutdownHookEnabled=false-Dlog4j2.disable.jmx=true-Djava.io.tmpdir=/tmp/elasticsearch-4042163088121998817-XX:+HeapDumpOnOutOfMemoryError-XX:HeapDumpPath=data-XX:ErrorFile=logs/hs_err_pid%p.log-Djava.locale.providers=COMPAT-XX:UseAVX=2-XX:+UnlockExperimentalVMOptions-XX:MaxRAMFraction=2-XX:InitialRAMFraction=2-XX:MinRAMFraction=2-Dlog4j2.formatMsgNoLookups=true-Xms4096m-Xmx4096m-XX:HeapDumpPath=/elasticsearch/persistent/heapdump.hprof-Xlog:gc*,gc+age=trace,safepoint:file=/elasticsearch/persistent/elasticsearch/logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m-XX:ErrorFile=/elasticsearch/persistent/elasticsearch/logs/error.log-Djdk.tls.ephemeralDHKeySize=2048-Des.path.home=/usr/share/elasticsearch-Des.path.conf=/etc/elasticsearch-Des.distribution.flavor=oss-Des.distribution.type=zip-cp/usr/share/elasticsearch/lib/*org.elasticsearch.bootstrap.Elasticsearch

OpenShift v4.x 버전의 Jaeger Operator에서 사용하는 ElasticSearch Operator에 log4j2 보안 취약점을 설정한다.

[root@bastion ~]# oc patch elasticsearch/elasticsearch -n istio-system --type=merge --patch '{"spec": {"managementState": "Unmanaged"}}'

[root@bastion ~]# oc get deployment -l component=elasticsearch --no-headers -n istio-system | awk '{print $1}'
elasticsearch-cdm-istiosystemjaeger-1

Deployment Config에 log4j2에 대한 NoLookups에 대한 변수를 추가 한다.

[root@bastion ~]# oc set env deployment/elasticsearch-cdm-istiosystemjaeger-1 -c elasticsearch -n istio-system ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true"

[root@bastion ~]# oc patch deployment/elasticsearch-cdm-istiosystemjaeger-1 --type=merge -p '{"spec":{"paused": false}}'

[root@bastion ~]# for i in $(oc get pod -l component=elasticsearch --no-headers -n istio-system | awk '{print $1}'); do oc exec $i -c elasticsearch -n istio-system -- env | grep ES_JAVA_OPTS; done

결과값에서 -Dlog4j2.formatMsgNoLookups=true 값이 있으면 된다.

[root@bastion ~]# for i in $(oc get pod -l component=elasticsearch --no-headers -n istio-system | awk '{print $1}'); do \
oc exec -c elasticsearch -n istio-system $i -- grep -a log4j2.formatMsgNoLookups /proc/1/cmdline && echo ; done

/usr/lib/jvm/jre/bin/java-XX:+UseConcMarkSweepGC-XX:CMSInitiatingOccupancyFraction=75-XX:+UseCMSInitiatingOccupancyOnly-Des.networkaddress.cache.ttl=60-Des.networkaddress.cache.negative.ttl=10-XX:+AlwaysPreTouch-Xss1m-Djava.awt.headless=true-Dfile.encoding=UTF-8-Djna.nosys=true-XX:-OmitStackTraceInFastThrow-Dio.netty.noUnsafe=true-Dio.netty.noKeySetOptimization=true-Dio.netty.recycler.maxCapacityPerThread=0-Dlog4j.shutdownHookEnabled=false-Dlog4j2.disable.jmx=true-Djava.io.tmpdir=/tmp/elasticsearch-5238943379919792649-XX:+HeapDumpOnOutOfMemoryError-XX:HeapDumpPath=data-XX:ErrorFile=logs/hs_err_pid%p.log-Djava.locale.providers=COMPAT-XX:UseAVX=2-XX:+UnlockExperimentalVMOptions-XX:MaxRAMFraction=2-XX:InitialRAMFraction=2-XX:MinRAMFraction=2-Dlog4j2.formatMsgNoLookups=true-Xms1024m-Xmx1024m-XX:HeapDumpPath=/elasticsearch/persistent/heapdump.hprof-Xlog:gc*,gc+age=trace,safepoint:file=/elasticsearch/persistent/elasticsearch/logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m-XX:ErrorFile=/elasticsearch/persistent/elasticsearch/logs/error.log-Djdk.tls.ephemeralDHKeySize=2048-Des.path.home=/usr/share/elasticsearch-Des.path.conf=/etc/elasticsearch-Des.distribution.flavor=oss-Des.distribution.type=zip-cp/usr/share/elasticsearch/lib/*org.elasticsearch.bootstrap.Elasticsearch

[1]: RedHat Knowledgebase – OCP4: CVE-2021-44228 affecting Elasticsearch (Red Hat OpenShift Logging)
[2]: RedHat Security – CVE-2021-44228
[3]: Redhat BugZilla – log4j-core: Remote code execution in Log4j 2.x when logs contain an attacker-controlled string value