DNSSEC Validation Failed

 

I was getting the following error in my browser DNSSEC validation failed for question IN DNSKEY: missing-key

I found out that ICANN scheduled a switch from an old DNS root key to a new one, TODAY. It seems that my Ubuntu 24.04 did not have the new key in place, so it was failing DNSSEC validation.

The fix is to add the new root key to the OS. To get the correct info, go to https://data.iana.org/root-anchors/root-anchors.xml.

Then add the key -

sudo mkdir -p /etc/dnssec-trust-anchors.d

sudo nano /etc/dnssec-trust-anchors.d/root.positive

#  Put these two lines in the file, you can review the values at the link above to make sure they are correct.

. IN DS 20326 8 2 E06D44B80B8F1D39A95C0B0D7C65D08458E880409BBC683457104237C7F8EC8D

. IN DS 38696 8 2 683D2D0ACB8C9B712A1948B27F741219298D0A450D612C483AF444A4C0FB2B16

# Restart the systemd-resolved service

sudo systemctl restart systemd-resolved

That should be it!

Read More

How to tracing processes using audit logs

 Một buổi sáng đẹp trời bạn nhận được cảnh báo từ SIEM:

Ghi nhận hành vi thực thi lệnh bất thường trên hệ thống

Log Source: SIEM @ a.b.c.d

Source IP: a.b.c.d

Command: "grep" a1="-qE" a2="209\.59\.141\.49|50\.28\.104\.57" a3="/root/.ssh/authorized_keys" a4="/root/.ssh/authorized_keys2"

Time: 15:30:05 8 thg 10, 2026

Rồi, xong.

Bước 1 : Lấy  full log theo ID

cat /var/log/cmdlog.log | grep 1580306382

Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.256:1580306382): arch=c000003e syscall=59 success=yes exit=0 a0=5595e7630c40 a1=5595e76303c0 a2=5595e7603700 a3=1b6 items=2 ppid=1498460 pid=1498463 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="grep" exe="/usr/bin/grep" key="T1081_Credentials_In_Files" ARCH=x86_64 SYSCALL=execve AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

Oct  8 01:30:01 a.b.c.d audispd: type=EXECVE msg=audit(1791397801.256:1580306382): argc=5 a0="grep" a1="-qE" a2="209\.59\.141\.49|50\.28\.104\.57" a3="/root/.ssh/authorized_keys" a4="/root/.ssh/authorized_keys2"

Oct  8 01:30:01 a.b.c.d audispd: type=PATH msg=audit(1791397801.256:1580306382): item=0 name="/usr/bin/grep" inode=693 dev=fc:02 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0 OUID="root" OGID="root"

Oct  8 01:30:01 a.b.c.d audispd: type=PATH msg=audit(1791397801.256:1580306382): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=2034 dev=fc:02 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0 OUID="root" OGID="root"

Oct  8 01:30:01 a.b.c.d audispd: type=PROCTITLE msg=audit(1791397801.256:1580306382): proctitle=67726570002D7145003230395C2E35395C2E3134315C2E34397C35305C2E32385C2E3130345C2E3537002F726F6F742F2E7373682F617574686F72697A65645F6B657973002F726F6F742F2E7373682F617574686F72697A65645F6B65797332

Bước 2: Lấy log theo process ID :

grep -nHE 'pid=1498460|ppid=1498460|ses=4090958' /var/log/cmdlog.log

grep -nE 'pid=1498460|ppid=1498460' /var/log/cmdlog.log

/var/log/cmd.log:181790:Oct  8 01:30:01 a.b.c.d audispd: type=LOGIN msg=audit(1791397801.184:1580305632): pid=1498388 uid=0 old-auid=4294967295 auid=0 tty=(none) old-ses=4294967295 ses=4090958 res=1 UID="root" OLD-AUID="unset" AUID="root"

/var/log/cmd.log:181791:Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.184:1580305632): arch=c000003e syscall=1 success=yes exit=1 a0=7 a1=7ffeb0b7b050 a2=1 a3=7f0c21b9d371 items=0 ppid=2962086 pid=1498388 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="cron" exe="/usr/sbin/cron" key=(null) ARCH=x86_64 SYSCALL=write AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

/var/log/cmd.log:181794:Oct  8 01:30:01 a.b.c.d audispd: type=USER_START msg=audit(1791397801.184:1580305634): pid=1498388 uid=0 auid=0 ses=4090958 msg='op=PAM:session_open grantors=pam_loginuid,pam_env,pam_env,pam_permit,pam_umask,pam_unix,pam_limits acct="root" exe="/usr/sbin/cron" hostname=? addr=? terminal=cron res=success' UID="root" AUID="root"

/var/log/cmd.log:181809:Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.184:1580305643): arch=c000003e syscall=59 success=yes exit=0 a0=55d1cc9f3fc6 a1=7ffeb0b7b0d0 a2=55d1cc9f3fe0 a3=7ffeb0b7b0d0 items=2 ppid=1498388 pid=1498397 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="sh" exe="/usr/bin/dash" key="susp_shell" ARCH=x86_64 SYSCALL=execve AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

/var/log/cmd.log:181825:Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.188:1580305647): arch=c000003e syscall=59 success=yes exit=0 a0=7ffc0b0b0ed0 a1=7ffc0b0b10f0 a2=7ffc0b0b1110 a3=7f15598c0640 items=2 ppid=1498397 pid=1498400 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="bash" exe="/usr/bin/bash" key="susp_shell" ARCH=x86_64 SYSCALL=execve AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

/var/log/cmd.log:182082:Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.196:1580305791): arch=c000003e syscall=59 success=yes exit=0 a0=5595e76042f0 a1=5595e76045f0 a2=5595e76043c0 a3=8 items=2 ppid=1498400 pid=1498412 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="whoami" exe="/usr/bin/whoami" key="recon" ARCH=x86_64 SYSCALL=execve AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

/var/log/cmd.log:182294:Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.252:1580306378): arch=c000003e syscall=59 success=yes exit=0 a0=5595e7630760 a1=5595e7631c00 a2=5595e7603700 a3=1b6 items=2 ppid=1498460 pid=1498461 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="grep" exe="/usr/bin/grep" key="T1081_Credentials_In_Files" ARCH=x86_64 SYSCALL=execve AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

/var/log/cmd.log:182295:Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.252:1580306379): arch=c000003e syscall=59 success=yes exit=0 a0=5595e7630400 a1=5595e7631c00 a2=5595e7603700 a3=1b6 items=2 ppid=1498460 pid=1498462 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="grep" exe="/usr/bin/grep" key="T1081_Credentials_In_Files" ARCH=x86_64 SYSCALL=execve AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

/var/log/cmd.log:182302:Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.256:1580306382): arch=c000003e syscall=59 success=yes exit=0 a0=5595e7630c40 a1=5595e76303c0 a2=5595e7603700 a3=1b6 items=2 ppid=1498460 pid=1498463 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="grep" exe="/usr/bin/grep" key="T1081_Credentials_In_Files" ARCH=x86_64 SYSCALL=execve AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

/var/log/cmd.log:182317:Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.820:1580306389): arch=c000003e syscall=59 success=yes exit=0 a0=5595e7632590 a1=5595e7636470 a2=5595e7632040 a3=8 items=2 ppid=1498460 pid=1499143 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="grep" exe="/usr/bin/grep" key="T1081_Credentials_In_Files" ARCH=x86_64 SYSCALL=execve AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

/var/log/cmd.log:182322:Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.832:1580306390): arch=c000003e syscall=59 success=yes exit=0 a0=5595e7630000 a1=5595e7636770 a2=5595e7632040 a3=8 items=2 ppid=1498460 pid=1499145 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="grep" exe="/usr/bin/grep" key="T1081_Credentials_In_Files" ARCH=x86_64 SYSCALL=execve AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

/var/log/cmd.log:182327:Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.848:1580306391): arch=c000003e syscall=59 success=yes exit=0 a0=5595e76192e0 a1=5595e7616c40 a2=5595e76154b0 a3=8 items=2 ppid=1498400 pid=1499151 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="hostname" exe="/usr/bin/hostname" key="recon" ARCH=x86_64 SYSCALL=execve AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

/var/log/cmd.log:182332:Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.860:1580306392): arch=c000003e syscall=59 success=yes exit=0 a0=5595e75fc3e0 a1=5595e75fcea0 a2=5595e75fc480 a3=5595e75f2010 items=2 ppid=1498400 pid=1499153 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="whoami" exe="/usr/bin/whoami" key="recon" ARCH=x86_64 SYSCALL=execve AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

/var/log/cmd.log:182337:Oct  8 01:30:01 a.b.c.d audispd: type=CRED_DISP msg=audit(1791397801.868:1580306393): pid=1498388 uid=0 auid=0 ses=4090958 msg='op=PAM:setcred grantors=pam_hulk,pam_permit acct="root" exe="/usr/sbin/cron" hostname=? addr=? terminal=cron res=success' UID="root" AUID="root"

/var/log/cmd.log:182338:Oct  8 01:30:01 a.b.c.d audispd: type=USER_END msg=audit(1791397801.868:1580306394): pid=1498388 uid=0 auid=0 ses=4090958 msg='op=PAM:session_close grantors=pam_loginuid,pam_env,pam_env,pam_permit,pam_umask,pam_unix,pam_limits acct="root" exe="/usr/sbin/cron" hostname=? addr=? terminal=cron res=success' UID="root" AUID="root"

182294:Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.252:1580306378): arch=c000003e syscall=59 success=yes exit=0 a0=5595e7630760 a1=5595e7631c00 a2=5595e7603700 a3=1b6 items=2 ppid=1498460 pid=1498461 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="grep" exe="/usr/bin/grep" key="T1081_Credentials_In_Files" ARCH=x86_64 SYSCALL=execve AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

182295:Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.252:1580306379): arch=c000003e syscall=59 success=yes exit=0 a0=5595e7630400 a1=5595e7631c00 a2=5595e7603700 a3=1b6 items=2 ppid=1498460 pid=1498462 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="grep" exe="/usr/bin/grep" key="T1081_Credentials_In_Files" ARCH=x86_64 SYSCALL=execve AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

182302:Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.256:1580306382): arch=c000003e syscall=59 success=yes exit=0 a0=5595e7630c40 a1=5595e76303c0 a2=5595e7603700 a3=1b6 items=2 ppid=1498460 pid=1498463 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="grep" exe="/usr/bin/grep" key="T1081_Credentials_In_Files" ARCH=x86_64 SYSCALL=execve AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

182317:Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.820:1580306389): arch=c000003e syscall=59 success=yes exit=0 a0=5595e7632590 a1=5595e7636470 a2=5595e7632040 a3=8 items=2 ppid=1498460 pid=1499143 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="grep" exe="/usr/bin/grep" key="T1081_Credentials_In_Files" ARCH=x86_64 SYSCALL=execve AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

182322:Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.832:1580306390): arch=c000003e syscall=59 success=yes exit=0 a0=5595e7630000 a1=5595e7636770 a2=5595e7632040 a3=8 items=2 ppid=1498460 pid=1499145 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="grep" exe="/usr/bin/grep" key="T1081_Credentials_In_Files" ARCH=x86_64 SYSCALL=execve AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

181809:Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.184:1580305643): arch=c000003e syscall=59 success=yes exit=0 a0=55d1cc9f3fc6 a1=7ffeb0b7b0d0 a2=55d1cc9f3fe0 a3=7ffeb0b7b0d0 items=2 ppid=1498388 pid=1498397 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="sh" exe="/usr/bin/dash" key="susp_shell" ARCH=x86_64 SYSCALL=execve AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

181825:Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.188:1580305647): arch=c000003e syscall=59 success=yes exit=0 a0=7ffc0b0b0ed0 a1=7ffc0b0b10f0 a2=7ffc0b0b1110 a3=7f15598c0640 items=2 ppid=1498397 pid=1498400 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="bash" exe="/usr/bin/bash" key="susp_shell" ARCH=x86_64 SYSCALL=execve AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

182082:Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.196:1580305791): arch=c000003e syscall=59 success=yes exit=0 a0=5595e76042f0 a1=5595e76045f0 a2=5595e76043c0 a3=8 items=2 ppid=1498400 pid=1498412 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="whoami" exe="/usr/bin/whoami" key="recon" ARCH=x86_64 SYSCALL=execve AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

182327:Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.848:1580306391): arch=c000003e syscall=59 success=yes exit=0 a0=5595e76192e0 a1=5595e7616c40 a2=5595e76154b0 a3=8 items=2 ppid=1498400 pid=1499151 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="hostname" exe="/usr/bin/hostname" key="recon" ARCH=x86_64 SYSCALL=execve AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

182332:Oct  8 01:30:01 a.b.c.d audispd: type=SYSCALL msg=audit(1791397801.860:1580306392): arch=c000003e syscall=59 success=yes exit=0 a0=5595e75fc3e0 a1=5595e75fcea0 a2=5595e75fc480 a3=5595e75f2010 items=2 ppid=1498400 pid=1499153 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4090958 comm="whoami" exe="/usr/bin/whoami" key="recon" ARCH=x86_64 SYSCALL=execve AUID="root" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"

Bước 3: Đã xác định được Cron, lấy log Cron 

grep -hE 'Oct  8 01:30:0[01].*CRON'   /var/log/syslog* /var/log/cron* 2>/dev/null

Oct  8 01:30:01 a.b.c.d CRON[1498397]: (root) CMD (/usr/local/maldetect/maldet --mkpubpaths >> /dev/null 2>&1)

Oct  8 01:30:01 a.b.c.d CRON[1498396]: (root) CMD (   bash -c "sleep $((RANDOM % 60))" ; /opt/imunify360/venv/share/imunify360/scripts/check-detached.py > /dev/null 2>&1 || :)

Oct  8 01:30:01 a.b.c.d CRON[1498398]: (root) CMD (/usr/sbin/imunify-notifier -update-cron)

Oct  8 01:30:01 a.b.c.d CRON[1498399]: (cicocom) CMD (/usr/local/bin/php/home/cicocom/public_html/artisan queue:work >/dev/null 2>&1)

Oct  8 01:30:01 a.b.c.d CRON[1498401]: (root) CMD (/usr/local/cpanel/scripts/dcpumon-wrapper >/dev/null 2>&1)

Oct  8 01:30:01 a.b.c.d CRON[1498397]: (root) CMD (/usr/local/maldetect/maldet --mkpubpaths >> /dev/null 2>&1)

Oct  8 01:30:01 a.b.c.d CRON[1498396]: (root) CMD (   bash -c "sleep $((RANDOM % 60))" ; /opt/imunify360/venv/share/imunify360/scripts/check-detached.py > /dev/null 2>&1 || :)

Oct  8 01:30:01 a.b.c.d CRON[1498398]: (root) CMD (/usr/sbin/imunify-notifier -update-cron)

Oct  8 01:30:01 a.b.c.d CRON[1498399]: (cicocom) CMD (/usr/local/bin/php/home/cicocom/public_html/artisan queue:work >/dev/null 2>&1)

Oct  8 01:30:01 a.b.c.d CRON[1498401]: (root) CMD (/usr/local/cpanel/scripts/dcpumon-wrapper >/dev/null 2>&1)

Xác định được ID : 1498397 thực thi là maldet

Thở phào nhẹ nhõm

Lệnh thực thi trong maldet 

    # G ?~@~T SSH keys carrying known-attacker IP labels. Broader Pattern G

    # (forged-mtime detection) has customer-paste FP risk; left to ioc-scan.

    grep -qE '209\.59\.141\.49|50\.28\.104\.57' \

        /root/.ssh/authorized_keys /root/.ssh/authorized_keys2 2>/dev/null \

        && h="$h G"

Mission Completed

Read More