Tmp: FreeBSD DTrace 產生 Flame Graph

可以看到有 profile-97, profile-199 … profile-4999 還有 tick-1 … tick-5000，前者是 CPU sampling profiler（基於 interrupt / PMC / sampling），後者是 scheduler tick / timer interrupt sampling 適合 long-run performance trend。

kola@:~/proj/mac_casper/test/performance/syscall/sysctl $ sudo dtrace \
-x stackframes=100 \
-o stacks.out \
-c ./perf_baseline \
-n '
profile-4999
/pid == $target/
{
    @[stack()] = count();
}'

kola@:~/proj/mac_casper/test/performance/syscall/sysctl $ perl ~/proj/FlameGraph/stackcollapse.pl stacks.out > folded.out

kola@:~/proj/mac_casper/test/performance/syscall/sysctl $ perl ~/proj/FlameGraph/flamegraph.pl \
    --title="sysctl kernel stack with mac" \
    folded.out > flame.svg

實際例子

存成 SVG 後你就可以在瀏覽器打開，可以自由跟它互動像是看到全名等。

sysctl 函式

沒有 MAC 模組執行 sysctl

有 MAC 模組時執行 sysctl

總共 2495 samples 因為時間花的比較多，追蹤如下：

1. Function: kernel`0xffffffff81050895 (333 samples, 13.35%)
2. Function: kernel`0xffffffff810508ca (3 samples, 0.12%)
3. Function: kernel`0xffffffff810508f9 (6 samples, 0.24%)
4. Function: kernel`_rm_runlock (7 samples, 0.28%)
5. Function: kernel`amd64_syscall (611 samples, 24.49%)
   1. Function: kernel`_rm_rlock (58 samples, 2.32%)
   2. Function: kernel`_rm_runlock (40 samples, 1.60%)
   3. Function: kernel`mac_error_select (4 samples, 0.16%)
   4. Function: kernel`mac_policy_sunlock_nosleep (9 samples, 0.36%)
   5. Function: kernel`mac_system_check_sysctl (19 samples, 0.76%)
   6. Function: kernel`prison_allow (7 samples, 0.28%)
   7. Function: kernel`prison_priv_check (2 samples, 0.08%)
   8. Function: kernel`sys___sysctl (434 samples, 17.39%)
      1. Function: kernel`_rm_rlock (82 samples, 3.29%)
      2. Function: kernel`_rm_runlock (51 samples, 2.04%)
      3. Function: kernel`mac_error_select (8 samples, 0.32%)
      4. Function: kernel`mac_label_get (1 samples, 0.04%)
      5. Function: kernel`mac_policy_slock_nosleep (2 samples, 0.08%)
      6. Function: kernel`mac_policy_sunlock_nosleep (1 samples, 0.04%)
      7. Function: kernel`strlen (3 samples, 0.12%)
      8. Function: kernel`sysctl_handle_string (35 samples, 1.40%)
      9. Function: kernel`sysctl_old_user (16 samples, 0.64%)
         1. Function: kernel`copyout_smap_erms (3 samples, 0.12%)
         2. Function: kernel`mac_label_get (1 samples, 0.04%)
         3. Function: kernel`strlen (33 samples, 1.32%)
         4. Function: kernel`sysctl_old_user (22 samples, 0.88%)
         5. Function: kernel`sysctl_root (172 samples, 6.89%)
         6. Function: mac_casper.ko`casper_mpo_system_check_sysctl_t (4 samples, 0.16%)

最前面的 unresolved kernel symbols (或者是 inlined / stripped functions)，可以用 addr2line 來看一下：

kola@:~ $ for a in 0xffffffff81050895 0xffffffff810508ca 0xffffffff810508f9; do
    addr2line -e /boot/kernel/kernel  -f -C $a
done
fast_syscall_common
/usr/src/sys/amd64/amd64/exception.S:553
fast_syscall_common
/usr/src/sys/amd64/amd64/exception.S:561
fast_syscall_common
/usr/src/sys/amd64/amd64/exception.S:568
kola@:~ $ 

這些是 FreeBSD 的 syscall entry path（快速 syscall 入口）。

然後再來使用 read-mostly lock，執行 amd64_syscall 這是 amd64 最上層 trap。

MAC 的

1. mac_error_select: 對很多 policy 做 select，只要有人 deny　就不過
2. mac_policy_sunlock_nosleep: lock policy list
3. mac_system_check_sysctl: 可不可以 do sysctl
4. prison_allow: 看看 sysctl allowed inside jail
5. prison_priv_check: privilege escalation check inside jail
6. sys___sysctl: 開始做 sysctl
   1. sysctl_handle_string: 處理 string 類型
   2. sysctl_old_user: 把 kernel　資料複製到 user space
      1. copyout_smap_erms（真正寫到 user space）
      2. mac_label_get: 獲得 label
      3. sysctl_root: 整個 sysctl dispatcher
      4. casper_mpo_system_check_sysctl_t: 我的 mac module

比較

MAC module 會做：

1. mac_error_select: 8 samples
2. mac_policy_sunlock_nosleep: 23 samples
3. mac_system_check_sysctl: 從 38 samples 變成了 88 samples
4. _rm_rlock: 從 180 samples 變成 210 samples
5. _rm_runlock: 從 13 samples 變成 204 samples
6. mac_error_select: 57 samples
7. mac_policy_slock_nosleep: 13 samples
8. mac_label_get: 8 samples
9. strlen: 34 samples 變成 83 samples