# Tmp: FreeBSD DTrace 產生 Flame Graph


可以看到有 profile-97, profile-199 ... profile-4999 還有 tick-1 ... tick-5000，前者是 CPU sampling profiler（基於 interrupt / PMC / sampling），後者是 scheduler tick / timer interrupt sampling 適合 long-run performance trend。

```sh
kola@:~/proj/mac_casper/test/performance/syscall/sysctl $ sudo dtrace \
-x stackframes=100 \
-o stacks.out \
-c ./perf_baseline \
-n '
profile-4999
/pid == $target/
{
    @[stack()] = count();
}'

kola@:~/proj/mac_casper/test/performance/syscall/sysctl $ perl ~/proj/FlameGraph/stackcollapse.pl stacks.out > folded.out

kola@:~/proj/mac_casper/test/performance/syscall/sysctl $ perl ~/proj/FlameGraph/flamegraph.pl \
    --title="sysctl kernel stack with mac" \
    folded.out > flame.svg
```

## 實際例子

存成 SVG 後你就可以在瀏覽器打開，可以自由跟它互動像是看到全名等。

### sysctl 函式

#### 沒有 MAC 模組執行 sysctl

#### 有 MAC 模組時執行 sysctl

![alt text](image.png)

總共 2495 samples 因為時間花的比較多，追蹤如下：

1. Function: kernel`0xffffffff81050895 (333 samples, 13.35%)
2. Function: kernel`0xffffffff810508ca (3 samples, 0.12%)
3. Function: kernel`0xffffffff810508f9 (6 samples, 0.24%)
4. Function: kernel`_rm_runlock (7 samples, 0.28%)
5. Function: kernel`amd64_syscall (611 samples, 24.49%)
   1. Function: kernel`_rm_rlock (58 samples, 2.32%)
   2. Function: kernel`_rm_runlock (40 samples, 1.60%)
   3. Function: kernel`mac_error_select (4 samples, 0.16%)
   4. Function: kernel`mac_policy_sunlock_nosleep (9 samples, 0.36%)
   5. Function: kernel`mac_system_check_sysctl (19 samples, 0.76%)
   6. Function: kernel`prison_allow (7 samples, 0.28%)
   7. Function: kernel`prison_priv_check (2 samples, 0.08%)
   8. Function: kernel`sys___sysctl (434 samples, 17.39%)
      1. Function: kernel`_rm_rlock (82 samples, 3.29%)
      2. Function: kernel`_rm_runlock (51 samples, 2.04%)
      3. Function: kernel`mac_error_select (8 samples, 0.32%)
      4. Function: kernel`mac_label_get (1 samples, 0.04%)
      5. Function: kernel`mac_policy_slock_nosleep (2 samples, 0.08%)
      6. Function: kernel`mac_policy_sunlock_nosleep (1 samples, 0.04%)
      7. Function: kernel`strlen (3 samples, 0.12%)
      8. Function: kernel`sysctl_handle_string (35 samples, 1.40%)
      9. Function: kernel`sysctl_old_user (16 samples, 0.64%)
         1. Function: kernel`copyout_smap_erms (3 samples, 0.12%)
         2. Function: kernel`mac_label_get (1 samples, 0.04%)
         3. Function: kernel`strlen (33 samples, 1.32%)
         4. Function: kernel`sysctl_old_user (22 samples, 0.88%)
         5. Function: kernel`sysctl_root (172 samples, 6.89%)
         6. Function: mac_casper.ko`casper_mpo_system_check_sysctl_t (4 samples, 0.16%)

最前面的 unresolved kernel symbols (或者是 inlined / stripped functions)，可以用 `addr2line` 來看一下：

```sh
kola@:~ $ for a in 0xffffffff81050895 0xffffffff810508ca 0xffffffff810508f9; do
    addr2line -e /boot/kernel/kernel  -f -C $a
done
fast_syscall_common
/usr/src/sys/amd64/amd64/exception.S:553
fast_syscall_common
/usr/src/sys/amd64/amd64/exception.S:561
fast_syscall_common
/usr/src/sys/amd64/amd64/exception.S:568
kola@:~ $ 
```

這些是 FreeBSD 的 syscall entry path（快速 syscall 入口）。

然後再來使用 read-mostly lock，執行 `amd64_syscall` 這是 amd64 最上層 trap。

MAC 的

1. `mac_error_select`: 對很多 policy 做 select，只要有人 deny　就不過
2. `mac_policy_sunlock_nosleep`: lock policy list
3. `mac_system_check_sysctl`: 可不可以 do sysctl
4. `prison_allow`: 看看 sysctl allowed inside jail
5. `prison_priv_check`: privilege escalation check inside jail
6. `sys___sysctl`: 開始做 sysctl
   1. `sysctl_handle_string`: 處理 string 類型
   2. `sysctl_old_user`: 把 kernel　資料複製到 user space
      1. `copyout_smap_erms`（真正寫到 user space）
      2. `mac_label_get`: 獲得 label
      3. `sysctl_root`: 整個 sysctl dispatcher
      4. `casper_mpo_system_check_sysctl_t`: 我的 mac module

#### 比較

MAC module 會做：

1. `mac_error_select`: 8 samples
2. `mac_policy_sunlock_nosleep`: 23 samples
3. `mac_system_check_sysctl`: 從 38 samples 變成了 88 samples
4. `_rm_rlock`: 從 180 samples 變成 210 samples
5. `_rm_runlock`: 從 13 samples 變成 204 samples
6. `mac_error_select`: 57 samples
7. `mac_policy_slock_nosleep`: 13 samples
8. `mac_label_get`: 8 samples
9. `strlen`: 34 samples 變成 83 samples