# Appendix: NVIDIA Nsight Systems profiling

SVT-AV1 has an opt-in build flag (`SVT_AV1_NVTX`, default OFF) that wires the
encoder up for **NVIDIA Nsight Systems**. The target is NVIDIA Grace CPU
(aarch64 Neoverse V2) but it works on any Linux host with the NVTX3 headers
installed. One `nsys profile` run replaces the typical mix of
`perf record`, `perf stat`, `perf sched`, and `perf trace` and adds per-stage
NVTX ranges on top.

| Capability                              | Common ad-hoc tool   | This integration                       |
| --                                      | --                   | --                                     |
| CPU sampling / hot functions             | `perf record -g`     | `nsys profile --sample=process-tree`   |
| PMU counters (cycles, IPC, cache, branch)| `perf stat`          | `nsys profile --cpu-core-events=...` (root or `kernel.perf_event_paranoid <= 0`) |
| Thread state timeline                    | `perf sched`         | `nsys profile --cpuctxsw=process-tree` |
| OS runtime trace (futex, syscall, pthread) | `perf trace -s`    | `nsys profile -t osrt`                 |
| Per-stage timing                         | (none, ad-hoc)       | NVTX `:wait` ranges per pipeline stage thread (gap == busy) |
| End-to-end FPS                           | encoder built-in     | unchanged (encoder reports as before)  |
| Per-frame PSNR/SSIM                      | `--enable-stat-report 1` | unchanged                          |

Default is OFF. With NVTX disabled, the encoder bitstream is byte-identical to
a build without the option (verified via md5 between `-DSVT_AV1_NVTX=OFF` and
`-DSVT_AV1_NVTX=ON` builds with the same input).

## Prerequisites

- **NVTX3 headers**: shipped with the CUDA Toolkit (`/usr/local/cuda/include/nvtx3/`)
  or the NVIDIA HPC SDK (`/opt/nvidia/hpc_sdk/Linux_aarch64/<ver>/cuda/include/nvtx3/`).
  CMake searches `CUDA_HOME`, `NVHPC_ROOT`, the standard CUDA install paths, and
  finally `/usr/include`. Set `CUDA_HOME` if your install lives elsewhere.
- **Nsight Systems CLI** (`nsys`): for capture and post-processing. Comes with
  the CUDA Toolkit and the HPC SDK; also available as a standalone download
  from NVIDIA.
- (Optional) **`kernel.perf_event_paranoid <= 2`** for CPU sampling, and `<= 0`
  (or root) for `--cpu-core-events`. On a fresh Grace install, sampling
  typically works out-of-the-box; in containers the sysctl is often locked to
  3 or 4 and only NVTX/OSRT/ctxsw will populate.

## Build

```sh
cd Build/linux
./build.sh release --enable-nvtx
```

Maps to `-DSVT_AV1_NVTX=ON`. The library gains lazy `dlopen`/`dlsym` imports
for the NVTX3 runtime; there is **no hard link dependency** on
`libnvToolsExt`. If the runtime is absent at execution, NVTX calls become
no-ops courtesy of NVTX3's lazy loader.

```sh
ldd Bin/Release/libSvtAv1Enc.so | grep -i nvtx     # expect: empty
nm  Bin/Release/libSvtAv1Enc.so | grep -i nvtx     # expect: empty (NVTX3 funcs are static inline)
nm -D Bin/Release/libSvtAv1Enc.so | grep -E " U dlopen| U dlsym"   # expect: present
```

`SvtAv1EncApp` and every pipeline thread it spawns are named via
`pthread_setname_np` regardless of the build flag. Worker arrays use the form
`svt-me0`...`svt-meN`, `svt-md0`...`svt-mdN`, `svt-cdef0`...`svt-cdefN`, etc.
Singleton stage threads take the kernel function name, truncated by Linux to
the 15-char `TASK_COMM_LEN`:

| Kernel | Thread name |
|---|---|
| `svt_aom_resource_coordination_kernel` | `resource_coordi` |
| `svt_aom_picture_decision_kernel`      | `picture_decisio` |
| `svt_aom_initial_rate_control_kernel`  | `initial_rate_co` |
| `svt_aom_picture_manager_kernel`       | `picture_manager` |
| `svt_aom_rate_control_kernel`          | `rate_control_ke` |
| `svt_aom_packetization_kernel`         | `packetization_k` |

The `svt_aom_` prefix is stripped at thread-creation time so each singleton
fits the 15-char `TASK_COMM_LEN` window distinctly. Without the strip,
`picture_decision_kernel` and `picture_manager_kernel` would both truncate to
`svt_aom_picture` and become indistinguishable in `/proc/<pid>/task/*/comm`
and the Nsight ThreadNames table. The application thread is `svt-app-main`.
All names are visible in `top`, `htop`, `ps -L -o pid,tid,comm`,
`/proc/<pid>/task/*/comm`, and the Nsight Systems timeline lane labels.

## Standalone runtime

Capture on a Grace host (or any aarch64/x86_64 Linux):

```sh
nsys profile -t osrt,nvtx \
    --sample=process-tree --cpuctxsw=process-tree \
    --output=/tmp/svt --force-overwrite=true \
    ./Bin/Release/SvtAv1EncApp -i in.y4m -b out.ivf --preset 8 --lp 16
```

Add `--cpu-core-events=cycles,instructions,cache-misses,branch-misses` for
Neoverse V2 PMU counters (requires root or `paranoid <= 0`).

Inspect:

```sh
nsys stats --report nvtx_sum   --format=csv --force-export=true /tmp/svt.nsys-rep
nsys stats --report osrt_sum   --format=csv /tmp/svt.nsys-rep
nsys-ui /tmp/svt.nsys-rep   # GUI; per-thread lanes labelled svt-me0, svt-cdef0, ...
```

## Harness runtime (`test/benchmarking/`)

The Python benchmark harness can wrap the encoder with `nsys profile` and add
parity metrics to the per-job CSV alongside BD-rate / VMAF / SSIMULACRA2.

In your config (e.g. `test/benchmarking/configs/test_video_config.yaml`):

```yaml
profiler:
  enabled: true                 # default false; existing configs unaffected
  command: "nsys profile -t osrt,nvtx -s cpu --cpuctxsw=process-tree --sample=process-tree --cpu-core-events=cycles,instructions,cache-misses,branch-misses --output={profile_path} --force-overwrite=true "
  apply_to: ["svtav1", "svtav1_rtc"]

paths:
  profile_dir: "{out_dir}/profiles/{tag}"

binaries:
  linux_aarch64:                # NEW slot: picked automatically on aarch64 Linux
    svtav1_enc: "{root_dir}/bin/linux_aarch64/SvtAv1EncApp"
    # ... mirror the linux_x86_64 entries you need
```

Run as usual:

```sh
./run_comparison.sh configs/test_video_config.yaml
```

The encoded CSV gains three columns next to `encode_time` / `output_size`:

| Column                    | Source                                                  |
| --                        | --                                                      |
| `nsys_report_path`        | path to the per-job `.nsys-rep` (open in `nsys-ui`)     |
| `osrt_total_ms`           | `SUM(end-start)` over `OSRT_API` rows in the SQLite     |
| `cpu_sampling_top1_func`  | top sampled symbol from `SAMPLING_CALLCHAINS` (empty when sampling did not capture — short run, paranoid restriction, etc.) |

## Grace / containers note

Nsight Systems' CPU sampling and PMU counters go through `perf_event_open`,
which respects `kernel.perf_event_paranoid`:

- `paranoid >= 3`: no sampling, no PMU. NVTX, OSRT, and context-switch trace
  still work.
- `paranoid == 2`: per-process sampling allowed. PMU still restricted.
- `paranoid <= 0`: full PMU access (or run nsys as root).

On a bare-metal Grace host you typically already have `paranoid <= 2`. In
shared containers the sysctl is often locked higher; the integration degrades
gracefully — `cpu_sampling_top1_func` ends up empty in the CSV but every other
column populates and the timeline still shows NVTX ranges, OSRT calls, and
thread states.

## Verifying the integration

A 30-frame smoke run is enough to verify the wiring:

```sh
nsys profile -t osrt,nvtx --output=/tmp/smoke --force-overwrite=true \
    ./Bin/Release/SvtAv1EncApp -i sample.y4m -b /tmp/smoke.ivf --preset 11 -n 30
nsys stats --report nvtx_sum --format=csv --force-export=true /tmp/smoke.nsys-rep
# expect a single ":wait" range with Instances >> frame count
```

For the `pthread_setname_np` side, while a longer encode is running:

```sh
ls /proc/$(pidof SvtAv1EncApp)/task/*/comm | xargs -I{} cat {} | sort -u
# expect svt-app-main, 6 distinct singletons (picture_decisio, picture_manager, initial_rate_co, rate_control_ke, packetization_k, resource_coordi), svt-me0..N, svt-md0..N, svt-cdef0..N, ...
```
