software-metrics

most profilers require frame pointers, almost all require debug information(can store seperately)
sudo scx_lavd --performance for latency aware scheduler (gpu/drm is wip)
monitoring
- netdata
- prometheus-tech
profilers
- https://github.com/intel/oneapi-containers/tree/master/images/docker/vtune
- async profiler
  - native example LD_PRELOAD=/libasyncProfiler.so ASPROF_COMMAND=start,nativemem,total,event=cpu,wall,loop=10m,cstack=dwarf,file=profile-%t.jfr ./ree
perforator
- record flamegraph and compare (ebpf collection)
trace-cmd

trace-cmd record -i \
    -e "sched:sched_switch" \
    -e "sched:sched_process_exec" \
    -e "sched:sched_process_fork" \
    -e "sched:sched_process_exit" \
    -e "amdgpu:amdgpu_vm_flush" \
    -e "amdgpu:amdgpu_cs_ioctl" \
    -e "amdgpu:amdgpu_sched_run_job" \
    -e "*fence:*fence_signaled" \
    -e "drm:drm_vblank_event" \
    -e "drm:drm_vblank_event_queued" \
    -e "i915:i915_flip_request" \
    -e "i915:i915_flip_complete" \
    -e "i915:intel_gpu_freq_change" \
    -e "i915:i915_gem_request_add" \
    -e "i915:i915_gem_request_submit" \
    -e "i915:i915_gem_request_in" \
    -e "i915:i915_gem_request_out" \
    -e "i915:intel_engine_notify" \
    -e "i915:i915_gem_request_wait_begin" \
    -e "i915:i915_gem_request_wait_end" \
    sleep 4

pyroscope
- aggregates profiling data for flamegraphs
- compare flamegraphs with grafana and alloy(ebpf collection)
  - grafana config: https://github.com/grafana/pyroscope/tree/main/examples/grafana-alloy-auto-instrumentation/ebpf
    - rename urls for localhost and remove alloy as root dns on rootless podman was not resolving…
  - sudo ./alloy-linux-amd64 run /gnu/git/alloy.config --storage.path=/gnu/git/alloy

discovery.process "all" {}

discovery.relabel "btm" {
    targets = discovery.process.all.targets
    // Filter needed processes
    rule {
        source_labels = ["__meta_process_exe"]
        regex = ".*/btm"
        action = "keep"
    }
     // provide arbitrary service_name label, otherwise it will be "unspecified"
    rule {
        source_labels = ["__meta_process_exe"]
        target_label = "service_name"
        regex = ".*/btm"
        action = "replace"
        replacement = "bottom"
    }
}
pyroscope.ebpf "instance" {
    demangle = "full"
    off_cpu_threshold = 1
    forward_to = [pyroscope.write.endpoint.receiver]
    targets = discovery.relabel.btm.output
}

pyroscope.write "endpoint" {
    endpoint {
        url = "http://localhost:4040"
    }
}

Metrics

notes