bpftool no instructions returned

[vincentmli]

Hi @qmonnet

I am trying to run bpftool to dump jited bpf program instructions according to
loxilb-io/loxilb#972 (comment), but I came across a strange problem that bpftool returns no instructions returned. the bpftool is built from kernel source after kernel is built, linux from scratch lfs build script for kernel and bpftool
https://github.com/vincentmli/BPFire/blob/bpfire/lfs/linux

	# Build bpftool
	cd $(DIR_APP)/tools/bpf/bpftool && sed -i -e 's/^prefix ?= \/usr\/local/prefix ?= \/usr/' Makefile
	cd $(DIR_APP)/tools/bpf/bpftool && make $(MAKETUNING)

part of build log

cd /usr/src/linux-6.14-rc5/tools/bpf/bpftool && sed -i -e 's/^prefix ?= \/usr\/local/prefix ?= \/usr/' Makefile
cd /usr/src/linux-6.14-rc5/tools/bpf/bpftool && make -j8
make[1]: Entering directory '/usr/src/linux-6.14-rc5/tools/bpf/bpftool'

Auto-detecting system features:
...                         clang-bpf-co-re: [ ^[[32mon^[[m  ]
...                                    llvm: [ ^[[32mon^[[m  ]
...                                  libcap: [ ^[[32mon^[[m  ]
...                                  libbfd: [ ^[[32mon^[[m  ]
...

  CLANG   pid_iter.bpf.o
  CLANG   profiler.bpf.o
  GEN     profiler.skel.h
  CC      prog.o
  GEN     pid_iter.skel.h
  CC      pids.o
  LINK    bpftool

bpftool version:

[root@loongfire ~]# bpftool -V
bpftool v7.6.0
using libbpf v1.6
features: llvm, skeletons

clang is version 18.1.0

background:

I was asked to dump jited loxilb bpf progrom instruction on loongfire OS which is loongarch CPU architecture port of open source firewall IPFire. https://lore.kernel.org/bpf/CAK3+h2woEjG_N=-XzqEGaAeCmgu2eTCUc7p6bP4u8Q+DFHm-7g@mail.gmail.com/T/#m5ca0884901df7512f20f0ca5b3d73ef89558471c. but then I found out that bpftool I built in IPFire can't dump any jited instruction for any bpf program, so I suspect something I might be missing when build bpftool from kernel source. this same issue happens to me for both loongarch and x86 architecture, so should be irrelevant to CPU architecture.

[vincentmli]

here the bpftool shows the prog is jited, but could not dump the jited instruction

[root@loongfire ~]# bpftool prog show id 257
257: sched_cls  name tc_packet_func  tag 789f4fc9a2453f7d  gpl
	loaded_at 2025-03-09T08:55:01-0700  uid 0
	xlated 66384B  jited 42944B  memlock 81920B  map_ids 442,451,450,468,444,443,446,454,456,457,483,469,470,459,462,474,460,464,465,466,467,458,453,455,477,471,476,485
	btf_id 309

[root@loongfire ~]# bpftool prog dump jited id 257
no instructions returned

[vincentmli]

part of strace output:

10851 prctl(PR_CAPBSET_READ, CAP_MAC_OVERRIDE) = 1
10851 prctl(PR_CAPBSET_READ, 0x30 /* CAP_??? */) = -1 EINVAL (Invalid argument)
10851 prctl(PR_CAPBSET_READ, CAP_CHECKPOINT_RESTORE) = 1
10851 prctl(PR_CAPBSET_READ, 0x2c /* CAP_??? */) = -1 EINVAL (Invalid argument)
10851 prctl(PR_CAPBSET_READ, 0x2a /* CAP_??? */) = -1 EINVAL (Invalid argument)
10851 prctl(PR_CAPBSET_READ, 0x29 /* CAP_??? */) = -1 EINVAL (Invalid argument)
10851 lseek(2, 0, SEEK_CUR)             = -1 ESPIPE (Illegal seek)
10851 newfstatat(2, "", {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x2), ...}, AT_EMPTY_PATH) = 0
10851 bpf(BPF_PROG_GET_FD_BY_ID, {prog_id=154, next_id=0, open_flags=0}, 12) = 3
10851 bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=3, info_len=232, info=0x7ffec6fa7cc0}}, 16) = 0
10851 brk(0xdf87000)                    = 0xdf87000
10851 bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=3, info_len=232, info=0x7ffec6fa7cc0}}, 16) = 0
10851 write(2, "no instructions returned", 24) = 24

[qmonnet]

Hey Vincent!

I've never experimented with loongarch, but am I understanding correctly that you also fail to dump JIT-ed programs on x86? This is definitely not expected. I've never got reports of that so far.

Do you cross-compile bpftool, or do you compile it on the system where you try to run it?

I don't know what could cause the kernel to fail return the instructions. I see that you get:

10851 bpf(BPF_OBJ_GET_INFO_BY_FD, {info={bpf_fd=3, info_len=232, info=0x7ffec6fa7cc0}}, 16) = 0

The syscall succeeds; it would be interesting to debug bpftool to look at what info contains after this step.

Looking at kernel code from kernel/bpf/syscall.c, one potential candidate would be if bpf_dump_raw(file->f_cred) fails, in which case it seems we skip copying the JIT-ed instructions and set the length to 0. Looking at the relevant commit, this seems to happen if the process doesn't have CAP_SYSLOG? Probably something to check on your setup? (I haven't looked in details and may be wrong). If you're heading down the rabbit hole, retsnoop could help looking at what happens on kernel side when you try to dump the instructions, or maybe just bpftrace to check the return value from bpf_dump_raw(), as a starter.

[qmonnet]

Side note, looking at:

cd $(DIR_APP)/tools/bpf/bpftool && sed -i -e 's/^prefix ?= \/usr\/local/prefix ?= \/usr/' Makefile
cd $(DIR_APP)/tools/bpf/bpftool && make $(MAKETUNING)

You should be able to get rid of the sed command and set the prefix when calling make, something like prefix=/usr/ make $(MAKETUNING). The ?= in the affectation means we assign a default value is none is set from command line.

[vincentmli]

Here is information about IPFire build system https://www.ipfire.org/docs/devel/ipfire-2-x/build-howto which build the IPFire OS for everything including kernel from source code in a chroot environment with its own toolchain. I have IPFire OS build for loongarch CPU and x86 CPU, both bpftool has the same problem. I recall somewhere mentioned in IPFire the best practice is to set the prefix to /usr, not default /usr/local.

[vincentmli]

it turned out I need to have setting below, thanks a lot for your tip. now I recall I need following setting to run linux perf tool properly too, sigh. I should keep following setting as default to save me headache :)

    echo -1 > /proc/sys/kernel/perf_event_paranoid
    echo 0 > /proc/sys/kernel/kptr_restrict

[vincentmli]

do you think if it is going to be helpful for bpftool to print out some hint info for such case? I recall perf tool would print out something to remind user to check the kernel setting.

[qmonnet]

Glad you fixed it, and thanks for reporting back! 🎉

    echo -1 > /proc/sys/kernel/perf_event_paranoid
    echo 0 > /proc/sys/kernel/kptr_restrict

Do you know how this fixed the issue, exactly? Did you need to change both parameters, or just one of these to have it work?

do you think if it is going to be helpful for bpftool to print out some hint info for such case? I recall perf tool would print out something to remind user to check the kernel setting.

Perf does indeed a great job at accompanying the user when they hit an error. I'm not opposed to adding something to bpftool, but I'd like to have a good understanding of what caused the issue and what fixed it before we offer a hint. Would you be able to explain? Otherwise I'll try to dive in the related code at some point, to understand.

[vincentmli]

it is your clue to check kernel/bpf/syscall.c, and bpf_dump_raw(file->f_cred).

static int bpf_prog_get_info_by_fd(struct file *file,
                                   struct bpf_prog *prog,
                                   const union bpf_attr *attr,
                                   union bpf_attr __user *uattr)
{
...
        if (info.jited_prog_len && ulen) {
                if (bpf_dump_raw_ok(file->f_cred)) { <================
                        uinsns = u64_to_user_ptr(info.jited_prog_insns);
                        ulen = min_t(u32, info.jited_prog_len, ulen);

                        /* for multi-function programs, copy the JITed
                         * instructions for all the functions
                         */
                        if (prog->aux->func_cnt) {
                                u32 len, free, i;
                                u8 *img;

                                free = ulen;
                                for (i = 0; i < prog->aux->func_cnt; i++) {
                                        len = prog->aux->func[i]->jited_len;
                                        len = min_t(u32, len, free);
                                        img = (u8 *) prog->aux->func[i]->bpf_func;
                                        if (copy_to_user(uinsns, img, len))
                                                return -EFAULT;
                                        uinsns += len;
                                        free -= len;
                                        if (!free)
                                                break;
                                }
                        } else {
                                if (copy_to_user(uinsns, prog->bpf_func, ulen))
                                        return -EFAULT;
                        }
                } else {
                        info.jited_prog_insns = 0;
                }
        }


static inline bool bpf_dump_raw_ok(const struct cred *cred)
{                   
        /* Reconstruction of call-sites is dependent on kallsyms,
         * thus make dump the same restriction. 
         */             
        return kallsyms_show_value(cred);
}                       
                
/*
 * We show kallsyms information even to normal users if we've enabled
 * kernel profiling and are explicitly not paranoid (so kptr_restrict
 * is clear, and sysctl_perf_event_paranoid isn't set).
 *                  
 * Otherwise, require CAP_SYSLOG (assuming kptr_restrict isn't set to
 * block even that).
 */
bool kallsyms_show_value(const struct cred *cred)
{                       
        switch (kptr_restrict) {
        case 0: 
                if (kallsyms_for_perf())
                        return true; 
                fallthrough;
        case 1:
                if (security_capable(cred, &init_user_ns, CAP_SYSLOG,
                                     CAP_OPT_NOAUDIT) == 0)
                        return true;
                fallthrough;
        default:
                return false;
        }
}                       

the loongfire/bpfire kernel kptr_restrict inherited from IPFire has default value set to 2 to restrict kernel pointer access, the perf_event_paranoid is set to 3

[root@loongfire ~]# cat /proc/sys/kernel/kptr_restrict 
2

so set /proc/sys/kernel/kptr_restrict to either 0 or 1 alone resolves this issue as I just tested, no need to change setting perf_event_paranoid

[qmonnet]

Thanks! 🙏 OK then yes, we could add a small hint about kptr_restrict and/or CAP_SYSLOG if you want.