Issues running the images

[prabhuteja12]

Hello,

I'm trying to run some code on the docker images provided. For instance, i'm trying to do:

docker run -it 1.51it/s]jefzda/sweap-images:future-architect.vuls-future-architect__vuls-407407d306e9431d6aa0ab566baa6e44e5ba2904 /bin.bash

which ideally lets me get bash access. But it fails with

/bin/bash: /bin/bash: cannot execute binary file

This happens for quite a few images. I'm wondering if I'm doing something wrong here. Can you please help me with this?

Thank you!

[jeff-da]

Hi! I should clarify this in the README. But we run bash by default, so the command should be like:

docker run -it --rm jefzda/sweap-images:gravitational.teleport-gravitational__teleport-32bcd71591c234f0d8 b091ec01f1f5cbfdc0f13c-vee9b09fb20c43af7e520f57e9239bbcf46b7113

[prabhuteja12]

Hi @jeff-da ,

Thank you for the clarification. I'm trying to run this package with local docker (and not modal) and I can't seem to run anything successfully with golden patches too. Is there any chance you have some code for using local docker instead of modal?

[eliasyin]

try this:
docker run -it --rm --entrypoint sh jefzda/sweap-images:gravitational.teleport-gravitational__teleport-32bcd71591c234f0d8 b091ec01f1f5cbfdc0f13c-vee9b09fb20c43af7e520f57e9239bbcf46b7113 -c 'echo "hello world"'

[prabhuteja12]

Hi @eliasyin , that exact command doesn't seem to work

But the following works:

docker run -it --rm --entrypoint sh jefzda/sweap-images:gravitational.teleport-gravitational__teleport-32bcd71591c234f0d8b091ec01f1f5cbfdc0f13c-vee9b09fb20c43af7e520f57e9239bbcf46b7113

thank you! Is there any plan to release a local docker evaluation instead of using modal?

[jeff-da]

Hi, It's in the plans to work on a working Docker evaluation so folks can run it locally. Anyone can feel free to make a PR for this too

[prabhuteja12]

Hi @jeff-da ,

I have a starting script for eval_with_docker as follows:

def eval_with_docker(
    patch,
    sample,
    output_dir,
    dockerhub_username,
    scripts_dir,
    prefix="",
    redo=False,
    block_network=False,
):
    uid = sample["instance_id"]
    os.makedirs(os.path.join(output_dir, uid), exist_ok=True)

    output_path = os.path.join(output_dir, uid, f"{prefix}_output.json")
    if not redo and os.path.exists(output_path):
        print(f"Skipping {uid} - output already exists")
        with open(output_path, "r") as f:
            return json.load(f)

    print(f"Running evaluation for {uid}")
    docker_client = docker.from_env()

    # Load local scripts
    try:
        run_script = load_local_script(scripts_dir, uid, "run_script.sh")
        parser_script = load_local_script(scripts_dir, uid, "parser.py")
    except FileNotFoundError as e:
        print(f"Error loading scripts for {uid}: {e}")
        return None

    # Create workspace directory for this instance
    workspace_dir = os.path.join(output_dir, uid, "workspace")
    os.makedirs(workspace_dir, exist_ok=True)

    # Write all required files to workspace
    with open(os.path.join(workspace_dir, "patch.diff"), "w") as f:
        f.write(patch)
    with open(os.path.join(workspace_dir, "run_script.sh"), "w") as f:
        f.write(run_script)
    with open(os.path.join(workspace_dir, "parser.py"), "w") as f:
        f.write(parser_script)
    with open(os.path.join(workspace_dir, "entryscript.sh"), "w") as f:
        f.write(create_entryscript(sample))

    # Use Docker Hub image instead of building from scratch
    dockerhub_image_uri = get_dockerhub_image_uri(
        uid, dockerhub_username, sample.get("repo", "")
    )
    print(f"Using Docker Hub image: {dockerhub_image_uri}")

    # Pull the image with platform specification
    docker_client.images.pull(dockerhub_image_uri)

    network_mode = "none" if block_network else "bridge"

    container = docker_client.containers.run(
        dockerhub_image_uri,
        command=["/workspace/entryscript.sh"],
        volumes={os.path.abspath(workspace_dir): {"bind": "/workspace", "mode": "rw"}},
        cpu_quota=400000,
        mem_limit="30g",
        network_mode=network_mode,
        platform="linux/amd64",
        detach=True,
    )

    # Wait for container to finish
    result = container.wait()

    # Get logs
    logs = container.logs().decode()
    # print(f"Container logs for {uid}:")
    # print(logs)

    # Clean up container
    container.remove()

    # Read output files
    output_file = os.path.join(workspace_dir, "output.json")
    if os.path.exists(output_file):
        with open(output_file, "r") as f:
            output = json.load(f)
        # Copy to final location
        with open(output_path, "w") as f:
            json.dump(output, f)
    else:
        print(f"Warning: output.json not found for {uid}")
        return None

    # Copy log files to final location
    for log_file in ["stdout.log", "stderr.log", "entryscript.sh"]:
        src = os.path.join(workspace_dir, log_file)
        dst = os.path.join(output_dir, uid, f"{prefix}_{log_file}")
        if os.path.exists(src):
            with open(src, "r") as f_in, open(dst, "w") as f_out:
                f_out.write(f_in.read())

    return output

It doesn't yet fully work. I'm trying to use it with ground truth predictions, but the accuracy is arond 44%. Can you confirm that with modal you get 100% pass rate with the publicly available code, images and data?

[jeff-da]

Thank you! We gave it a shot on our side too and updated the script to support Docker. Feel free to give the Docker version a shot, I'll close this for now.