[Bug]: Incorrect GPU name displayed in Hotaisle VM Fleet Resources

[Bihan]

Steps to reproduce

1. Create Hotaisle Fleet with following config

type: fleet
name: hotaisle-fleet-vm

placement: any

ssh_config:
  user: hotaisle
  identity_file: ~/.ssh/id_rsa
  

  hosts:
    - 23.183.40.86

2. Once the fleet is created check the fleet with below command:
   dstack fleet

Actual behaviour

FLEET              INSTANCE  BACKEND       RESOURCES                                                  PRICE  STATUS  CREATED     
 hotaisle-fleet-vm  0         ssh (remote)  cpu=13 mem=220GB disk=11451GB AMD Radeon Graphics:192GB:1  $0     idle    28 mins ago 

Expected behaviour

FLEET              INSTANCE  BACKEND       RESOURCES                                                  PRICE  STATUS  CREATED     
 hotaisle-fleet-vm  0         ssh (remote)  cpu=13 mem=220GB disk=11451GB MI300X:192GB:1  $0     idle    28 mins ago 

dstack version

0.19.18

Server logs

...
...
[18:09:01] DEBUG    dstack._internal.server.background.tasks.process_instances:450 Received a host_info {'gpu_vendor': 'amd',          
                    'gpu_name': 'AMD Radeon Graphics', 'gpu_memory': 196288, 'gpu_count': 1, 'addresses': ['23.183.40.86/26',          
                    'fe80::5054:ff:fe54:bfcd/64', '172.17.0.1/16', 'fe80::1c6c:22ff:fee5:2656/64'], 'disk_size': 12295348760576,       
                    'cpus': 13, 'memory': 236435066880}                                                                                
[18:09:02] INFO     dstack._internal.server.background.tasks.process_instances:286 The instance hotaisle-fleet-vm-0 (23.183.40.86) was 
                    successfully added

Additional information

When I run rocm-smi --showproductname in the VM it shows Mi300x

============================ ROCm System Management Interface ============================
====================================== Product Info ======================================
GPU[0]		: Card Series: 		AMD Instinct MI300X VF
GPU[0]		: Card Model: 		0x74b5
GPU[0]		: Card Vendor: 		Advanced Micro Devices, Inc. [AMD/ATI]
GPU[0]		: Card SKU: 		M3000100
GPU[0]		: Subsystem ID: 	0x74a1
GPU[0]		: Device Rev: 		0x00
GPU[0]		: Node ID: 		1
GPU[0]		: GUID: 		22361
GPU[0]		: GFX Version: 		gfx942
==========================================================================================
================================== End of ROCm SMI Log ===================================

[Bihan]

@un-def I think the issue is from Hotaisle side because output from amd-smi is as below.

docker run --rm \
  --device /dev/kfd \
  --device /dev/dri \
  dstackai/amd-smi:latest \
  static --json --asic --vram --bus
[
    {
        "gpu": 0,
        "asic": {
            "market_name": "AMD Radeon Graphics",
            "vendor_id": "0x1002",
            "vendor_name": "Advanced Micro Devices Inc. [AMD/ATI]",
            "subvendor_id": "0x1002",
            "device_id": "0x74b5",
            "subsystem_id": "0x74a1",
            "rev_id": "0x00",
            "asic_serial": "0x8002DDCD1732FDAB",
            "oam_id": 3,
            "num_compute_units": 304,
            "target_graphics_version": "gfx942"
        },
        "bus": {
            "bdf": "0000:05:00.0",
            "max_pcie_width": 16,
            "max_pcie_speed": {
                "value": 32,
                "unit": "GT/s"
            },
            "pcie_interface_version": "Gen 5",
            "slot_type": "OAM"
        },
        "vram": {
            "type": "HBM",
            "vendor": "UNKNOWN",
            "size": {
                "value": 196288,
                "unit": "MB"
            },
            "bit_width": 8192,
            "max_bandwidth": {
                "value": 5325,
                "unit": "GB/s"
            }
        }
    }
]

[un-def]

@Bihan we expect [].asic.market_name to be "MI300X[-O]", but for some reason it's "AMD Radeon Graphics" (seems like a generic/fallback value, as MI cards are not even in Radeon family, it's Instinct family)

This is a diff of the amd-smi output we used to add MI300X support:

         "asic": {
-            "market_name": "MI300X-O",
+            "market_name": "AMD Radeon Graphics",
             "vendor_id": "0x1002",
             "vendor_name": "Advanced Micro Devices Inc. [AMD/ATI]",
             "subvendor_id": "0x1002",
-            "device_id": "0x74a1",
+            "device_id": "0x74b5",
+            "subsystem_id": "0x74a1",
             "rev_id": "0x00",
-            "asic_serial": "0xE237D890B3EEF476",
-            "oam_id": 1
+            "asic_serial": "0x8002DDCD1732FDAB",
+            "oam_id": 3,
+            "num_compute_units": 304,
+            "target_graphics_version": "gfx942"
         }

[peterschmidt85]

@Bihan I guess this issue only occurs when running amd-smi via docker. Can you confirm that?

I've just reproduced with a DigitalOcean's VM too.

When running via a container, I get the response similar to yours above.

However, when I run on the command host, I get the following:

amd-smi static --asic --vram
GPU: 0
    ASIC:
        MARKET_NAME: AMD Instinct MI300X VF
        VENDOR_ID: 0x1002
        VENDOR_NAME: Advanced Micro Devices Inc. [AMD/ATI]
        SUBVENDOR_ID: 0x1002
        DEVICE_ID: 0x74b5
        SUBSYSTEM_ID: 0x74a1
        REV_ID: 0x00
        ASIC_SERIAL: 0xD497EF8CEB9BDD7E
        OAM_ID: 4
        NUM_COMPUTE_UNITS: 304
        TARGET_GRAPHICS_VERSION: gfx942
    VRAM:
        TYPE: HBM
        VENDOR: UNKNOWN
        SIZE: 196288 MB
        BIT_WIDTH: 8192
        MAX_BANDWIDTH: 5325 GB/s

[Bihan]

Currently dstack would report GPU incorrectly for VM SSH fleet with AMD MI - because either AMD MI VMs needs rocm installation to report correct GPU. (Note: Currently our amd-smi container does not have rocm installed) or we have old amd-smi version. After investigating we need to fix our amd-smi image.

To verify whether amd-smi reports correct GPU after ROCM installation use these steps:

Step1: Run Container
docker run --rm -it
--device=/dev/kfd
--device=/dev/dri
--entrypoint /bin/bash
dstackai/amd-smi:latest

Step2: Install ROCM
wget https://repo.radeon.com/amdgpu-install/6.4.2/ubuntu/noble/amdgpu-install_6.4.60402-1_all.deb
sudo apt install ./amdgpu-install_6.4.60402-1_all.deb
sudo apt update
sudo apt install python3-setuptools python3-wheel
sudo apt install rocm

Step3: amd-smi static --json --asic --vram --bus

[
    {
        "gpu": 0,
        "asic": {
            "market_name": "AMD Instinct MI300X VF",
            "vendor_id": "0x1002",
            "vendor_name": "Advanced Micro Devices Inc. [AMD/ATI]",
            "subvendor_id": "0x1002",
            "device_id": "0x74b5",
            "subsystem_id": "0x74a1",
            "rev_id": "0x00",
            "asic_serial": "0x601813B29D54B540",
            "oam_id": 7,
            "num_compute_units": 304,
            "target_graphics_version": "gfx942"
        },
        "bus": {
            "bdf": "0000:05:00.0",
            "max_pcie_width": 16,
            "max_pcie_speed": {
                "value": 32,
                "unit": "GT/s"
            },
            "pcie_interface_version": "Gen 5",
            "slot_type": "OAM"
        },
        "vram": {
            "type": "HBM",
            "vendor": "UNKNOWN",
            "size": {
                "value": 196288,
                "unit": "MB"
            },
            "bit_width": 8192,
            "max_bandwidth": {
                "value": 5325,
                "unit": "GB/s"
            }
        }
    }
]

Additional Info
amd-smi version after ROCM installation
AMDSMI Tool: 25.5.1

**Current amd-smi version"
AMDSMI Tool: 25.3.0

[Bihan]

Updating amd-smi's ROCm Version to 6.4.2 resolved the issue.