Skip to content

[Feature]: Support for Tenstorrent Galaxy #2898

@manproai

Description

@manproai

Problem

I tested dstack on the Tenstorrent Galaxy Wormhole server, which has 32 N300 cards, but it doesn't appear to be supported yet.

This is what I got when I created a fleet:

Project        main                   
User           admin                  
Configuration  galaxyfleet.dstack.yml 
Type           fleet                  
Fleet type     ssh                    
Nodes          1                      
Placement      any                    

Fleet galaxy-fleet does not exist yet.
Create the fleet? [y/n]: y
FLEET         INSTANCE  BACKEND       RESOURCES                           PRICE  STATUS  CREATED 
galaxy-fleet  0         ssh (remote)  cpu=64 mem=566GB disk=2980GB N/A:1  $0     idle    02:44   

When I implemented the service with n300:1 specified, I got this No offers error.
When I did not specifed the resources at all, I got this error:

Project main
User admin
Configuration demo-tt-infer-ser-galaxy.dstack.yml
Type service
Resources cpu=2.. mem=8GB.. disk=100GB..
Spot policy on-demand
Max price -
Retry policy -
Creation policy reuse-or-create
Idle duration 5m
Max duration -
Reservation -

 #  BACKEND       RESOURCES                           INSTANCE TYPE  PRICE       
 1  ssh (remote)  cpu=64 mem=566GB disk=2980GB N/A:1  instance       $0     idle 

Submit the run galaxy-tt-inference-server? [y/n]: y
 NAME                                     BACKEND       RESOURCES                           INSTANCE TYPE  PRICE  STATUS  SUBMITTED  ERROR        
 galaxy-tt-inference-server deployment=0                                                                          error   02:52                   
   replica=0 job=0 deployment=0           ssh (remote)  cpu=64 mem=566GB disk=2980GB N/A:1  instance       $0     error   02:52      runner error 

galaxy-tt-inference-server provisioning completed (failed)
Error (Runner error)
pullImage error: failed pulling ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-20.04-amd64:0.0.4-v0.56.0-rc47-e2e0002ac7dc: 
downloaded 4765400833/9559153870 bytes (3.79MB/s), errors: []

These are some of the error from /root/.dstack/shim.log

time=2025-07-10T17:17:47.707232Z level=debug msg=downloading runner url=https://dstack-runner-downloads.s3.eu-west-1.amazonaws.com/0.19.18/binaries/dstack-runner-linux-amd64
time=2025-07-10T17:17:53.610624Z level=info msg=the runner was downloaded successfully bytes=19499902
time=2025-07-10T17:17:53.610721Z level=debug msg=Shim args={10998 /root/.dstack 6}
time=2025-07-10T17:17:53.610743Z level=debug msg=Runner args={10999 10022 https://dstack-runner-downloads.s3.eu-west-1.amazonaws.com/0.19.18/binaries/dstack-runner-linux-amd64 /usr/local/bin/dstack-runner 6}
time=2025-07-10T17:17:53.610753Z level=debug msg=Docker args={... main@dstack false }
time=2025-07-10T17:20:19.508439Z level=debug method=GET endpoint=/api/healthcheck status=200
time=2025-07-10T17:45:01.19057Z level=debug msg=downloading runner url=https://dstack-runner-downloads.s3.eu-west-1.amazonaws.com/0.19.18/binaries/dstack-runner-linux-amd64
time=2025-07-10T17:47:20.387309Z level=info msg=the runner was downloaded successfully bytes=19499902
time=2025-07-10T17:47:20.387402Z level=debug msg=Shim args={10998 /root/.dstack 6}
time=2025-07-10T17:47:20.387417Z level=debug msg=Runner args={10999 10022 https://dstack-runner-downloads.s3.eu-west-1.amazonaws.com/0.19.18/binaries/dstack-runner-linux-amd64 /usr/local/bin/dstack-runner 6}
time=2025-07-10T17:47:20.387427Z level=debug msg=Docker args={... main@dstack false }
time=2025-07-10T17:47:25.414113Z level=debug endpoint=/api/healthcheck status=200 method=GET

......

time=2025-07-10T18:12:49.795965Z level=debug endpoint=/api/tasks/b2d58713-2f46-4953-976c-3808d08b8eb8 status=200 method=GET
time=2025-07-10T18:12:50.558725Z level=debug method=GET endpoint=/api/healthcheck status=200
time=2025-07-10T18:13:04.162759Z level=error msg=pullImage error: failed pulling ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-20.04-amd64:0.0.4-v0.56.0-rc47-e2e0002ac7dc: downloaded 4765400833/9559153870 bytes (3.79MB/s), errors: []
time=2025-07-10T18:13:04.163499Z level=debug msg=released GPU(s) task=b2d58713-2f46-4953-976c-3808d08b8eb8 gpus=[0]
time=2025-07-10T18:13:04.163533Z level=error msg=failed to run err=failed pulling ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-20.04-amd64:0.0.4-v0.56.0-rc47-e2e0002ac7dc: downloaded 4765400833/9559153870 bytes (3.79MB/s), errors: [] task=b2d58713-2f46-4953-976c-3808d08b8eb8
time=2025-07-10T18:13:05.100237Z level=debug method=GET endpoint=/api/healthcheck status=200

Solution

Here is the snapshot of the tt-smi for the Galaxy Wormhole. Please use this as a reference for supporting the device.


ls /dev/tenstorrent
0  1  10  11  12  13  14  15  16  17  18  19  2  20  21  22  23  24  25  26  27  28  29  3  30  31  4  5  6  7  8  9

Workaround

Here is the half of tt-smi -s becuase it crossed characters limit

{
    "time": "...",
    "host_info": {
        "OS": "Linux",
        "Distro": "Ubuntu 22.04.5 LTS",
        "Kernel": "5.15.0-143-generic",
        "Hostname": "...",
        "Platform": "x86_64",
        "Python": "3.10.12",
        "Memory": "566.12 GB",
        "Driver": "TT-KMD 2.0.0"
    },
    "host_sw_vers": {
        "tt_smi": "3.0.23",
        "pyluwen": "0.7.7"
    },
    "device_info": [
        {
            "smbus_telem": {
                "BOARD_ID": "0x100035100000000",
                "ENUM_VERSION": "0xba5e0001",
                "DEVICE_ID": "0x401e1e52",
                "ASIC_RO": "0x2bc72",
                "ASIC_IDD": "0xca5",
                "BOARD_ID_HIGH": "0x1000351",
                "BOARD_ID_LOW": null,
                "ARC0_FW_VERSION": "0x2220000",
                "ARC1_FW_VERSION": "0x2220000",
                "ARC2_FW_VERSION": null,
                "ARC3_FW_VERSION": "0x2220000",
                "SPIBOOTROM_FW_VERSION": "0x30d0000",
                "ETH_FW_VERSION": "0x70000",
                "M3_BL_FW_VERSION": "0x81020000",
                "M3_APP_FW_VERSION": "0x5040000",
                "DDR_SPEED": null,
                "DDR_STATUS": "0x1222222",
                "ETH_STATUS0": "0x22222222",
                "ETH_STATUS1": "0x22222222",
                "PCIE_STATUS": "0x10140000",
                "FAULTS": null,
                "ARC0_HEALTH": "0x7acf627",
                "ARC1_HEALTH": "0x1d3bc",
                "ARC2_HEALTH": null,
                "ARC3_HEALTH": "0x4ca20",
                "FAN_SPEED": "0xffffffff",
                "AICLK": "0x3e803e8",
                "AXICLK": "0x384",
                "ARCCLK": "0x21c",
                "THROTTLER": null,
                "VCORE": "0x3a7",
                "ASIC_TEMPERATURE": "0x1d901c6",
                "VREG_TEMPERATURE": null,
                "BOARD_TEMPERATURE": "0x212f2d",
                "TDP": "0xaa0021",
                "TDC": "0xf00023",
                "VDD_LIMITS": "0x3e80320",
                "THM_LIMITS": "0x53004b",
                "WH_FW_DATE": "0x5616150e",
                "ASIC_TMON0": "0x1c1c1e1c",
                "ASIC_TMON1": "0x1c1d",
                "MVDDQ_POWER": "0x190000",
                "GDDR_TRAIN_TEMP0": null,
                "GDDR_TRAIN_TEMP1": null,
                "BOOT_DATE": "0xda7e",
                "RT_SECONDS": "0x7c92",
                "AUX_STATUS": null,
                "ETH_DEBUG_STATUS0": null,
                "ETH_DEBUG_STATUS1": null,
                "TT_FLASH_VERSION": "0x30305",
                "FW_BUNDLE_VERSION": "0x12060000",
                "TIMER_HEARTBEAT": "0x7acf627"
            },
            "board_info": {
                "bus_id": "0000:c3:00.0",
                "board_type": "tt-galaxy-wh L",
                "board_id": "100035100000000",
                "coords": "(0, 0, 0, 0)",
                "dram_status": true,
                "dram_speed": "14G",
                "pcie_speed": 4,
                "pcie_width": "1"
            },
            "telemetry": {
                "voltage": "0.94",
                "current": " 35.0",
                "power": " 33.0",
                "aiclk": "1000",
                "asic_temperature": "28.4",
                "heartbeat": "62777"
            },
            "firmwares": {
                "fw_bundle_version": "18.6.0.0",
                "tt_flash_version": "0.3.3.5",
                "cm_fw": "2.34.0.0",
                "cm_fw_date": "2025-06-22",
                "eth_fw": "7.0.0",
                "bm_bl_fw": "129.2.0.0",
                "bm_app_fw": "5.4.0.0"
            },
            "limits": {
                "vdd_min": "0.80",
                "vdd_max": "1.00",
                "tdp_limit": "170",
                "tdc_limit": "240",
                "asic_fmax": "1000",
                "therm_trip_l1_limit": "83",
                "thm_limit": "75",
                "bus_peak_limit": null
            }
        },
        {
            "smbus_telem": {
                "BOARD_ID": "0x100035100000000",
                "ENUM_VERSION": "0xba5e0001",
                "DEVICE_ID": "0x401e1e52",
                "ASIC_RO": "0x2dcce",
                "ASIC_IDD": "0xd5c",
                "BOARD_ID_HIGH": "0x1000351",
                "BOARD_ID_LOW": null,
                "ARC0_FW_VERSION": "0x2220000",
                "ARC1_FW_VERSION": "0x2220000",
                "ARC2_FW_VERSION": null,
                "ARC3_FW_VERSION": "0x2220000",
                "SPIBOOTROM_FW_VERSION": "0x30d0000",
                "ETH_FW_VERSION": "0x70000",
                "M3_BL_FW_VERSION": "0x81020000",
                "M3_APP_FW_VERSION": "0x5040000",
                "DDR_SPEED": null,
                "DDR_STATUS": "0x1222222",
                "ETH_STATUS0": "0x22222222",
                "ETH_STATUS1": "0x22222222",
                "PCIE_STATUS": "0x10140000",
                "FAULTS": null,
                "ARC0_HEALTH": "0x7acf5eb",
                "ARC1_HEALTH": "0x1d3bc",
                "ARC2_HEALTH": null,
                "ARC3_HEALTH": "0x4ca21",
                "FAN_SPEED": "0xffffffff",
                "AICLK": "0x3e803e8",
                "AXICLK": "0x384",
                "ARCCLK": "0x21c",
                "THROTTLER": null,
                "VCORE": "0x389",
                "ASIC_TEMPERATURE": "0x1d901c6",
                "VREG_TEMPERATURE": null,
                "BOARD_TEMPERATURE": "0x212f2d",
                "TDP": "0xaa001e",
                "TDC": "0xf00021",
                "VDD_LIMITS": "0x3e80320",
                "THM_LIMITS": "0x53004b",
                "WH_FW_DATE": "0x5616150e",
                "ASIC_TMON0": "0x1d1d1c1b",
                "ASIC_TMON1": "0x1d1d",
                "MVDDQ_POWER": "0x190000",
                "GDDR_TRAIN_TEMP0": null,
                "GDDR_TRAIN_TEMP1": null,
                "BOOT_DATE": "0xda7e",
                "RT_SECONDS": "0x7c92",
                "AUX_STATUS": null,
                "ETH_DEBUG_STATUS0": null,
                "ETH_DEBUG_STATUS1": null,
                "TT_FLASH_VERSION": "0x30305",
                "FW_BUNDLE_VERSION": "0x12060000",
                "TIMER_HEARTBEAT": "0x7acf5eb"
            },
            "board_info": {
                "bus_id": "0000:c4:00.0",
                "board_type": "tt-galaxy-wh L",
                "board_id": "100035100000000",
                "coords": "(0, 0, 0, 0)",
                "dram_status": true,
                "dram_speed": "14G",
                "pcie_speed": 4,
                "pcie_width": "1"
            },
            "telemetry": {
                "voltage": "0.91",
                "current": " 33.0",
                "power": " 30.0",
                "aiclk": "1000",
                "asic_temperature": "28.4",
                "heartbeat": "62777"
            },
            "firmwares": {
                "fw_bundle_version": "18.6.0.0",
                "tt_flash_version": "0.3.3.5",
                "cm_fw": "2.34.0.0",
                "cm_fw_date": "2025-06-22",
                "eth_fw": "7.0.0",
                "bm_bl_fw": "129.2.0.0",
                "bm_app_fw": "5.4.0.0"
            },
            "limits": {
                "vdd_min": "0.80",
                "vdd_max": "1.00",
                "tdp_limit": "170",
                "tdc_limit": "240",
                "asic_fmax": "1000",
                "therm_trip_l1_limit": "83",
                "thm_limit": "75",
                "bus_peak_limit": null
            }
        },
        {
            "smbus_telem": {
                "BOARD_ID": "0x100035100000000",
                "ENUM_VERSION": "0xba5e0001",
                "DEVICE_ID": "0x401e1e52",
                "ASIC_RO": "0x2caa6",
                "ASIC_IDD": "0x9d9",
                "BOARD_ID_HIGH": "0x1000351",
                "BOARD_ID_LOW": null,
                "ARC0_FW_VERSION": "0x2220000",
                "ARC1_FW_VERSION": "0x2220000",
                "ARC2_FW_VERSION": null,
                "ARC3_FW_VERSION": "0x2220000",
                "SPIBOOTROM_FW_VERSION": "0x30d0000",
                "ETH_FW_VERSION": "0x70000",
                "M3_BL_FW_VERSION": "0x81020000",
                "M3_APP_FW_VERSION": "0x5040000",
                "DDR_SPEED": null,
                "DDR_STATUS": "0x1222222",
                "ETH_STATUS0": "0x22222222",
                "ETH_STATUS1": "0x22222222",
                "PCIE_STATUS": "0x10140000",
                "FAULTS": null,
                "ARC0_HEALTH": "0x7acf251",
                "ARC1_HEALTH": "0x1d3bc",
                "ARC2_HEALTH": null,
                "ARC3_HEALTH": "0x4ca21",
                "FAN_SPEED": "0xffffffff",
                "AICLK": "0x3e803e8",
                "AXICLK": "0x384",
                "ARCCLK": "0x21c",
                "THROTTLER": null,
                "VCORE": "0x39d",
                "ASIC_TEMPERATURE": "0x1d401c2",
                "VREG_TEMPERATURE": null,
                "BOARD_TEMPERATURE": "0x212f2d",
                "TDP": "0xaa001f",
                "TDC": "0xf00022",
                "VDD_LIMITS": "0x3e80320",
                "THM_LIMITS": "0x53004b",
                "WH_FW_DATE": "0x5616150e",
                "ASIC_TMON0": "0x1b1d1d1b",
                "ASIC_TMON1": "0x1d1d",
                "MVDDQ_POWER": "0x190000",
                "GDDR_TRAIN_TEMP0": null,
                "GDDR_TRAIN_TEMP1": null,
                "BOOT_DATE": "0xda7e",
                "RT_SECONDS": "0x7c92",
                "AUX_STATUS": null,
                "ETH_DEBUG_STATUS0": null,
                "ETH_DEBUG_STATUS1": null,
                "TT_FLASH_VERSION": "0x30305",
                "FW_BUNDLE_VERSION": "0x12060000",
                "TIMER_HEARTBEAT": "0x7acf251"
            },
            "board_info": {
                "bus_id": "0000:c8:00.0",
                "board_type": "tt-galaxy-wh L",
                "board_id": "100035100000000",
                "coords": "(0, 0, 0, 0)",
                "dram_status": true,
                "dram_speed": "14G",
                "pcie_speed": 4,
                "pcie_width": "1"
            },
            "telemetry": {
                "voltage": "0.93",
                "current": " 34.0",
                "power": " 31.0",
                "aiclk": "1000",
                "asic_temperature": "28.1",
                "heartbeat": "62777"
            },
            "firmwares": {
                "fw_bundle_version": "18.6.0.0",
                "tt_flash_version": "0.3.3.5",
                "cm_fw": "2.34.0.0",
                "cm_fw_date": "2025-06-22",
                "eth_fw": "7.0.0",
                "bm_bl_fw": "129.2.0.0",
                "bm_app_fw": "5.4.0.0"
            },
            "limits": {
                "vdd_min": "0.80",
                "vdd_max": "1.00",
                "tdp_limit": "170",
                "tdc_limit": "240",
                "asic_fmax": "1000",
                "therm_trip_l1_limit": "83",
                "thm_limit": "75",
                "bus_peak_limit": null
            }
        },
        {
            "smbus_telem": {
                "BOARD_ID": "0x100035100000000",
                "ENUM_VERSION": "0xba5e0001",
                "DEVICE_ID": "0x401e1e52",
                "ASIC_RO": "0x2cfa5",
                "ASIC_IDD": "0xc54",
                "BOARD_ID_HIGH": "0x1000351",
                "BOARD_ID_LOW": null,
                "ARC0_FW_VERSION": "0x2220000",
                "ARC1_FW_VERSION": "0x2220000",
                "ARC2_FW_VERSION": null,
                "ARC3_FW_VERSION": "0x2220000",
                "SPIBOOTROM_FW_VERSION": "0x30d0000",
                "ETH_FW_VERSION": "0x70000",
                "M3_BL_FW_VERSION": "0x81020000",
                "M3_APP_FW_VERSION": "0x5040000",
                "DDR_SPEED": null,
                "DDR_STATUS": "0x1222222",
                "ETH_STATUS0": "0x22222222",
                "ETH_STATUS1": "0x22222222",
                "PCIE_STATUS": "0x10140000",
                "FAULTS": null,
                "ARC0_HEALTH": "0x7acf0e0",
                "ARC1_HEALTH": "0x1d3bc",
                "ARC2_HEALTH": null,
                "ARC3_HEALTH": "0x4ca20",
                "FAN_SPEED": "0xffffffff",
                "AICLK": "0x3e803e8",
                "AXICLK": "0x384",
                "ARCCLK": "0x21c",
                "THROTTLER": null,
                "VCORE": "0x393",
                "ASIC_TEMPERATURE": "0x1d401c5",
                "VREG_TEMPERATURE": null,
                "BOARD_TEMPERATURE": "0x212f2d",
                "TDP": "0xaa0020",
                "TDC": "0xf00023",
                "VDD_LIMITS": "0x3e80320",
                "THM_LIMITS": "0x53004b",
                "WH_FW_DATE": "0x5616150e",
                "ASIC_TMON0": "0x1b1d1d1b",
                "ASIC_TMON1": "0x1d1d",
                "MVDDQ_POWER": "0x190000",
                "GDDR_TRAIN_TEMP0": null,
                "GDDR_TRAIN_TEMP1": null,
                "BOOT_DATE": "0xda7e",
                "RT_SECONDS": "0x7c92",
                "AUX_STATUS": null,
                "ETH_DEBUG_STATUS0": null,
                "ETH_DEBUG_STATUS1": null,
                "TT_FLASH_VERSION": "0x30305",
                "FW_BUNDLE_VERSION": "0x12060000",
                "TIMER_HEARTBEAT": "0x7acf0e0"
            },
            "board_info": {
                "bus_id": "0000:c1:00.0",
                "board_type": "tt-galaxy-wh L",
                "board_id": "100035100000000",
                "coords": "(0, 0, 0, 0)",
                "dram_status": true,
                "dram_speed": "14G",
                "pcie_speed": 4,
                "pcie_width": "1"
            },
            "telemetry": {
                "voltage": "0.92",
                "current": " 35.0",
                "power": " 32.0",
                "aiclk": "1000",
                "asic_temperature": "28.3",
                "heartbeat": "62777"
            },
            "firmwares": {
                "fw_bundle_version": "18.6.0.0",
                "tt_flash_version": "0.3.3.5",
                "cm_fw": "2.34.0.0",
                "cm_fw_date": "2025-06-22",
                "eth_fw": "7.0.0",
                "bm_bl_fw": "129.2.0.0",
                "bm_app_fw": "5.4.0.0"
            },
            "limits": {
                "vdd_min": "0.80",
                "vdd_max": "1.00",
                "tdp_limit": "170",
                "tdc_limit": "240",
                "asic_fmax": "1000",
                "therm_trip_l1_limit": "83",
                "thm_limit": "75",
                "bus_peak_limit": null
            }
        },
        {
            "smbus_telem": {
                "BOARD_ID": "0x100035100000000",
                "ENUM_VERSION": "0xba5e0001",
                "DEVICE_ID": "0x401e1e52",
                "ASIC_RO": "0x2c7e4",
                "ASIC_IDD": "0xc62",
                "BOARD_ID_HIGH": "0x1000351",
                "BOARD_ID_LOW": null,
                "ARC0_FW_VERSION": "0x2220000",
                "ARC1_FW_VERSION": "0x2220000",
                "ARC2_FW_VERSION": null,
                "ARC3_FW_VERSION": "0x2220000",
                "SPIBOOTROM_FW_VERSION": "0x30d0000",
                "ETH_FW_VERSION": "0x70000",
                "M3_BL_FW_VERSION": "0x81020000",
                "M3_APP_FW_VERSION": "0x5040000",
                "DDR_SPEED": null,
                "DDR_STATUS": "0x1222222",
                "ETH_STATUS0": "0x22222222",
                "ETH_STATUS1": "0x22222222",
                "PCIE_STATUS": "0x10140000",
                "FAULTS": null,
                "ARC0_HEALTH": "0x7acf4bb",
                "ARC1_HEALTH": "0x1d3bc",
                "ARC2_HEALTH": null,
                "ARC3_HEALTH": "0x4ca21",
                "FAN_SPEED": "0xffffffff",
                "AICLK": "0x3e803e8",
                "AXICLK": "0x384",
                "ARCCLK": "0x21c",
                "THROTTLER": null,
                "VCORE": "0x39d",
                "ASIC_TEMPERATURE": "0x1cc01b8",
                "VREG_TEMPERATURE": null,
                "BOARD_TEMPERATURE": "0x212f2d",
                "TDP": "0xaa0022",
                "TDC": "0xf00024",
                "VDD_LIMITS": "0x3e80320",
                "THM_LIMITS": "0x53004b",
                "WH_FW_DATE": "0x5616150e",
                "ASIC_TMON0": "0x1b1c1c1a",
                "ASIC_TMON1": "0x1d1c",
                "MVDDQ_POWER": "0x190000",
                "GDDR_TRAIN_TEMP0": null,
                "GDDR_TRAIN_TEMP1": null,
                "BOOT_DATE": "0xda7e",
                "RT_SECONDS": "0x7c92",
                "AUX_STATUS": null,
                "ETH_DEBUG_STATUS0": null,
                "ETH_DEBUG_STATUS1": null,
                "TT_FLASH_VERSION": "0x30305",
                "FW_BUNDLE_VERSION": "0x12060000",
                "TIMER_HEARTBEAT": "0x7acf4bb"
            },
            "board_info": {
                "bus_id": "0000:c2:00.0",
                "board_type": "tt-galaxy-wh L",
                "board_id": "100035100000000",
                "coords": "(0, 0, 0, 0)",
                "dram_status": true,
                "dram_speed": "14G",
                "pcie_speed": 4,
                "pcie_width": "1"
            },
            "telemetry": {
                "voltage": "0.93",
                "current": " 36.0",
                "power": " 34.0",
                "aiclk": "1000",
                "asic_temperature": "27.5",
                "heartbeat": "62777"
            },
            "firmwares": {
                "fw_bundle_version": "18.6.0.0",
                "tt_flash_version": "0.3.3.5",
                "cm_fw": "2.34.0.0",
                "cm_fw_date": "2025-06-22",
                "eth_fw": "7.0.0",
                "bm_bl_fw": "129.2.0.0",
                "bm_app_fw": "5.4.0.0"
            },
            "limits": {
                "vdd_min": "0.80",
                "vdd_max": "1.00",
                "tdp_limit": "170",
                "tdc_limit": "240",
                "asic_fmax": "1000",
                "therm_trip_l1_limit": "83",
                "thm_limit": "75",
                "bus_peak_limit": null
            }
        },
        {
            "smbus_telem": {
                "BOARD_ID": "0x100035100000000",
                "ENUM_VERSION": "0xba5e0001",
                "DEVICE_ID": "0x401e1e52",
                "ASIC_RO": "0x2bd06",
                "ASIC_IDD": "0xb9d",
                "BOARD_ID_HIGH": "0x1000351",
                "BOARD_ID_LOW": null,
                "ARC0_FW_VERSION": "0x2220000",
                "ARC1_FW_VERSION": "0x2220000",
                "ARC2_FW_VERSION": null,
                "ARC3_FW_VERSION": "0x2220000",
                "SPIBOOTROM_FW_VERSION": "0x30d0000",
                "ETH_FW_VERSION": "0x70000",
                "M3_BL_FW_VERSION": "0x81020000",
                "M3_APP_FW_VERSION": "0x5040000",
                "DDR_SPEED": null,
                "DDR_STATUS": "0x1222222",
                "ETH_STATUS0": "0x22222222",
                "ETH_STATUS1": "0x22222222",
                "PCIE_STATUS": "0x10140000",
                "FAULTS": null,
                "ARC0_HEALTH": "0x7acf4e7",
                "ARC1_HEALTH": "0x1d3d2",
                "ARC2_HEALTH": null,
                "ARC3_HEALTH": "0x4ca20",
                "FAN_SPEED": "0xffffffff",
                "AICLK": "0x3e803e8",
                "AXICLK": "0x384",
                "ARCCLK": "0x21c",
                "THROTTLER": null,
                "VCORE": "0x3a7",
                "ASIC_TEMPERATURE": "0x1dc01ca",
                "VREG_TEMPERATURE": null,
                "BOARD_TEMPERATURE": "0x212f2d",
                "TDP": "0xaa0021",
                "TDC": "0xf00023",
                "VDD_LIMITS": "0x3e80320",
                "THM_LIMITS": "0x53004b",
                "WH_FW_DATE": "0x5616150e",
                "ASIC_TMON0": "0x1c1d1e1b",
                "ASIC_TMON1": "0x1d1d",
                "MVDDQ_POWER": "0x190000",
                "GDDR_TRAIN_TEMP0": null,
                "GDDR_TRAIN_TEMP1": null,
                "BOOT_DATE": "0xda7e",
                "RT_SECONDS": "0x7c92",
                "AUX_STATUS": null,
                "ETH_DEBUG_STATUS0": null,
                "ETH_DEBUG_STATUS1": null,
                "TT_FLASH_VERSION": "0x30305",
                "FW_BUNDLE_VERSION": "0x12060000",
                "TIMER_HEARTBEAT": "0x7acf4e7"
            },
            "board_info": {
                "bus_id": "0000:c5:00.0",
                "board_type": "tt-galaxy-wh L",
                "board_id": "100035100000000",
                "coords": "(0, 0, 0, 0)",
                "dram_status": true,
                "dram_speed": "14G",
                "pcie_speed": 4,
                "pcie_width": "1"
            },
            "telemetry": {
                "voltage": "0.94",
                "current": " 35.0",
                "power": " 33.0",
                "aiclk": "1000",
                "asic_temperature": "28.6",
                "heartbeat": "62777"
            },
            "firmwares": {
                "fw_bundle_version": "18.6.0.0",
                "tt_flash_version": "0.3.3.5",
                "cm_fw": "2.34.0.0",
                "cm_fw_date": "2025-06-22",
                "eth_fw": "7.0.0",
                "bm_bl_fw": "129.2.0.0",
                "bm_app_fw": "5.4.0.0"
            },
            "limits": {
                "vdd_min": "0.80",
                "vdd_max": "1.00",
                "tdp_limit": "170",
                "tdc_limit": "240",
                "asic_fmax": "1000",
                "therm_trip_l1_limit": "83",
                "thm_limit": "75",
                "bus_peak_limit": null
            }
        },
        {
            "smbus_telem": {
                "BOARD_ID": "0x100035100000000",
                "ENUM_VERSION": "0xba5e0001",
                "DEVICE_ID": "0x401e1e52",
                "ASIC_RO": "0x2c5af",
                "ASIC_IDD": "0xb8d",
                "BOARD_ID_HIGH": "0x1000351",
                "BOARD_ID_LOW": null,
                "ARC0_FW_VERSION": "0x2220000",
                "ARC1_FW_VERSION": "0x2220000",
                "ARC2_FW_VERSION": null,
                "ARC3_FW_VERSION": "0x2220000",
                "SPIBOOTROM_FW_VERSION": "0x30d0000",
                "ETH_FW_VERSION": "0x70000",
                "M3_BL_FW_VERSION": "0x81020000",
                "M3_APP_FW_VERSION": "0x5040000",
                "DDR_SPEED": null,
                "DDR_STATUS": "0x1222222",
                "ETH_STATUS0": "0x22222222",
                "ETH_STATUS1": "0x22222222",
                "PCIE_STATUS": "0x10140000",
                "FAULTS": null,
                "ARC0_HEALTH": "0x7acefba",
                "ARC1_HEALTH": "0x1d3bc",
                "ARC2_HEALTH": null,
                "ARC3_HEALTH": "0x4ca21",
                "FAN_SPEED": "0xffffffff",
                "AICLK": "0x3e803e8",
                "AXICLK": "0x384",
                "ARCCLK": "0x21c",
                "THROTTLER": null,
                "VCORE": "0x39d",
                "ASIC_TEMPERATURE": "0x1cc01b5",
                "VREG_TEMPERATURE": null,
                "BOARD_TEMPERATURE": "0x212f2d",
                "TDP": "0xaa0020",
                "TDC": "0xf00022",
                "VDD_LIMITS": "0x3e80320",
                "THM_LIMITS": "0x53004b",
                "WH_FW_DATE": "0x5616150e",
                "ASIC_TMON0": "0x1b1c1b1b",
                "ASIC_TMON1": "0x1c1c",
                "MVDDQ_POWER": "0x190000",
                "GDDR_TRAIN_TEMP0": null,
                "GDDR_TRAIN_TEMP1": null,
                "BOOT_DATE": "0xda7e",
                "RT_SECONDS": "0x7c92",
                "AUX_STATUS": null,
                "ETH_DEBUG_STATUS0": null,
                "ETH_DEBUG_STATUS1": null,
                "TT_FLASH_VERSION": "0x30305",
                "FW_BUNDLE_VERSION": "0x12060000",
                "TIMER_HEARTBEAT": "0x7acefba"
            },
            "board_info": {
                "bus_id": "0000:c7:00.0",
                "board_type": "tt-galaxy-wh L",
                "board_id": "100035100000000",
                "coords": "(0, 0, 0, 0)",
                "dram_status": true,
                "dram_speed": "14G",
                "pcie_speed": 4,
                "pcie_width": "1"
            },
            "telemetry": {
                "voltage": "0.93",
                "current": " 34.0",
                "power": " 32.0",
                "aiclk": "1000",
                "asic_temperature": "27.3",
                "heartbeat": "62777"
            },
            "firmwares": {
                "fw_bundle_version": "18.6.0.0",
                "tt_flash_version": "0.3.3.5",
                "cm_fw": "2.34.0.0",
                "cm_fw_date": "2025-06-22",
                "eth_fw": "7.0.0",
                "bm_bl_fw": "129.2.0.0",
                "bm_app_fw": "5.4.0.0"
            },
            "limits": {
                "vdd_min": "0.80",
                "vdd_max": "1.00",
                "tdp_limit": "170",
                "tdc_limit": "240",
                "asic_fmax": "1000",
                "therm_trip_l1_limit": "83",
                "thm_limit": "75",
                "bus_peak_limit": null
            }
        },
        {
            "smbus_telem": {
                "BOARD_ID": "0x100035100000000",
                "ENUM_VERSION": "0xba5e0001",
                "DEVICE_ID": "0x401e1e52",
                "ASIC_RO": "0x2c0ea",
                "ASIC_IDD": "0xbd3",
                "BOARD_ID_HIGH": "0x1000351",
                "BOARD_ID_LOW": null,
                "ARC0_FW_VERSION": "0x2220000",
                "ARC1_FW_VERSION": "0x2220000",
                "ARC2_FW_VERSION": null,
                "ARC3_FW_VERSION": "0x2220000",
                "SPIBOOTROM_FW_VERSION": "0x30d0000",
                "ETH_FW_VERSION": "0x70000",
                "M3_BL_FW_VERSION": "0x81020000",
                "M3_APP_FW_VERSION": "0x5040000",
                "DDR_SPEED": null,
                "DDR_STATUS": "0x1222222",
                "ETH_STATUS0": "0x22222222",
                "ETH_STATUS1": "0x22222222",
                "PCIE_STATUS": "0x10840000",
                "FAULTS": null,
                "ARC0_HEALTH": "0x7acf047",
                "ARC1_HEALTH": "0x1d3d2",
                "ARC2_HEALTH": null,
                "ARC3_HEALTH": "0x4ca20",
                "FAN_SPEED": "0xffffffff",
                "AICLK": "0x3e803e8",
                "AXICLK": "0x384",
                "ARCCLK": "0x21c",
                "THROTTLER": null,
                "VCORE": "0x3a6",
                "ASIC_TEMPERATURE": "0x1ce01bc",
                "VREG_TEMPERATURE": null,
                "BOARD_TEMPERATURE": "0x212f2d",
                "TDP": "0xaa0021",
                "TDC": "0xf00024",
                "VDD_LIMITS": "0x3e80320",
                "THM_LIMITS": "0x53004b",
                "WH_FW_DATE": "0x5616150e",
                "ASIC_TMON0": "0x1b1c1c1d",
                "ASIC_TMON1": "0x1c1c",
                "MVDDQ_POWER": "0x190000",
                "GDDR_TRAIN_TEMP0": null,
                "GDDR_TRAIN_TEMP1": null,
                "BOOT_DATE": "0xda7e",
                "RT_SECONDS": "0x7c92",
                "AUX_STATUS": null,
                "ETH_DEBUG_STATUS0": null,
                "ETH_DEBUG_STATUS1": null,
                "TT_FLASH_VERSION": "0x30305",
                "FW_BUNDLE_VERSION": "0x12060000",
                "TIMER_HEARTBEAT": "0x7acf047"
            },
            "board_info": {
                "bus_id": "0000:c6:00.0",
                "board_type": "tt-galaxy-wh L",
                "board_id": "100035100000000",
                "coords": "(0, 0, 0, 0)",
                "dram_status": true,
                "dram_speed": "14G",
                "pcie_speed": 4,
                "pcie_width": "8"
            },
            "telemetry": {
                "voltage": "0.93",
                "current": " 36.0",
                "power": " 33.0",
                "aiclk": "1000",
                "asic_temperature": "27.8",
                "heartbeat": "62777"
            },
            "firmwares": {
                "fw_bundle_version": "18.6.0.0",
                "tt_flash_version": "0.3.3.5",
                "cm_fw": "2.34.0.0",
                "cm_fw_date": "2025-06-22",
                "eth_fw": "7.0.0",
                "bm_bl_fw": "129.2.0.0",
                "bm_app_fw": "5.4.0.0"
            },
            "limits": {
                "vdd_min": "0.80",
                "vdd_max": "1.00",
                "tdp_limit": "170",
                "tdc_limit": "240",
                "asic_fmax": "1000",
                "therm_trip_l1_limit": "83",
                "thm_limit": "75",
                "bus_peak_limit": null
            }
        },
        {
            "smbus_telem": {
                "BOARD_ID": "0x100035100000000",
                "ENUM_VERSION": "0xba5e0001",
                "DEVICE_ID": "0x401e1e52",
                "ASIC_RO": "0x2e4ea",
                "ASIC_IDD": "0x991",
                "BOARD_ID_HIGH": "0x1000351",
                "BOARD_ID_LOW": null,
                "ARC0_FW_VERSION": "0x2220000",
                "ARC1_FW_VERSION": "0x2220000",
                "ARC2_FW_VERSION": null,
                "ARC3_FW_VERSION": "0x2220000",
                "SPIBOOTROM_FW_VERSION": "0x30d0000",
                "ETH_FW_VERSION": "0x70000",
                "M3_BL_FW_VERSION": "0x81020000",
                "M3_APP_FW_VERSION": "0x5040000",
                "DDR_SPEED": null,
                "DDR_STATUS": "0x1222222",
                "ETH_STATUS0": "0x22222222",
                "ETH_STATUS1": "0x22222222",
                "PCIE_STATUS": "0x10140000",
                "FAULTS": null,
                "ARC0_HEALTH": "0x7aceb71",
                "ARC1_HEALTH": "0x1d3bc",
                "ARC2_HEALTH": null,
                "ARC3_HEALTH": "0x4ca20",
                "FAN_SPEED": "0xffffffff",
                "AICLK": "0x3e803e8",
                "AXICLK": "0x384",
                "ARCCLK": "0x21c",
                "THROTTLER": null,
                "VCORE": "0x380",
                "ASIC_TEMPERATURE": "0x1c901b8",
                "VREG_TEMPERATURE": null,
                "BOARD_TEMPERATURE": "0x212f2d",
                "TDP": "0xaa001d",
                "TDC": "0xf00020",
                "VDD_LIMITS": "0x3e80320",
                "THM_LIMITS": "0x53004b",
                "WH_FW_DATE": "0x5616150e",
                "ASIC_TMON0": "0x1c1d1c1b",
                "ASIC_TMON1": "0x1b1b",
                "MVDDQ_POWER": "0x190000",
                "GDDR_TRAIN_TEMP0": null,
                "GDDR_TRAIN_TEMP1": null,
                "BOOT_DATE": "0xda7e",
                "RT_SECONDS": "0x7c92",
                "AUX_STATUS": null,
                "ETH_DEBUG_STATUS0": null,
                "ETH_DEBUG_STATUS1": null,
                "TT_FLASH_VERSION": "0x30305",
                "FW_BUNDLE_VERSION": "0x12060000",
                "TIMER_HEARTBEAT": "0x7aceb71"
            },
            "board_info": {
                "bus_id": "0000:81:00.0",
                "board_type": "tt-galaxy-wh L",
                "board_id": "100035100000000",
                "coords": "(0, 0, 0, 0)",
                "dram_status": true,
                "dram_speed": "14G",
                "pcie_speed": 4,
                "pcie_width": "1"
            },
            "telemetry": {
                "voltage": "0.90",
                "current": " 32.0",
                "power": " 29.0",
                "aiclk": "1000",
                "asic_temperature": "27.5",
                "heartbeat": "62777"
            },
            "firmwares": {
                "fw_bundle_version": "18.6.0.0",
                "tt_flash_version": "0.3.3.5",
                "cm_fw": "2.34.0.0",
                "cm_fw_date": "2025-06-22",
                "eth_fw": "7.0.0",
                "bm_bl_fw": "129.2.0.0",
                "bm_app_fw": "5.4.0.0"
            },
            "limits": {
                "vdd_min": "0.80",
                "vdd_max": "1.00",
                "tdp_limit": "170",
                "tdc_limit": "240",
                "asic_fmax": "1000",
                "therm_trip_l1_limit": "83",
                "thm_limit": "75",
                "bus_peak_limit": null
            }
        },
        {
            "smbus_telem": {
                "BOARD_ID": "0x100035100000000",
                "ENUM_VERSION": "0xba5e0001",
                "DEVICE_ID": "0x401e1e52",
                "ASIC_RO": "0x2c368",
                "ASIC_IDD": "0x793",
                "BOARD_ID_HIGH": "0x1000351",
                "BOARD_ID_LOW": null,
                "ARC0_FW_VERSION": "0x2220000",
                "ARC1_FW_VERSION": "0x2220000",
                "ARC2_FW_VERSION": null,
                "ARC3_FW_VERSION": "0x2220000",
                "SPIBOOTROM_FW_VERSION": "0x30d0000",
                "ETH_FW_VERSION": "0x70000",
                "M3_BL_FW_VERSION": "0x81020000",
                "M3_APP_FW_VERSION": "0x5040000",
                "DDR_SPEED": null,
                "DDR_STATUS": "0x1222222",
                "ETH_STATUS0": "0x22222222",
                "ETH_STATUS1": "0x22222222",
                "PCIE_STATUS": "0x10140000",
                "FAULTS": null,
                "ARC0_HEALTH": "0x7acefd4",
                "ARC1_HEALTH": "0x1d3bc",
                "ARC2_HEALTH": null,
                "ARC3_HEALTH": "0x4ca20",
                "FAN_SPEED": "0xffffffff",
                "AICLK": "0x3e803e8",
                "AXICLK": "0x384",
                "ARCCLK": "0x21c",
                "THROTTLER": null,
                "VCORE": "0x39d",
                "ASIC_TEMPERATURE": "0x1d601c2",
                "VREG_TEMPERATURE": null,
                "BOARD_TEMPERATURE": "0x212f2d",
                "TDP": "0xaa001e",
                "TDC": "0xf00021",
                "VDD_LIMITS": "0x3e80320",
                "THM_LIMITS": "0x53004b",
                "WH_FW_DATE": "0x5616150e",
                "ASIC_TMON0": "0x1a1d1d1d",
                "ASIC_TMON1": "0x1d1c",
                "MVDDQ_POWER": "0x190000",
                "GDDR_TRAIN_TEMP0": null,
                "GDDR_TRAIN_TEMP1": null,
                "BOOT_DATE": "0xda7e",
                "RT_SECONDS": "0x7c92",
                "AUX_STATUS": null,
                "ETH_DEBUG_STATUS0": null,
                "ETH_DEBUG_STATUS1": null,
                "TT_FLASH_VERSION": "0x30305",
                "FW_BUNDLE_VERSION": "0x12060000",
                "TIMER_HEARTBEAT": "0x7acefd4"
            },
            "board_info": {
                "bus_id": "0000:87:00.0",
                "board_type": "tt-galaxy-wh L",
                "board_id": "100035100000000",
                "coords": "(0, 0, 0, 0)",
                "dram_status": true,
                "dram_speed": "14G",
                "pcie_speed": 4,
                "pcie_width": "1"
            },
            "telemetry": {
                "voltage": "0.93",
                "current": " 33.0",
                "power": " 30.0",
                "aiclk": "1000",
                "asic_temperature": "28.1",
                "heartbeat": "62777"
            },
            "firmwares": {
                "fw_bundle_version": "18.6.0.0",
                "tt_flash_version": "0.3.3.5",
                "cm_fw": "2.34.0.0",
                "cm_fw_date": "2025-06-22",
                "eth_fw": "7.0.0",
                "bm_bl_fw": "129.2.0.0",
                "bm_app_fw": "5.4.0.0"
            },
            "limits": {
                "vdd_min": "0.80",
                "vdd_max": "1.00",
                "tdp_limit": "170",
                "tdc_limit": "240",
                "asic_fmax": "1000",
                "therm_trip_l1_limit": "83",
                "thm_limit": "75",
                "bus_peak_limit": null
            }
        },
        {
            "smbus_telem": {
                "BOARD_ID": "0x100035100000000",
                "ENUM_VERSION": "0xba5e0001",
                "DEVICE_ID": "0x401e1e52",
                "ASIC_RO": "0x2c3b7",
                "ASIC_IDD": "0x7c9",
                "BOARD_ID_HIGH": "0x1000351",
                "BOARD_ID_LOW": null,
                "ARC0_FW_VERSION": "0x2220000",
                "ARC1_FW_VERSION": "0x2220000",
                "ARC2_FW_VERSION": null,
                "ARC3_FW_VERSION": "0x2220000",
                "SPIBOOTROM_FW_VERSION": "0x30d0000",
                "ETH_FW_VERSION": "0x70000",
                "M3_BL_FW_VERSION": "0x81020000",
                "M3_APP_FW_VERSION": "0x5040000",
                "DDR_SPEED": null,
                "DDR_STATUS": "0x1222222",
                "ETH_STATUS0": "0x22222222",
                "ETH_STATUS1": "0x22222222",
                "PCIE_STATUS": "0x10140000",
                "FAULTS": null,
                "ARC0_HEALTH": "0x7acf094",
                "ARC1_HEALTH": "0x1d3bc",
                "ARC2_HEALTH": null,
                "ARC3_HEALTH": "0x4ca20",
                "FAN_SPEED": "0xffffffff",
                "AICLK": "0x3e803e8",
                "AXICLK": "0x384",
                "ARCCLK": "0x21c",
                "THROTTLER": null,
                "VCORE": "0x39d",
                "ASIC_TEMPERATURE": "0x1cc01bc",
                "VREG_TEMPERATURE": null,
                "BOARD_TEMPERATURE": "0x212f2d",
                "TDP": "0xaa0020",
                "TDC": "0xf00022",
                "VDD_LIMITS": "0x3e80320",
                "THM_LIMITS": "0x53004b",
                "WH_FW_DATE": "0x5616150e",
                "ASIC_TMON0": "0x1c1c1c1b",
                "ASIC_TMON1": "0x1c1c",
                "MVDDQ_POWER": "0x190000",
                "GDDR_TRAIN_TEMP0": null,
                "GDDR_TRAIN_TEMP1": null,
                "BOOT_DATE": "0xda7e",
                "RT_SECONDS": "0x7c92",
                "AUX_STATUS": null,
                "ETH_DEBUG_STATUS0": null,
                "ETH_DEBUG_STATUS1": null,
                "TT_FLASH_VERSION": "0x30305",
                "FW_BUNDLE_VERSION": "0x12060000",
                "TIMER_HEARTBEAT": "0x7acf094"
            },
            "board_info": {
                "bus_id": "0000:84:00.0",
                "board_type": "tt-galaxy-wh L",
                "board_id": "100035100000000",
                "coords": "(0, 0, 0, 0)",
                "dram_status": true,
                "dram_speed": "14G",
                "pcie_speed": 4,
                "pcie_width": "1"
            },
            "telemetry": {
                "voltage": "0.93",
                "current": " 34.0",
                "power": " 32.0",
                "aiclk": "1000",
                "asic_temperature": "27.8",
                "heartbeat": "62777"
            },
            "firmwares": {
                "fw_bundle_version": "18.6.0.0",
                "tt_flash_version": "0.3.3.5",
                "cm_fw": "2.34.0.0",
                "cm_fw_date": "2025-06-22",
                "eth_fw": "7.0.0",
                "bm_bl_fw": "129.2.0.0",
                "bm_app_fw": "5.4.0.0"
            },
            "limits": {
                "vdd_min": "0.80",
                "vdd_max": "1.00",
                "tdp_limit": "170",
                "tdc_limit": "240",
                "asic_fmax": "1000",
                "therm_trip_l1_limit": "83",
                "thm_limit": "75",
                "bus_peak_limit": null
            }
        },
        {
            "smbus_telem": {
                "BOARD_ID": "0x100035100000000",
                "ENUM_VERSION": "0xba5e0001",
                "DEVICE_ID": "0x401e1e52",
                "ASIC_RO": "0x2b3fa",
                "ASIC_IDD": "0x70c",
                "BOARD_ID_HIGH": "0x1000351",
                "BOARD_ID_LOW": null,
                "ARC0_FW_VERSION": "0x2220000",
                "ARC1_FW_VERSION": "0x2220000",
                "ARC2_FW_VERSION": null,
                "ARC3_FW_VERSION": "0x2220000",
                "SPIBOOTROM_FW_VERSION": "0x30d0000",
                "ETH_FW_VERSION": "0x70000",
                "M3_BL_FW_VERSION": "0x81020000",
                "M3_APP_FW_VERSION": "0x5040000",
                "DDR_SPEED": null,
                "DDR_STATUS": "0x1222222",
                "ETH_STATUS0": "0x22222222",
                "ETH_STATUS1": "0x22222222",
                "PCIE_STATUS": "0x10140000",
                "FAULTS": null,
                "ARC0_HEALTH": "0x7acef5f",
                "ARC1_HEALTH": "0x1d3bc",
                "ARC2_HEALTH": null,
                "ARC3_HEALTH": "0x4ca1f",
                "FAN_SPEED": "0xffffffff",
                "AICLK": "0x3e803e8",
                "AXICLK": "0x384",
                "ARCCLK": "0x21c",
                "THROTTLER": null,
                "VCORE": "0x3b1",
                "ASIC_TEMPERATURE": "0x1dc01c6",
                "VREG_TEMPERATURE": null,
                "BOARD_TEMPERATURE": "0x212f2d",
                "TDP": "0xaa0020",
                "TDC": "0xf00022",
                "VDD_LIMITS": "0x3e80320",
                "THM_LIMITS": "0x53004b",
                "WH_FW_DATE": "0x5616150e",
                "ASIC_TMON0": "0x1c1d1d1c",
                "ASIC_TMON1": "0x1d1d",
                "MVDDQ_POWER": "0x190000",
                "GDDR_TRAIN_TEMP0": null,
                "GDDR_TRAIN_TEMP1": null,
                "BOOT_DATE": "0xda7e",
                "RT_SECONDS": "0x7c92",
                "AUX_STATUS": null,
                "ETH_DEBUG_STATUS0": null,
                "ETH_DEBUG_STATUS1": null,
                "TT_FLASH_VERSION": "0x30305",
                "FW_BUNDLE_VERSION": "0x12060000",
                "TIMER_HEARTBEAT": "0x7acef5f"
            },
            "board_info": {
                "bus_id": "0000:82:00.0",
                "board_type": "tt-galaxy-wh L",
                "board_id": "100035100000000",
                "coords": "(0, 0, 0, 0)",
                "dram_status": true,
                "dram_speed": "14G",
                "pcie_speed": 4,
                "pcie_width": "1"
            },
            "telemetry": {
                "voltage": "0.94",
                "current": " 34.0",
                "power": " 32.0",
                "aiclk": "1000",
                "asic_temperature": "28.4",
                "heartbeat": "62777"
            },
            "firmwares": {
                "fw_bundle_version": "18.6.0.0",
                "tt_flash_version": "0.3.3.5",
                "cm_fw": "2.34.0.0",
                "cm_fw_date": "2025-06-22",
                "eth_fw": "7.0.0",
                "bm_bl_fw": "129.2.0.0",
                "bm_app_fw": "5.4.0.0"
            },
            "limits": {
                "vdd_min": "0.80",
                "vdd_max": "1.00",
                "tdp_limit": "170",
                "tdc_limit": "240",
                "asic_fmax": "1000",
                "therm_trip_l1_limit": "83",
                "thm_limit": "75",
                "bus_peak_limit": null
            }
        },
        {
            "smbus_telem": {
                "BOARD_ID": "0x100035100000000",
                "ENUM_VERSION": "0xba5e0001",
                "DEVICE_ID": "0x401e1e52",
                "ASIC_RO": "0x2ce11",
                "ASIC_IDD": "0x897",
                "BOARD_ID_HIGH": "0x1000351",
                "BOARD_ID_LOW": null,
                "ARC0_FW_VERSION": "0x2220000",
                "ARC1_FW_VERSION": "0x2220000",
                "ARC2_FW_VERSION": null,
                "ARC3_FW_VERSION": "0x2220000",
                "SPIBOOTROM_FW_VERSION": "0x30d0000",
                "ETH_FW_VERSION": "0x70000",
                "M3_BL_FW_VERSION": "0x81020000",
                "M3_APP_FW_VERSION": "0x5040000",
                "DDR_SPEED": null,
                "DDR_STATUS": "0x1222222",
                "ETH_STATUS0": "0x22222222",
                "ETH_STATUS1": "0x22222222",
                "PCIE_STATUS": "0x10140000",
                "FAULTS": null,
                "ARC0_HEALTH": "0x7acf3a6",
                "ARC1_HEALTH": "0x1d3bc",
                "ARC2_HEALTH": null,
                "ARC3_HEALTH": "0x4ca20",
                "FAN_SPEED": "0xffffffff",
                "AICLK": "0x3e803e8",
                "AXICLK": "0x384",
                "ARCCLK": "0x21c",
                "THROTTLER": null,
                "VCORE": "0x393",
                "ASIC_TEMPERATURE": "0x1cc01bc",
                "VREG_TEMPERATURE": null,
                "BOARD_TEMPERATURE": "0x212f2d",
                "TDP": "0xaa0020",
                "TDC": "0xf00023",
                "VDD_LIMITS": "0x3e80320",
                "THM_LIMITS": "0x53004b",
                "WH_FW_DATE": "0x5616150e",
                "ASIC_TMON0": "0x1b1c1d1b",
                "ASIC_TMON1": "0x1c1c",
                "MVDDQ_POWER": "0x190000",
                "GDDR_TRAIN_TEMP0": null,
                "GDDR_TRAIN_TEMP1": null,
                "BOOT_DATE": "0xda7e",
                "RT_SECONDS": "0x7c92",
                "AUX_STATUS": null,
                "ETH_DEBUG_STATUS0": null,
                "ETH_DEBUG_STATUS1": null,
                "TT_FLASH_VERSION": "0x30305",
                "FW_BUNDLE_VERSION": "0x12060000",
                "TIMER_HEARTBEAT": "0x7acf3a6"
            },
            "board_info": {
                "bus_id": "0000:88:00.0",
                "board_type": "tt-galaxy-wh L",
                "board_id": "100035100000000",
                "coords": "(0, 0, 0, 0)",
                "dram_status": true,
                "dram_speed": "14G",
                "pcie_speed": 4,
                "pcie_width": "1"
            },
            "telemetry": {
                "voltage": "0.92",
                "current": " 35.0",
                "power": " 32.0",
                "aiclk": "1000",
                "asic_temperature": "27.8",
                "heartbeat": "62777"
            },
            "firmwares": {
                "fw_bundle_version": "18.6.0.0",
                "tt_flash_version": "0.3.3.5",
                "cm_fw": "2.34.0.0",
                "cm_fw_date": "2025-06-22",
                "eth_fw": "7.0.0",
                "bm_bl_fw": "129.2.0.0",
                "bm_app_fw": "5.4.0.0"
            },
            "limits": {
                "vdd_min": "0.80",
                "vdd_max": "1.00",
                "tdp_limit": "170",
                "tdc_limit": "240",
                "asic_fmax": "1000",
                "therm_trip_l1_limit": "83",
                "thm_limit": "75",
                "bus_peak_limit": null
            }
        },
        {
            "smbus_telem": {
                "BOARD_ID": "0x100035100000000",
                "ENUM_VERSION": "0xba5e0001",
                "DEVICE_ID": "0x401e1e52",
                "ASIC_RO": "0x2cf83",
                "ASIC_IDD": "0x846",
                "BOARD_ID_HIGH": "0x1000351",
                "BOARD_ID_LOW": null,
                "ARC0_FW_VERSION": "0x2220000",
                "ARC1_FW_VERSION": "0x2220000",
                "ARC2_FW_VERSION": null,
                "ARC3_FW_VERSION": "0x2220000",
                "SPIBOOTROM_FW_VERSION": "0x30d0000",
                "ETH_FW_VERSION": "0x70000",
                "M3_BL_FW_VERSION": "0x81020000",
                "M3_APP_FW_VERSION": "0x5040000",
                "DDR_SPEED": null,
                "DDR_STATUS": "0x1222222",
                "ETH_STATUS0": "0x22222222",
                "ETH_STATUS1": "0x22222222",
                "PCIE_STATUS": "0x10140000",
                "FAULTS": null,
                "ARC0_HEALTH": "0x7acf2c4",
                "ARC1_HEALTH": "0x1d3bc",
                "ARC2_HEALTH": null,
                "ARC3_HEALTH": "0x4ca20",
                "FAN_SPEED": "0xffffffff",
                "AICLK": "0x3e803e8",
                "AXICLK": "0x384",
                "ARCCLK": "0x21c",
                "THROTTLER": null,
                "VCORE": "0x393",
                "ASIC_TEMPERATURE": "0x1d401c0",
                "VREG_TEMPERATURE": null,
                "BOARD_TEMPERATURE": "0x212f2d",
                "TDP": "0xaa0020",
                "TDC": "0xf00023",
                "VDD_LIMITS": "0x3e80320",
                "THM_LIMITS": "0x53004b",
                "WH_FW_DATE": "0x5616150e",
                "ASIC_TMON0": "0x1b1d1d1b",
                "ASIC_TMON1": "0x1c1c",
                "MVDDQ_POWER": "0x190000",
                "GDDR_TRAIN_TEMP0": null,
                "GDDR_TRAIN_TEMP1": null,
                "BOOT_DATE": "0xda7e",
                "RT_SECONDS": "0x7c92",
                "AUX_STATUS": null,
                "ETH_DEBUG_STATUS0": null,
                "ETH_DEBUG_STATUS1": null,
                "TT_FLASH_VERSION": "0x30305",
                "FW_BUNDLE_VERSION": "0x12060000",
                "TIMER_HEARTBEAT": "0x7acf2c4"
            },
            "board_info": {
                "bus_id": "0000:83:00.0",
                "board_type": "tt-galaxy-wh L",
                "board_id": "100035100000000",
                "coords": "(0, 0, 0, 0)",
                "dram_status": true,
                "dram_speed": "14G",
                "pcie_speed": 4,
                "pcie_width": "1"
            },
            "telemetry": {
                "voltage": "0.92",
                "current": " 35.0",
                "power": " 32.0",
                "aiclk": "1000",
                "asic_temperature": "28.0",
                "heartbeat": "62777"
            },
            "firmwares": {
                "fw_bundle_version": "18.6.0.0",
                "tt_flash_version": "0.3.3.5",
                "cm_fw": "2.34.0.0",
                "cm_fw_date": "2025-06-22",
                "eth_fw": "7.0.0",
                "bm_bl_fw": "129.2.0.0",
                "bm_app_fw": "5.4.0.0"
            },
            "limits": {
                "vdd_min": "0.80",
                "vdd_max": "1.00",
                "tdp_limit": "170",
                "tdc_limit": "240",
                "asic_fmax": "1000",
                "therm_trip_l1_limit": "83",
                "thm_limit": "75",
                "bus_peak_limit": null
            }
        },
       

Would you like to help us implement this feature by sending a PR?

Yes

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions