Add image file viewing support to FileEditor

[SmartManoj]

Fixes: #1003

FileEditor now supports viewing image files (.png, .jpg, .jpeg, .gif, .webp, .bmp) by returning base64-encoded image data and displaying it as ImageContent. Updated FileEditorObservation and tool description to handle image data, and added tests to verify image handling and backward compatibility with text files.

Before:

After:

[simonrosenberg]

Thank you @SmartManoj !

Perhaps we should wait for #929 to be merged before merging this though. Will let you know very soon

[blacksmith-sh]

[Automatic Post]: I have assigned @xingyaoww as a reviewer based on git blame information. Thanks in advance for the help!

[xingyaoww]

Can we add a testcase that actually load an image file and assert the file editor will return ImageContent without error?

[enyst]

This is very nice, thank you for the work on it!

[simonrosenberg]

Thank you for doing this!

I am no expert but all those if/elifs seem a bit hacky here. Would it make sense to use a library instead?

import mimetypes

mime_type, _ = mimetypes.guess_type(path)
mime_type = mime_type or "application/octet-stream"

or

import imghdr

image_type = imghdr.what(path)
mime_type = f"image/{image_type}" if image_type else "application/octet-stream"

[SmartManoj]

Would it make sense to use a library instead?

Yes.

---

Similar snippet in codebase:

software-agent-sdk/openhands-tools/openhands/tools/browser_use/definition.py

Lines 50 to 59 in 3aa647e

	if self.screenshot_data:
	mime_type = "image/png"
	if self.screenshot_data.startswith("/9j/"):
	mime_type = "image/jpeg"
	elif self.screenshot_data.startswith("iVBORw0KGgo"):
	mime_type = "image/png"
	elif self.screenshot_data.startswith("R0lGODlh"):
	mime_type = "image/gif"
	elif self.screenshot_data.startswith("UklGR"):
	mime_type = "image/webp"

[simonrosenberg]

@OpenHands /codereview-roasted read the comments above
the following block of code is 1. duplicated (copy-pasted from software-agent-sdk/openhands-tools/openhands/tools/browser_use/definition.py#L50-L59) 2. a bit hacky.
What are you suggestions to improve this?

 if self.screenshot_data: 
     mime_type = "image/png" 
     if self.screenshot_data.startswith("/9j/"): 
         mime_type = "image/jpeg" 
     elif self.screenshot_data.startswith("iVBORw0KGgo"): 
         mime_type = "image/png" 
     elif self.screenshot_data.startswith("R0lGODlh"): 
         mime_type = "image/gif" 
     elif self.screenshot_data.startswith("UklGR"): 
         mime_type = "image/webp"

[openhands-ai]

I'm on it! simonrosenberg can track my progress at all-hands.dev

[openhands-ai]

Code Review Summary

I've analyzed the duplicated and hacky MIME type detection code in PR #1016. Here's my assessment:

Problems Identified:

1. Code Duplication: The exact same MIME type detection logic (lines 344-354 in file_editor/editor.py) is copy-pasted from browser_use/definition.py (lines 50-59)

2. Wrong Approach: The code detects MIME types by pattern-matching base64-encoded strings rather than checking actual file data. This is brittle because you're inspecting an encoding artifact, not the actual file format.

3. Ignoring Available Data: In file_editor/editor.py, you have the file path but still read the entire file, encode it to base64, and pattern-match strings - when Python's mimetypes.guess_type(path) would do this correctly in one line.

Recommended Solutions:

For file_editor.py (SIMPLEST FIX):
Replace lines 344-354 with:

import mimetypes

mime_type, _ = mimetypes.guess_type(path)
if not mime_type or not mime_type.startswith('image/'):
    mime_type = 'application/octet-stream'

This is simpler (3 lines vs 10), faster (no file inspection needed), more correct (uses standard library), and more maintainable (handles all formats automatically).

For long-term maintainability:
Create a shared utility function in openhands-tools/openhands/tools/utils/mime_type.py that:

• Uses mimetypes.guess_type() when a file path is available
• Falls back to checking actual magic numbers (not base64 prefixes) for raw data
• Can be used by both file_editor and browser_use to eliminate duplication

Verdict:

❌ Needs rework - Violates DRY principle and uses incorrect approach for type detection

The current approach works but is unnecessarily complex and will become a maintenance burden. Using Python's standard library is the pragmatic, simple solution that eliminates special cases.

_{View full conversation}

[simonrosenberg]

Perhaps we can implement the recommended solution @SmartManoj ? I dont think we need the long-term maintainability solution yet

[SmartManoj]

mime_type = 'application/octet-stream'

@simonrosenberg, what do @OpenHands think about quote-style?

[simonrosenberg]

Indeed " is preferred!

[xingyaoww]

@OpenHands review the changes introduce in the current PR, then help me add an integration test under tests/integration/tests very similar to its existing format.

The test should work similar to examples/01_standalone_sdk/17_image_input.py:

1. it sets up the workspace by downloading the logo in example 17
2. it ask the agent what the color of the logo is (blue, yellow, or green)
3. in check phase, it should check the agent answered logo have yellow color

[openhands-ai]

I'm on it! xingyaoww can track my progress at all-hands.dev

[openhands-ai]

Summary

I've successfully completed your request! Here's what was done:

✅ Reviewed PR #1016 Changes

The PR adds image file viewing support to FileEditor:

• FileEditor now supports viewing image files (.png, .jpg, .jpeg, .gif, .webp, .bmp)
• Images are returned as base64-encoded data wrapped in ImageContent
• File validation was updated to allow image files (previously rejected as binary)
• Unit tests verify image handling and backward compatibility

✅ Created Integration Test

Added tests/integration/tests/t08_image_file_viewing.py that:

1. Setup Phase: Downloads the OpenHands logo from the same URL used in example 17
2. Instruction: Asks the agent to view logo.png and identify if it's blue, yellow, or green
3. Verification Phase: Checks that the agent's response mentions "yellow" color
4. Vision Check: Ensures the test runs only with vision-capable LLM models (like example 17)

The test follows the existing integration test pattern (extends BaseIntegrationTest) and will be automatically discovered by the test runner.

✅ Quality Checks

• All pre-commit hooks passed (formatting, linting, type checking)
• Changes committed with proper co-author attribution
• Pushed to branch img to update PR Add image file viewing support to FileEditor #1016

The integration test is now ready to be executed either through GitHub Actions (using the integration-test label) or locally via run_infer.py.

_{View full conversation}