Skip to content

Disaggregated serving #365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 15 commits into from
Apr 18, 2025
Merged

Disaggregated serving #365

merged 15 commits into from
Apr 18, 2025

Conversation

quic-amitraj
Copy link
Contributor

@quic-amitraj quic-amitraj commented Apr 16, 2025 •
edited
Loading

Adding support of-

  1. prefill_only
  2. mdp_ts_json_path

ochougul
ochougul reviewed Apr 17, 2025
View reviewed changes
QEfficient/base/modeling_qeff.py Outdated
@@ -300,7 +308,7 @@ def _compile(
command.append(f"-custom-IO-list-file={custom_io_yaml}")

# Write mdp_config.json file
if mdp_ts_num_devices > 1:
if not mdp_ts_json_path:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if not mdp_ts_json_path and mdp_ts_num_devices>1

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not addressed yet?

ochougul
ochougul reviewed Apr 17, 2025
View reviewed changes
ochougul
ochougul reviewed Apr 17, 2025
View reviewed changes
ochougul
ochougul reviewed Apr 17, 2025
View reviewed changes
ochougul
ochougul requested changes Apr 18, 2025
View reviewed changes
QEfficient/base/modeling_qeff.py Outdated
@@ -300,7 +308,7 @@ def _compile(
command.append(f"-custom-IO-list-file={custom_io_yaml}")

# Write mdp_config.json file
if mdp_ts_num_devices > 1:
if not mdp_ts_json_path:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not addressed yet?

ochougul and others added 4 commits April 18, 2025 14:25
@ochougul
simplified _compile code
3516bb6 
Signed-off-by: Onkar Chougule <[email protected]>
@ochougul
Merge branch 'main' into dist_serve
eb92774 
Signed-off-by: Onkar Chougule <[email protected]>
@quic-amitraj
Comments addressed
e82eaca 
Signed-off-by: Amit Raj <[email protected]>
@quic-amitraj
Lint and formated
35bfd35 
Signed-off-by: Amit Raj <[email protected]>
@ochougul ochougul marked this pull request as ready for review April 18, 2025 11:02
@ochougul ochougul requested a review from quic-rishinr as a code owner April 18, 2025 11:02
@ochougul
fixed modelling qeff
1c372cf 
Signed-off-by: Onkar Chougule <[email protected]>
@ochougul ochougul mentioned this pull request Apr 18, 2025
Closed
@quic-amitraj
test bug fixed
8243ea2 
Signed-off-by: Amit Raj <[email protected]>
@quic-amitraj quic-amitraj merged commit 3de4072 into main Apr 18, 2025
5 checks passed
@quic-xiyushi
Copy link

Is the prefill_only flag available only for QEFFAutoModelForCausalLM? Why don't we support it for other classes as well, such as multimodal models?

eplatero97 pushed a commit to eplatero97/efficient-transformers that referenced this pull request Apr 29, 2025
@quic-amitraj @quic-rishinr
@ochougul @eplatero97
Disaggregated serving (quic#365)
51b1cf1 
Adding support of-
1. `prefill_only`
2. `compile_for` for VLM
3. `mdp_ts_json_path`

---------

Signed-off-by: Rishin Raj <[email protected]>
Signed-off-by: Amit Raj <[email protected]>
Signed-off-by: Onkar Chougule <[email protected]>
Signed-off-by: Onkar Chougule <[email protected]>
Co-authored-by: Rishin Raj <[email protected]>
Co-authored-by: Onkar Chougule <[email protected]>
Co-authored-by: Onkar Chougule <[email protected]>
eplatero97 pushed a commit to eplatero97/efficient-transformers that referenced this pull request Apr 29, 2025
@quic-amitraj @quic-rishinr
@ochougul @eplatero97
Disaggregated serving (quic#365)
91eeb68 
Adding support of-
1. `prefill_only`
2. `compile_for` for VLM
3. `mdp_ts_json_path`

---------

Signed-off-by: Rishin Raj <[email protected]>
Signed-off-by: Amit Raj <[email protected]>
Signed-off-by: Onkar Chougule <[email protected]>
Signed-off-by: Onkar Chougule <[email protected]>
Co-authored-by: Rishin Raj <[email protected]>
Co-authored-by: Onkar Chougule <[email protected]>
Co-authored-by: Onkar Chougule <[email protected]>
eplatero97 pushed a commit to eplatero97/efficient-transformers that referenced this pull request Apr 29, 2025
@quic-amitraj @quic-rishinr @ochougul
Disaggregated serving (quic#365)
c45c3e9 
Adding support of-
1. `prefill_only`
2. `compile_for` for VLM
3. `mdp_ts_json_path`

---------

Signed-off-by: Rishin Raj <[email protected]>
Signed-off-by: Amit Raj <[email protected]>
Signed-off-by: Onkar Chougule <[email protected]>
Signed-off-by: Onkar Chougule <[email protected]>
Co-authored-by: Rishin Raj <[email protected]>
Co-authored-by: Onkar Chougule <[email protected]>
Co-authored-by: Onkar Chougule <[email protected]>
Signed-off-by: eplatero <[email protected]>
eplatero97 pushed a commit to eplatero97/efficient-transformers that referenced this pull request Apr 29, 2025
@quic-amitraj @eplatero97
Disaggregated serving (quic#365)
cd06158 
Adding support of-
1. `prefill_only`
2. `compile_for` for VLM
3. `mdp_ts_json_path`

---------

Signed-off-by: Rishin Raj <[email protected]>
Signed-off-by: Amit Raj <[email protected]>
Signed-off-by: Onkar Chougule <[email protected]>
Signed-off-by: Onkar Chougule <[email protected]>
Co-authored-by: Rishin Raj <[email protected]>
Co-authored-by: Onkar Chougule <[email protected]>
Co-authored-by: Onkar Chougule <[email protected]>

Signed-off-by: eplatero <[email protected]>
eplatero97 pushed a commit to eplatero97/efficient-transformers that referenced this pull request Apr 29, 2025
@quic-amitraj @eplatero97
Disaggregated serving (quic#365)
8f0c080 
Adding support of-
1. `prefill_only`
2. `compile_for` for VLM
3. `mdp_ts_json_path`

---------

Signed-off-by: Rishin Raj <[email protected]>
Signed-off-by: Amit Raj <[email protected]>
Signed-off-by: Onkar Chougule <[email protected]>
Signed-off-by: Onkar Chougule <[email protected]>
Co-authored-by: Rishin Raj <[email protected]>
Co-authored-by: Onkar Chougule <[email protected]>
Co-authored-by: Onkar Chougule <[email protected]>

Signed-off-by: eplatero <[email protected]>
eplatero97 pushed a commit to eplatero97/efficient-transformers that referenced this pull request Apr 29, 2025
@quic-amitraj @eplatero97
Disaggregated serving (quic#365)
c5e9513 
Adding support of-
1. `prefill_only`
2. `compile_for` for VLM
3. `mdp_ts_json_path`

---------

Signed-off-by: Rishin Raj <[email protected]>
Signed-off-by: Amit Raj <[email protected]>
Signed-off-by: Onkar Chougule <[email protected]>
Signed-off-by: Onkar Chougule <[email protected]>
Co-authored-by: Rishin Raj <[email protected]>
Co-authored-by: Onkar Chougule <[email protected]>
Co-authored-by: Onkar Chougule <[email protected]>

Signed-off-by: eplatero <[email protected]>
eplatero97 pushed a commit to eplatero97/efficient-transformers that referenced this pull request Apr 29, 2025
@quic-amitraj @eplatero97
Disaggregated serving (quic#365)
ac539ab 
Adding support of-
1. `prefill_only`
2. `compile_for` for VLM
3. `mdp_ts_json_path`

---------

Signed-off-by: Rishin Raj <[email protected]>
Signed-off-by: Amit Raj <[email protected]>
Signed-off-by: Onkar Chougule <[email protected]>
Signed-off-by: Onkar Chougule <[email protected]>
Co-authored-by: Rishin Raj <[email protected]>
Co-authored-by: Onkar Chougule <[email protected]>
Co-authored-by: Onkar Chougule <[email protected]>

Signed-off-by: eplatero <[email protected]>
@quic-rishinr quic-rishinr deleted the dist_serve branch June 13, 2025 08:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ochougul ochougul ochougul requested changes

@quic-rishinr quic-rishinr Awaiting requested review from quic-rishinr quic-rishinr is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@quic-amitraj @quic-xiyushi @ochougul @quic-rishinr