Qwen3.5: is there a separate vision encoder?¶
In [1]:
import inspect
import torch
import transformers
from IPython.display import display
from PIL import Image
from transformers import AutoConfig, AutoModelForMultimodalLM, AutoProcessor
print('torch:', torch.__version__)
print('transformers:', transformers.__version__)
print('cuda:', torch.cuda.is_available(), 'num_gpus:', torch.cuda.device_count())
torch: 2.12.0+cu130 transformers: 5.12.0 cuda: True num_gpus: 2
1. Fully load Qwen/Qwen3.5-2B with the normal Hugging Face path¶
In [2]:
model_id = 'Qwen/Qwen3.5-2B'
config = AutoConfig.from_pretrained(model_id, trust_remote_code=True)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForMultimodalLM.from_pretrained(
model_id,
dtype=torch.bfloat16,
device_map={'': 'cuda:0'},
trust_remote_code=True,
)
print('HF repo:', model_id)
print('config class:', config.__class__.__module__ + '.' + config.__class__.__name__)
print('config model_type:', config.model_type)
print('config architectures:', config.architectures)
print('processor class:', processor.__class__.__module__ + '.' + processor.__class__.__name__)
print('model class:', model.__class__.__module__ + '.' + model.__class__.__name__)
print('device map:', getattr(model, 'hf_device_map', None))
[transformers] The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
[ERROR] `loss` is part of Qwen3_5CausalLMOutputWithPast.__init__'s signature, but not documented. Make sure to add it to the docstring of the function in /mnt/lustre/work/oh/owl661/patches-playground/.venv/lib/python3.13/site-packages/transformers/models/qwen3_5/modeling_qwen3_5.py. [ERROR] `logits` is part of Qwen3_5CausalLMOutputWithPast.__init__'s signature, but not documented. Make sure to add it to the docstring of the function in /mnt/lustre/work/oh/owl661/patches-playground/.venv/lib/python3.13/site-packages/transformers/models/qwen3_5/modeling_qwen3_5.py.
HF repo: Qwen/Qwen3.5-2B config class: transformers.models.qwen3_5.configuration_qwen3_5.Qwen3_5Config config model_type: qwen3_5 config architectures: ['Qwen3_5ForConditionalGeneration'] processor class: transformers.models.qwen3_vl.processing_qwen3_vl.Qwen3VLProcessor model class: transformers.models.qwen3_5.modeling_qwen3_5.Qwen3_5ForConditionalGeneration device map: None
2. The loaded model has separate visual and language_model children¶
In [3]:
print('Children inside model.model:')
print(list(model.model._modules.keys()))
print('\nSeparate objects:')
print('model.model.visual:', model.model.visual.__class__.__module__ + '.' + model.model.visual.__class__.__name__)
print('model.model.language_model:', model.model.language_model.__class__.__module__ + '.' + model.model.language_model.__class__.__name__)
print('visual is language_model:', model.model.visual is model.model.language_model)
visual_params = sum(p.numel() for _, p in model.model.visual.named_parameters(recurse=True, remove_duplicate=True))
language_params = sum(p.numel() for _, p in model.model.language_model.named_parameters(recurse=True, remove_duplicate=True))
lm_head_params = sum(p.numel() for _, p in model.lm_head.named_parameters(recurse=True, remove_duplicate=True))
print('\nSeparate parameter counts:')
print('model.visual params:', f'{visual_params:,}')
print('model.language_model params:', f'{language_params:,}')
print('lm_head params:', f'{lm_head_params:,}')
Children inside model.model: ['visual', 'language_model'] Separate objects: model.model.visual: transformers.models.qwen3_5.modeling_qwen3_5.Qwen3_5VisionModel model.model.language_model: transformers.models.qwen3_5.modeling_qwen3_5.Qwen3_5TextModel visual is language_model: False Separate parameter counts: model.visual params: 331,416,576 model.language_model params: 1,881,825,088 lm_head params: 508,559,360
2b. self.visual is a full vision model, not a tiny helper¶
In [4]:
print(model.model.visual)
print('\nDirect children of model.model.visual:')
for name, child in model.model.visual.named_children():
child_params = sum(p.numel() for _, p in child.named_parameters(recurse=True, remove_duplicate=True))
print(f'{name:15s} {child.__class__.__name__:28s} params={child_params:,}')
print('\nVision block count:', len(model.model.visual.blocks))
print('Vision total params:', f'{visual_params:,}')
print('First vision block:')
print(model.model.visual.blocks[0])
Qwen3_5VisionModel(
(patch_embed): Qwen3_5VisionPatchEmbed(
(proj): Conv3d(3, 1024, kernel_size=(2, 16, 16), stride=(2, 16, 16))
)
(pos_embed): Embedding(2304, 1024)
(rotary_pos_emb): Qwen3_5VisionRotaryEmbedding()
(blocks): ModuleList(
(0-23): 24 x Qwen3_5VisionBlock(
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True, bias=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True, bias=True)
(attn): Qwen3_5VisionAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): Qwen3_5VisionMLP(
(linear_fc1): Linear(in_features=1024, out_features=4096, bias=True)
(linear_fc2): Linear(in_features=4096, out_features=1024, bias=True)
(act_fn): GELUTanh()
)
)
)
(merger): Qwen3_5VisionPatchMerger(
(norm): LayerNorm((1024,), eps=1e-06, elementwise_affine=True, bias=True)
(linear_fc1): Linear(in_features=4096, out_features=4096, bias=True)
(act_fn): GELU(approximate='none')
(linear_fc2): Linear(in_features=4096, out_features=2048, bias=True)
)
)
Direct children of model.model.visual:
patch_embed Qwen3_5VisionPatchEmbed params=1,573,888
pos_embed Embedding params=2,359,296
rotary_pos_emb Qwen3_5VisionRotaryEmbedding params=0
blocks ModuleList params=302,309,376
merger Qwen3_5VisionPatchMerger params=25,174,016
Vision block count: 24
Vision total params: 331,416,576
First vision block:
Qwen3_5VisionBlock(
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True, bias=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True, bias=True)
(attn): Qwen3_5VisionAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): Qwen3_5VisionMLP(
(linear_fc1): Linear(in_features=1024, out_features=4096, bias=True)
(linear_fc2): Linear(in_features=4096, out_features=1024, bias=True)
(act_fn): GELUTanh()
)
)
3. Runtime hook trace from a real forward pass¶
In [5]:
image = Image.new('RGB', (64, 64), color=(255, 0, 0))
messages = [
{
'role': 'user',
'content': [
{'type': 'image', 'image': image},
{'type': 'text', 'text': 'color?'},
],
}
]
batch = processor.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_dict=True,
return_tensors='pt',
)
batch = {key: value.to('cuda:0') if hasattr(value, 'to') else value for key, value in batch.items()}
print('Actual model input tensors:')
for key, value in batch.items():
if hasattr(value, 'shape'):
print(key, tuple(value.shape), value.dtype)
trace = []
watched_modules = {
'root': model,
'model': model.model,
'model.visual': model.model.visual,
'model.language_model': model.model.language_model,
'lm_head': model.lm_head,
}
def record_forward_pre_hook(module_name):
def hook(module, args, kwargs):
seen_shapes = []
for value in args:
if hasattr(value, 'shape'):
seen_shapes.append(tuple(value.shape))
for key, value in kwargs.items():
if hasattr(value, 'shape'):
seen_shapes.append((key, tuple(value.shape)))
trace.append(
{
'module': module_name,
'class': module.__class__.__name__,
'input_shapes_seen_by_hook': seen_shapes,
}
)
return hook
handles = []
for module_name, module in watched_modules.items():
handles.append(module.register_forward_pre_hook(record_forward_pre_hook(module_name), with_kwargs=True))
with torch.inference_mode():
outputs = model(**batch, use_cache=False)
for handle in handles:
handle.remove()
print('\nRuntime trace captured by hooks during this forward pass:')
for index, event in enumerate(trace, start=1):
print(index, event['module'], '=>', event['class'], event['input_shapes_seen_by_hook'])
print('\nOutput logits shape:', tuple(outputs.logits.shape))
runtime_edges = [(trace[i]['module'], trace[i + 1]['module']) for i in range(len(trace) - 1)]
print('\nRuntime edges derived directly from hook order:')
for left, right in runtime_edges:
print(left, '->', right)
Actual model input tensors: input_ids (1, 80) torch.int64 attention_mask (1, 80) torch.int64 mm_token_type_ids (1, 80) torch.int64 pixel_values (256, 1536) torch.float32 image_grid_thw (1, 3) torch.int64
Runtime trace captured by hooks during this forward pass:
1 root => Qwen3_5ForConditionalGeneration [('input_ids', (1, 80)), ('attention_mask', (1, 80)), ('mm_token_type_ids', (1, 80)), ('pixel_values', (256, 1536)), ('image_grid_thw', (1, 3))]
2 model => Qwen3_5Model [('input_ids', (1, 80)), ('pixel_values', (256, 1536)), ('image_grid_thw', (1, 3)), ('attention_mask', (1, 80)), ('mm_token_type_ids', (1, 80))]
3 model.visual => Qwen3_5VisionModel [(256, 1536), ('grid_thw', (1, 3))]
4 model.language_model => Qwen3_5TextModel [('position_ids', (3, 1, 80)), ('attention_mask', (1, 80)), ('inputs_embeds', (1, 80, 2048))]
5 lm_head => Linear [(1, 80, 2048)]
Output logits shape: (1, 80, 248320)
Runtime edges derived directly from hook order:
root -> model
model -> model.visual
model.visual -> model.language_model
model.language_model -> lm_head
4. Block-level flow from the same real forward pass¶
In [6]:
def first_tensor_shape(value):
if hasattr(value, 'shape'):
return tuple(value.shape)
if isinstance(value, dict):
for item in value.values():
shape = first_tensor_shape(item)
if shape is not None:
return shape
if isinstance(value, (list, tuple)):
for item in value:
shape = first_tensor_shape(item)
if shape is not None:
return shape
for attr in ['last_hidden_state', 'pooler_output', 'logits']:
if hasattr(value, attr):
shape = first_tensor_shape(getattr(value, attr))
if shape is not None:
return shape
return None
block_trace = []
block_handles = []
def add_block_hook(module_name, module):
def hook(mod, args, kwargs, output):
block_trace.append({
'name': module_name,
'class': mod.__class__.__name__,
'input_shape': first_tensor_shape(args) or first_tensor_shape(kwargs),
'output_shape': first_tensor_shape(output),
})
block_handles.append(module.register_forward_hook(hook, with_kwargs=True))
add_block_hook('model.visual.patch_embed', model.model.visual.patch_embed)
for i, block in enumerate(model.model.visual.blocks):
add_block_hook(f'model.visual.blocks.{i}', block)
add_block_hook('model.visual.merger', model.model.visual.merger)
add_block_hook('model.language_model.embed_tokens', model.model.language_model.embed_tokens)
for i, layer in enumerate(model.model.language_model.layers):
add_block_hook(f'model.language_model.layers.{i}', layer)
add_block_hook('model.language_model.norm', model.model.language_model.norm)
add_block_hook('lm_head', model.lm_head)
with torch.inference_mode():
block_output = model(**batch, use_cache=False)
for handle in block_handles:
handle.remove()
image_pad_id = processor.tokenizer.convert_tokens_to_ids('<|image_pad|>')
image_positions = (batch['input_ids'][0] == image_pad_id).nonzero(as_tuple=False).flatten()
non_image_positions = (batch['input_ids'][0] != image_pad_id).nonzero(as_tuple=False).flatten()
merge_row = {
'name': 'Qwen3_5Model.forward image-token insertion',
'class': 'masked_scatter / replace image placeholders',
'input_shape': f"text_embeds (1, 80, 2048) + visual_embeds ({int(image_positions.numel())}, 2048)",
'output_shape': '(1, 80, 2048)',
}
merger_index = next(i for i, row in enumerate(block_trace) if row['name'] == 'model.visual.merger')
layer0_index = next(i for i, row in enumerate(block_trace) if row['name'] == 'model.language_model.layers.0')
display_trace = block_trace[: merger_index + 1] + [merge_row] + block_trace[layer0_index:]
print('Executed modules in order, captured by forward hooks:')
print('(One row is an explicit non-module operation from Qwen3_5Model.forward: image-token insertion.)')
for i, row in enumerate(display_trace, start=1):
print(f"{i:02d} {row['name']:34s} {row['class']:28s} in={row['input_shape']} out={row['output_shape']}")
print('\nfinal logits:', tuple(block_output.logits.shape))
print('\nImportant: the first language layer input is (1, 80, 2048) because the row immediately before it inserted 64 visual embeddings into 64 image-placeholder positions inside the 80-position sequence. The next cell verifies that replacement directly.')
Executed modules in order, captured by forward hooks: (One row is an explicit non-module operation from Qwen3_5Model.forward: image-token insertion.) 01 model.language_model.embed_tokens Embedding in=(1, 80) out=(1, 80, 2048) 02 model.visual.patch_embed Qwen3_5VisionPatchEmbed in=(256, 1536) out=(256, 1024) 03 model.visual.blocks.0 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 04 model.visual.blocks.1 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 05 model.visual.blocks.2 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 06 model.visual.blocks.3 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 07 model.visual.blocks.4 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 08 model.visual.blocks.5 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 09 model.visual.blocks.6 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 10 model.visual.blocks.7 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 11 model.visual.blocks.8 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 12 model.visual.blocks.9 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 13 model.visual.blocks.10 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 14 model.visual.blocks.11 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 15 model.visual.blocks.12 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 16 model.visual.blocks.13 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 17 model.visual.blocks.14 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 18 model.visual.blocks.15 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 19 model.visual.blocks.16 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 20 model.visual.blocks.17 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 21 model.visual.blocks.18 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 22 model.visual.blocks.19 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 23 model.visual.blocks.20 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 24 model.visual.blocks.21 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 25 model.visual.blocks.22 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 26 model.visual.blocks.23 Qwen3_5VisionBlock in=(256, 1024) out=(256, 1024) 27 model.visual.merger Qwen3_5VisionPatchMerger in=(256, 1024) out=(64, 2048) 28 Qwen3_5Model.forward image-token insertion masked_scatter / replace image placeholders in=text_embeds (1, 80, 2048) + visual_embeds (64, 2048) out=(1, 80, 2048) 29 model.language_model.layers.0 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 30 model.language_model.layers.1 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 31 model.language_model.layers.2 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 32 model.language_model.layers.3 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 33 model.language_model.layers.4 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 34 model.language_model.layers.5 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 35 model.language_model.layers.6 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 36 model.language_model.layers.7 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 37 model.language_model.layers.8 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 38 model.language_model.layers.9 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 39 model.language_model.layers.10 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 40 model.language_model.layers.11 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 41 model.language_model.layers.12 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 42 model.language_model.layers.13 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 43 model.language_model.layers.14 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 44 model.language_model.layers.15 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 45 model.language_model.layers.16 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 46 model.language_model.layers.17 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 47 model.language_model.layers.18 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 48 model.language_model.layers.19 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 49 model.language_model.layers.20 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 50 model.language_model.layers.21 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 51 model.language_model.layers.22 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 52 model.language_model.layers.23 Qwen3_5DecoderLayer in=(1, 80, 2048) out=(1, 80, 2048) 53 model.language_model.norm Qwen3_5RMSNorm in=(1, 80, 2048) out=(1, 80, 2048) 54 lm_head Linear in=(1, 80, 2048) out=(1, 80, 248320) final logits: (1, 80, 248320) Important: the first language layer input is (1, 80, 2048) because the row immediately before it inserted 64 visual embeddings into 64 image-placeholder positions inside the 80-position sequence. The next cell verifies that replacement directly.
5. Merge proof: 64 visual tokens replace 64 image-placeholder positions inside the 80-position sequence¶
In [7]:
merge_capture = {}
def capture_visual_output(module, args, output):
pooler = output.pooler_output
if isinstance(pooler, (list, tuple)):
pooler = torch.cat(pooler, dim=0)
merge_capture['visual_embeds_after_merger'] = pooler.detach()
def capture_language_input(module, args, kwargs):
merge_capture['inputs_embeds_entering_language_model'] = kwargs['inputs_embeds'].detach()
visual_handle = model.model.visual.register_forward_hook(capture_visual_output)
language_pre_handle = model.model.language_model.register_forward_pre_hook(capture_language_input, with_kwargs=True)
with torch.inference_mode():
_ = model(**batch, use_cache=False)
visual_handle.remove()
language_pre_handle.remove()
input_ids = batch['input_ids']
image_pad_id = processor.tokenizer.convert_tokens_to_ids('<|image_pad|>')
image_positions = (input_ids[0] == image_pad_id).nonzero(as_tuple=False).flatten()
non_image_positions = (input_ids[0] != image_pad_id).nonzero(as_tuple=False).flatten()
original_token_embeds = model.model.language_model.embed_tokens(input_ids).detach()
visual_embeds = merge_capture['visual_embeds_after_merger']
combined_embeds = merge_capture['inputs_embeds_entering_language_model']
combined_image_slice = combined_embeds[0, image_positions]
original_image_placeholder_slice = original_token_embeds[0, image_positions]
combined_non_image_slice = combined_embeds[0, non_image_positions]
original_non_image_slice = original_token_embeds[0, non_image_positions]
print('total sequence positions:', input_ids.shape[1])
print('image placeholder token id:', image_pad_id)
print('number of image placeholder positions:', int(image_positions.numel()))
print('number of non-image positions:', int(non_image_positions.numel()))
print('first 12 image placeholder positions:', image_positions[:12].detach().cpu().tolist())
print('original token embeddings:', tuple(original_token_embeds.shape))
print('visual embeddings after model.visual merger:', tuple(visual_embeds.shape))
print('combined inputs_embeds entering model.language_model:', tuple(combined_embeds.shape))
print('visual_embeds shape equals combined image-position slice:', tuple(visual_embeds.shape) == tuple(combined_image_slice.shape))
print('combined image positions equal visual embeddings:', torch.allclose(combined_image_slice, visual_embeds, atol=0, rtol=0))
print('combined image positions still equal original placeholder embeddings:', torch.allclose(combined_image_slice, original_image_placeholder_slice, atol=0, rtol=0))
print('combined non-image positions equal original token embeddings:', torch.allclose(combined_non_image_slice, original_non_image_slice, atol=0, rtol=0))
total sequence positions: 80 image placeholder token id: 248056 number of image placeholder positions: 64 number of non-image positions: 16 first 12 image placeholder positions: [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] original token embeddings: (1, 80, 2048) visual embeddings after model.visual merger: (64, 2048) combined inputs_embeds entering model.language_model: (1, 80, 2048) visual_embeds shape equals combined image-position slice: True combined image positions equal visual embeddings: True combined image positions still equal original placeholder embeddings: False combined non-image positions equal original token embeddings: True
6. Graphviz graph generated from the block-level hook trace, with explicit image/text branches¶
In [8]:
from graphviz import Digraph
dot = Digraph('qwen35_two_branch_flow')
dot.attr(rankdir='LR', concentrate='true')
dot.attr('graph', splines='ortho', nodesep='0.45', ranksep='0.65')
dot.attr('node', shape='box', style='rounded,filled', fontname='Helvetica', fontsize='10')
rows = {row['name']: row for row in block_trace}
vision_blocks = [row for row in block_trace if row['name'].startswith('model.visual.blocks.')]
language_layers = [row for row in block_trace if row['name'].startswith('model.language_model.layers.')]
patch = rows['model.visual.patch_embed']
merger = rows['model.visual.merger']
embed = rows['model.language_model.embed_tokens']
norm = rows['model.language_model.norm']
head = rows['lm_head']
def node(name, label, color):
dot.node(name, label=label, fillcolor=color)
node('image_input', f"image input\npixel_values {tuple(batch['pixel_values'].shape)}\nimage_grid_thw {tuple(batch['image_grid_thw'].shape)}", '#d8ecff')
node('text_input', f"token sequence\ninput_ids {tuple(batch['input_ids'].shape)}\n{int(image_positions.numel())} image placeholders + {int(non_image_positions.numel())} other tokens", '#ffe7cf')
node('patch', f"visual patch_embed\n{patch['input_shape']} -> {patch['output_shape']}", '#e8f3ff')
node('vblocks', f"24 x Qwen3_5VisionBlock\n{vision_blocks[0]['input_shape']} -> {vision_blocks[-1]['output_shape']}", '#e8f3ff')
node('merger', f"visual merger/projector\n{merger['input_shape']} -> {merger['output_shape']}\n{int(image_positions.numel())} visual tokens, hidden 2048", '#e8f3ff')
node('embed', f"text embed_tokens\n{embed['input_shape']} -> {embed['output_shape']}", '#fff1e6')
node('insert', f"merge in Qwen3_5Model.forward\n{int(image_positions.numel())} visual embeddings replace {int(image_positions.numel())} image-placeholder positions\ncombined inputs_embeds {tuple(combined_embeds.shape)}", '#f5f0ff')
node('layers', f"24 x Qwen3_5DecoderLayer\n{language_layers[0]['input_shape']} -> {language_layers[-1]['output_shape']}", '#fff1e6')
node('norm', f"language norm\n{norm['input_shape']} -> {norm['output_shape']}", '#fff1e6')
node('head', f"lm_head\n{head['input_shape']} -> {head['output_shape']}", '#eef8ed')
dot.edge('image_input', 'patch')
dot.edge('patch', 'vblocks')
dot.edge('vblocks', 'merger')
dot.edge('merger', 'insert', label='image_embeds')
dot.edge('text_input', 'embed')
dot.edge('embed', 'insert', label='text inputs_embeds')
dot.edge('insert', 'layers')
dot.edge('layers', 'norm')
dot.edge('norm', 'head')
display(dot)
Warning: Orthogonal edges do not currently handle edge labels. Try using xlabels.
7. The loaded model's own forward code agrees with the runtime trace¶
In [9]:
source_lines = inspect.getsource(model.model.__class__.forward).splitlines()
for line_number, line in enumerate(source_lines, start=1):
if (
'pixel_values' in line
or 'image_grid_thw' in line
or 'inputs_embeds' in line
or 'image_embeds' in line
or 'masked_scatter' in line
or 'language_model' in line
):
print(f'{line_number:03d}: {line}')
009: inputs_embeds: torch.FloatTensor | None = None,
010: pixel_values: torch.Tensor | None = None,
011: pixel_values_videos: torch.FloatTensor | None = None,
012: image_grid_thw: torch.LongTensor | None = None,
018: image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*):
023: if (input_ids is None) ^ (inputs_embeds is not None):
024: raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
026: if inputs_embeds is None:
027: inputs_embeds = self.get_input_embeddings()(input_ids)
029: if pixel_values is not None:
031: pixel_values, image_grid_thw, return_dict=True, **kwargs
033: image_embeds = image_outputs.pooler_output
034: image_embeds = torch.cat(image_embeds, dim=0).to(inputs_embeds.device, inputs_embeds.dtype)
036: input_ids, inputs_embeds=inputs_embeds, image_features=image_embeds
038: inputs_embeds = inputs_embeds.masked_scatter(image_mask, image_embeds)
040: if pixel_values_videos is not None:
042: pixel_values_videos, video_grid_thw, return_dict=True, **kwargs
045: video_embeds = torch.cat(video_embeds, dim=0).to(inputs_embeds.device, inputs_embeds.dtype)
047: input_ids, inputs_embeds=inputs_embeds, video_features=video_embeds
049: inputs_embeds = inputs_embeds.masked_scatter(video_mask, video_embeds)
054: image_grid_thw=image_grid_thw,
056: inputs_embeds=inputs_embeds,
062: outputs = self.language_model(
067: inputs_embeds=inputs_embeds,
8. Hidden-state proof: vision channel and LLM channel return different hidden tensors¶
In [10]:
captured_outputs = {}
def save_module_output(module_name):
def hook(module, args, output):
captured_outputs[module_name] = output
return hook
visual_handle = model.model.visual.register_forward_hook(save_module_output('model.visual'))
language_handle = model.model.language_model.register_forward_hook(save_module_output('model.language_model'))
with torch.inference_mode():
final_output = model(
**batch,
use_cache=False,
output_hidden_states=True,
return_dict=True,
)
visual_handle.remove()
language_handle.remove()
vision_output = captured_outputs['model.visual']
language_output = captured_outputs['model.language_model']
print('model.visual output object:', type(vision_output))
print('model.visual last_hidden_state:', tuple(vision_output.last_hidden_state.shape), vision_output.last_hidden_state.dtype, vision_output.last_hidden_state.device)
print('model.visual hidden_states[-1]:', tuple(vision_output.hidden_states[-1].shape), vision_output.hidden_states[-1].dtype, vision_output.hidden_states[-1].device)
print('model.visual pooler_output[0]:', tuple(vision_output.pooler_output[0].shape), vision_output.pooler_output[0].dtype, vision_output.pooler_output[0].device)
print('\nmodel.language_model output object:', type(language_output))
print('model.language_model last_hidden_state:', tuple(language_output.last_hidden_state.shape), language_output.last_hidden_state.dtype, language_output.last_hidden_state.device)
print('model.language_model hidden_states[-1]:', tuple(language_output.hidden_states[-1].shape), language_output.hidden_states[-1].dtype, language_output.hidden_states[-1].device)
print('\nfinal model logits:', tuple(final_output.logits.shape), final_output.logits.dtype, final_output.logits.device)
print('final model hidden_states[-1]:', tuple(final_output.hidden_states[-1].shape), final_output.hidden_states[-1].dtype, final_output.hidden_states[-1].device)
print('\nConclusion from this forward pass:')
print('vision channel hidden size:', vision_output.last_hidden_state.shape[-1])
print('LLM channel hidden size:', language_output.last_hidden_state.shape[-1])
print('vision hidden tensor is language hidden tensor:', vision_output.last_hidden_state is language_output.last_hidden_state)
model.visual output object: <class 'transformers.modeling_outputs.BaseModelOutputWithPooling'> model.visual last_hidden_state: (256, 1024) torch.bfloat16 cuda:0 model.visual hidden_states[-1]: (256, 1024) torch.bfloat16 cuda:0 model.visual pooler_output[0]: (64, 2048) torch.bfloat16 cuda:0 model.language_model output object: <class 'transformers.models.qwen3_5.modeling_qwen3_5.Qwen3_5ModelOutputWithPast'> model.language_model last_hidden_state: (1, 80, 2048) torch.bfloat16 cuda:0 model.language_model hidden_states[-1]: (1, 80, 2048) torch.bfloat16 cuda:0 final model logits: (1, 80, 248320) torch.bfloat16 cuda:0 final model hidden_states[-1]: (1, 80, 2048) torch.bfloat16 cuda:0 Conclusion from this forward pass: vision channel hidden size: 1024 LLM channel hidden size: 2048 vision hidden tensor is language hidden tensor: False
9. Qwen/Qwen3.5-27B has the same Qwen3.5 multimodal architecture family¶
In [11]:
model_id_27b = 'Qwen/Qwen3.5-27B'
config_27b = AutoConfig.from_pretrained(model_id_27b, trust_remote_code=True)
processor_27b = AutoProcessor.from_pretrained(model_id_27b, trust_remote_code=True)
print('HF repo:', model_id_27b)
print('config class:', config_27b.__class__.__module__ + '.' + config_27b.__class__.__name__)
print('config model_type:', config_27b.model_type)
print('config architectures:', config_27b.architectures)
print('text_config:', config_27b.text_config.model_type, 'hidden_size=', config_27b.text_config.hidden_size, 'layers=', config_27b.text_config.num_hidden_layers)
print('vision_config:', config_27b.vision_config.model_type, 'hidden_size=', config_27b.vision_config.hidden_size, 'depth=', config_27b.vision_config.depth, 'out_hidden_size=', config_27b.vision_config.out_hidden_size)
print('processor class:', processor_27b.__class__.__module__ + '.' + processor_27b.__class__.__name__)
print('Full 27B checkpoint load command, if you want to execute it on a large enough GPU node:')
print("model_27b = AutoModelForMultimodalLM.from_pretrained('Qwen/Qwen3.5-27B', dtype=torch.bfloat16, device_map='auto', trust_remote_code=True)")
HF repo: Qwen/Qwen3.5-27B
config class: transformers.models.qwen3_5.configuration_qwen3_5.Qwen3_5Config
config model_type: qwen3_5
config architectures: ['Qwen3_5ForConditionalGeneration']
text_config: qwen3_5_text hidden_size= 5120 layers= 64
vision_config: qwen3_5_vision hidden_size= 1152 depth= 27 out_hidden_size= 5120
processor class: transformers.models.qwen3_vl.processing_qwen3_vl.Qwen3VLProcessor
Full 27B checkpoint load command, if you want to execute it on a large enough GPU node:
model_27b = AutoModelForMultimodalLM.from_pretrained('Qwen/Qwen3.5-27B', dtype=torch.bfloat16, device_map='auto', trust_remote_code=True)
In [12]:
# Free the fully loaded 2B checkpoint when running interactively.
del model
torch.cuda.empty_cache()
print('freed 2B model from GPU memory')
freed 2B model from GPU memory