GPT-OSS Vision – Multimodal AI Pipeline with Q-Former

GPT-OSS Vision was an ambitious research experiment attempting to bridge text-based Large Language Models with visual understanding. The project aimed to enable a 20B parameter text-only LLM (GPT-OSS) to understand and analyze satellite imagery through a novel Q-Former adapter architecture. While the project did not achieve production-ready results due to significant technical challenges, it represents valuable research in multimodal AI integration and adapter-based learning approaches. The experience directly informed my approach to current projects, emphasizing the importance of trainable models over massive frozen architectures. Key challenges included training instability, Q-Former convergence issues, domain overfitting to satellite imagery, and the fundamental limitations of adapter-only fine-tuning for models above 10B parameters. These learnings are now applied to AUTO-GIT and PRO_CODE, where I prioritize end-to-end optimization and extensive evaluation.

1. Image Input: Satellite imagery (224×224×3) is fed into the Remote CLIP vision encoder. 2. Vision Encoding: Remote CLIP (specialized for satellite imagery) converts the image into 49 patch embeddings of 768 dimensions each. 3. Q-Former Compression: A trainable Q-Former with 32 learnable query tokens performs cross-attention with visual patches, compressing features while preserving semantic information. 4. Adapter Projection: A linear layer with LoRA fine-tuning projects the 768-dimensional Q-Former outputs to 2048 dimensions (GPT-OSS embedding space). 5. Token Integration: Compressed visual tokens are prepended to the text prompt as special "visual tokens" for the language model. 6. Language Generation: GPT-OSS 20B (frozen) generates natural language descriptions, VQA answers, or classifications based on the visual context. 7. Training Loop: Only the Q-Former and adapter layers are trained using cross-entropy loss with gradient accumulation.