Phi-3.5-MoE LLM
(Redirected from Phi-3.5 MoE)
Jump to navigation
Jump to search
A Phi-3.5-MoE LLM is a Microsoft medium-sized language model (MedLM)
- AKA: Phi-3.5 MoE, Microsoft Phi-3.5 MoE.
- Context:
- It can activate Parameter Subset with only 6.6B parameters from its total 42B parameter architecture.
- It can leverage Mixture Of Expert architecture through 16 expert modules.
- It can deliver Performance Levels comparable to gemini 1.5 flash model with fewer active parameters.
- It can outperform competing models like llama 3.1 8B, gemma 2 9B, and mistral nemo 12B in standard benchmarks.
- It can utilize GRIN Training Method for improved parameter efficiency and expert specialization.
- It can operate through serverless deployment on azure AI studio and github platforms.
- It can process input tokens at 0.00013 USD per 1K tokens (as of 2024).
- It can generate output tokens at 0.00052 USD per 1K tokens (as of 2024).
- It can range from being a Basic Query Processor to being an Advanced Task Solver, depending on its application context.
- It can range from being a Single Domain Expert to being a Multi Domain Specialist, based on its expert module activation.
- ...
- Examples:
- Deployment Platforms, such as:
- Cloud Platforms, such as:
- Usage Scenarios, such as:
- Regional Deployments, such as:
- Azure Regions, such as:
- ...
- Deployment Platforms, such as:
- Counter-Examples:
- Traditional LLM Architecture, which lacks mixture of experts capability.
- Dense Parameter Model, which requires full parameter activation.
- Single Expert System, which cannot leverage specialized expert modules.
- See: Microsoft Phi Family, Azure AI Platform, Language Model Architecture, Mixture Of Experts Technology, Model Deployment Strategy.