The models behind Forge

01 Executive summary

Forge is building its own frontier-class model series rather than renting one. The models are trained from scratch on a corpus we control, and they are the engine of a product that already ships on desktop, web, VS Code, Android, and Chrome.

Forge 5.2 Flash is a 312-billion-parameter sparse mixture-of-experts model with 12 billion active parameters per token — the workhorse behind everyday agentic coding. Forge 5.3 is a 414-billion-parameter model with 14 billion active parameters, built on the same architecture and trained on a larger, richer corpus for the hardest reasoning and long-horizon tasks.

Both models are proprietary and closed-weights. The architecture, training pipeline, and data strategy are owned by SoftUp Agency and are not disclosed in full — this paper shares the design philosophy and the evidence of capability, not the recipe.

02 The company

SoftUp Agency is a US company with an engineering team based in Syria. It is fully self-funded — the model program is paid for by the company's own revenue, not by external capital.

That independence shapes how we build. Because we are not chasing a funding round to survive, we can make long-horizon technical bets: training our own weights from scratch, building our own data pipeline, and investing in the infrastructure to do it — rather than assembling a thin wrapper around someone else's API. The model series is the deepest expression of that strategy.

03 Design goals

The series was designed around one job: reliable, long-horizon agentic work on real codebases. Every architectural choice traces back to that.

• Agentic reliability over raw fluency
  The models are trained to plan, call tools, read and edit files, verify their own work, and recover from failure — not just to produce plausible text.
• Long context as a first-class citizen
  Real codebases are large. The architecture is built to hold and reason over long, structured context — a whole repository, not a snippet.
• Compute efficiency per token
  A sparse mixture-of-experts design activates only a fraction of parameters per token, so a very large model stays affordable to serve and to train.
• Multilingual and code-native
  The tokenizer and corpus are built for code, English, and Arabic — the languages the product and its users actually work in.
• Owned end to end
  Tokenizer, architecture, training loop, data pipeline, and evaluation are all ours. Nothing in the critical path is licensed from a competitor.

04 Architecture

Both models share a single decoder-only, sparse mixture-of-experts architecture. One architecture, two scales — so improvements proven on one transfer directly to the other.

The design follows the modern frontier-model playbook: a sparse mixture of experts so that capacity scales with total parameters while per-token compute stays bounded, grouped-query attention to keep the key-value cache small at long context, and a byte-level tokenizer trained on our own corpus so that code, English, and Arabic are all represented efficiently. The specific layer counts, expert counts, and training hyper-parameters are proprietary and intentionally not published.

05 Training

The models are trained from scratch in a staged curriculum — the same discipline used across the frontier labs, adapted to our own pipeline.

• Pretraining
  Massive next-token prediction over raw code and text teaches language, code structure, and world knowledge.
• Continued pretraining (code-heavy)
  A focused pass over code and agentic traces sharpens the model for the domain it will actually work in.
• Supervised fine-tuning on agentic trajectories
  Conversations, tool calls, and edit sequences teach the model to plan and act — not just to generate.
• Final low-rate fine-tune
  A small, high-quality set of real product interactions is applied gently at the end, shaping style and behaviour without erasing earlier skills.

Training runs on our own GPU infrastructure, with a pipeline built in-house for data ingestion, decontamination, and reproducible checkpoints. Every stage is gated on evaluation before the next begins.

06 Data

Data is the moat. The corpus is roughly half curated open-source data and half data collected by our own product — and the proprietary half is what compounds.

For Forge 5.2 Flash, the training corpus is a deliberate 50 / 50 blend:

Forge 5.3 is trained on the same foundation plus additional proprietary data collected by us — the larger model is fed by a larger, richer slice of our own flywheel.

All open-source components are used under their respective licences, and the pipeline records provenance and licence for every source. Details of the proprietary corpus are not disclosed.

07 Evaluation

We evaluate the models the way they are used: on real agentic tasks, not just static benchmarks.

• Agentic task completion
  End-to-end tasks on real repositories: plan, edit, verify, and finish without human intervention.
• Long-context retrieval
  Needle-in-a-haystack and repository-scale retrieval across long contexts.
• Code correctness
  Generated code is executed and tested, not just matched against a reference string.
• Tool-use reliability
  Correct tool selection, well-formed calls, and graceful recovery from errors.

Scores are measured on our internal evaluation harness across three axes — agentic task completion, code correctness, and general intelligence. Config refers to the reasoning-effort setting used for each model. Forge results are highlighted in bold.

08 Safety & responsibility

A model that can edit code and drive tools is a powerful one. We treat that seriously.

• Human-in-the-loop by design
  The product routes edits and commands through an approval queue. The model proposes; the user disposes.
• Provenance and licence tracking
  Every training source is recorded with its licence and origin, and the pipeline refuses sources that are not allowed.
• Decontamination
  The corpus is decontaminated against evaluation sets so benchmark scores reflect real capability, not memorization.
• Responsible release
  Weights are closed and served through the product, keeping the model under our control and out of misuse.

09 Roadmap

• Longer context
  Extending the usable context window further, toward whole-repository and multi-repository reasoning.
• Larger proprietary corpus
  Scaling the product-derived data flywheel as the installed base grows.
• Next model generation
  The architecture is designed to scale; the next generation builds directly on the same spine.
• Expanding published benchmarks
  Adding more axes and independent verification to the evaluation results published in this paper.