A private LLM API
in your own AWS account.
Self-host an open-source LLM like Qwen, Llama or Mistral with a RAG chatbot API. Pick a model, connect AWS, and forgestack generates a CloudFormation template you can read and keep, then applies it under your own IAM role. The model, your documents, the vector index and chat logs are stored in your account. Free to prototype — you pay AWS directly.
- open models
- 20
- AWS regions
- 30
- CPU & GPU sizes
- 8
- forgestack fee to prototype
- $0
See it run
One real run, start to finish
Sign up, pick a model, connect AWS, apply the template, upload a document and ask about it. Recorded against LocalStack with the same deploy code that targets real AWS; on a real account the model node takes a few minutes to boot.
The pipeline
From model pick to live API, without an ML platform team
Pick a model & region
Choose an open-weight model from the catalog, and where it runs — 30 AWS regions, sized to the model automatically.
- Open-weight, Hugging Face-hosted catalog
- 30 AWS regions, instance sized automatically
- Swap models with a stack update — same API
Connect your cloud
Pick whichever authentication your account supports — SSO, a cross-account role, access keys, or a local profile. SSO and the role give forgestack short-lived STS sessions only.
- 4 auth paths — SSO to plain access keys
- sts:AssumeRole verified live, not assumed
- Access keys are for testing — stored encrypted, and we say so
We forge the template
Your config becomes a real, prefilled CloudFormation template — inference host, API gateway, knowledge base. Downloadable, owned by you.
- Real CloudFormation, fully prefilled & readable
- VPC, IAM role, API gateway all defined
- Downloadable — the file is yours, not ours
Apply — we provision
Under your assumed role, we stand up a VPC, a dedicated inference node, an encrypted S3 knowledge base and an API gateway, then warm the model — with live per-resource progress.
- VPC + subnets stood up under your role
- GPU/CPU node launched, model weights pulled
- Encrypted S3 knowledge base + API gateway live
Live: chat, cite, fine-tune
Grounded chat with source citations, a documents API backed by your own bucket, and LoRA fine-tuning on your own examples — all in your account.
- Grounded chat, source citations on every reply
- Documents API — reads and writes your own bucket
- LoRA fine-tune on your own question/answer pairs
What we deploy for you
A real stack, not a chatbot wrapper
Every stack is a self-contained architecture in your account: your documents live in your S3 bucket, your model runs on a dedicated EC2 node in the region you chose, and an API gateway exposes it behind a scoped key.
Stored in your account
Weights, documents, vectors and chat logs live in your bucket and on your node. Nothing is shared across tenants.
Real cloud deployment
AWS role-assumption and stack apply happen against your account via STS + CloudFormation — auditable end to end.
Template you own
Every deployment is a downloadable, deployable template. Version it, audit it, run it yourself.
What runs where
The honest answer for your security review
Storage and compute are in your account. Requests to the public API pass through forgestack on their way there. Here is exactly what sits where.
Stored in your AWS account
- Model weights and the inference node (EC2)
- Knowledge-base documents and the vector index (S3 + node)
- Conversation logs, with retention set by you (S3 lifecycle)
- The API Gateway, Lambda router and IAM roles
Passes through forgestack, not stored
- Chat and document requests, which forgestack checks against your API key and forwards to your stack
- Playground messages, proxied to your node
Stored by forgestack
- Your account, project config and the generated template
- Document names, sizes and chunk counts (not their contents)
- Saved cloud credentials, encrypted (SSO and role use short-lived STS sessions)
- Your API keys and their scopes
Model catalog
Pick a model, it runs in your account
Open-weight models from the Hugging Face Hub, served by llama.cpp from a Q4_K_M GGUF (the same engine and format local tools like Ollama use) on an instance sized to fit. Swap models with a stack update — the API stays the same. Compare models and AWS cost.
Qwen/Qwen2.5-0.5B-Instruct
Best answers that still fit a free t3.micro. Chat + RAG, no GPU.
HuggingFaceTB/SmolLM2-360M-Instruct
Half the weights of Qwen 0.5B. Fastest replies on a 2 vCPU box.
HuggingFaceTB/SmolLM2-135M-Instruct
Smallest that still holds a conversation. Good for narrow RAG bots.
TinyLlama/TinyLlama-1.1B-Chat-v1.0
Featherweight. Runs on CPU-only nodes, sub-second replies.
microsoft/phi-2
Microsoft's compact reasoning model. Great for FAQ + support bots.
Qwen/Qwen2.5-3B-Instruct
Long context in a small footprint. Strong multilingual support.
HuggingFaceH4/zephyr-7b-beta
DPO-aligned chat model with a natural conversational tone.
mistralai/Mistral-7B-Instruct-v0.2
The general-purpose default — long documents, multi-turn support, RAG answers.
HuggingFaceTB/SmolLM2-1.7B-Instruct
Tiny but instruction-tuned properly. Good CPU-node default for short replies.
microsoft/Phi-3-mini-4k-instruct
Punches far above its size on reasoning and instruction following.
Qwen/Qwen2.5-7B-Instruct
Strongest all-rounder here — coding, maths, 29 languages, 32K context.
Qwen/Qwen2.5-1.5B-Instruct
The step up from 0.5B — noticeably better answers, still CPU-friendly.
Qwen/Qwen2.5-Coder-1.5B-Instruct
Small code assistant — snippets, explanations and docs Q&A on a CPU node.
ibm-granite/granite-3.1-2b-instruct
IBM's enterprise-tuned small model. Long context, built for RAG and tools.
microsoft/Phi-3.5-mini-instruct
Phi-3 Mini with a 128K window — reads long documents whole.
microsoft/Phi-4-mini-instruct
Newest Phi. Stronger maths and multilingual answers at the same size.
allenai/OLMo-2-1124-7B-Instruct
Fully open — weights, data and training code. For auditable deployments.
Qwen/Qwen2.5-Coder-7B-Instruct
Best coding model here — generation, review and repo-docs Q&A.
NousResearch/Hermes-3-Llama-3.1-8B
Ungated Llama 3.1 fine-tune. Steerable system prompts, strong tool use.
ibm-granite/granite-3.1-8b-instruct
Largest in the catalogue. Enterprise RAG over long documents, on one GPU.
Need a specific open-weight model? Ask and we'll add it to the catalog.
Access control
Keys scoped to exactly what they can do
Issue a key per environment, per team, or per vendor — and pick which of the three endpoints it can call. Narrow the scopes below to see the key that comes out the other side.
Revoke or re-scope any key instantly — nothing else changes.
The solution API
One key. Three endpoints.
Infinite integrations.
The moment your stack goes live, every endpoint answers, backed by the model and data in your account. Feed the knowledge base and kick off fine-tunes through the dashboard or the API, and read the full reference anytime at /docs.
- Per-stack API keys
Issue and revoke keys per environment, per team, per vendor.
- OpenAI-shaped responses
Replies come back as chat.completion objects, so your parsing code carries over.
- Grounded answers
Retrieval over your knowledge base with source citations on every completion.
- Documents endpoint
Add, list and remove knowledge-base documents — stored in your own S3 bucket.
- Fine-tune on your data
Kick off a real LoRA fine-tune from your own question/answer pairs, on your own node.
curl -X POST https://your-forgestack.app/api/{projectId}/chat \
-H "x-api-key: fsk_9f2c..." \
-H "content-type: application/json" \
-d '{"messages":[{"role":"user","content":"What is your refund policy?"}]}'Chatbot integrations
A RAG chatbot backend, with a human on call
Call the chat API from your website or app and get answers grounded in your documents. When the model can't answer, the visitor can hand the conversation to your team in the tool they already use.
See chatbot integrationsHuman handoff destinations. Knowledge sources: uploads, GitHub, Confluence and Jira.
Integrations
Built on infrastructure you already trust
AWS
EC2, S3, Lambda, API Gateway, IAM — real CloudFormation under your role.
Hugging Face Hub
Open-weight model weights pulled straight from the Hub at deploy time.
CloudFormation
Every stack is a real, downloadable, re-runnable template you own.
GCP
Compute Engine + GCS — emulated locally via fake-gcs-server (port 4443)
FAQ
Straight answers
Can I run a local LLM, like an Ollama model, in the cloud?
Yes. forgestack serves models with llama.cpp from GGUF files, the same engine and format local tools like Ollama use. Pick one from the catalog or any GGUF on the Hugging Face Hub, and it runs on an EC2 node in your AWS account with an API key, logs and RAG.
Can I add it to my website as a chatbot?
Yes, through the chat API: your frontend or backend sends the conversation and gets back a grounded answer with citations. Unanswered questions can be handed to Slack, Teams, Zendesk, Freshdesk, Intercom, email or a webhook. There is no drop-in website widget yet.
Does my data go through forgestack?
Requests do; storage doesn't. API and playground calls pass through the forgestack control plane, which checks the key and forwards them to your stack. The model, documents, vectors and conversation logs are stored only in your AWS account.
What does it cost?
forgestack is free to prototype, with no markup on AWS. You pay AWS directly for what the stack runs: for example a t3.medium node is about $30/month and a g5.xlarge GPU node about $1.01/hour, plus small charges for VPC endpoints, S3, Lambda and API Gateway. The Model screen shows a live estimate before you deploy.
Is there an AWS free tier?
The smallest models fit a t3.micro, which AWS's legacy free tier covers on accounts created before 15 July 2025. Newer accounts get AWS credits instead, so check your account before assuming $0.
Why not just use Amazon Bedrock?
Bedrock is a good default if its catalog fits. forgestack is for when you want an open-weight model from the Hugging Face Hub on your own EC2 node, with RAG, LoRA fine-tuning and logs bundled, and a CloudFormation template you own.
What happens if forgestack goes away?
Your stack and everything stored in it stay in your account, and the template is a normal CloudFormation file you can download and re-run yourself. Calling the API without forgestack in the path isn't a supported mode yet; it's on the roadmap.
Which AWS permissions does it need?
A deploy policy scoped to the resource types in the template (EC2 and VPC, S3, Lambda, API Gateway, IAM, SSM, CloudWatch Logs, CloudFormation). Use IAM Identity Center or a cross-account role with an External ID; access keys work but are meant for testing. The exact policy is on the docs page.
Is forgestack SOC 2 certified?
No. It's an early product. What we can offer instead is a short data-flow answer for your security review: what's stored in your account, what passes through us, and what we keep.
Stop wiring chatbots.
Start forging stacks.
Pick a model and a region, connect AWS, and get a grounded chat API plus the CloudFormation template that built it — running in your own account. Free to prototype; you pay AWS directly.