Model on AWS
Deploy Granite 3.1 8B Instruct on AWS
Largest in the catalogue. Enterprise RAG over long documents, on one GPU. ForgeStack runs ibm-granite/granite-3.1-8b-instruct as a private LLM API in your own AWS account: an g5.xlarge node serving it with llama.cpp, a knowledge base in your S3 bucket, and a chat endpoint with citations.
- Parameters
- 8.2B
- Context
- 128K
- License
- Apache-2.0
- Speed
- Fast
- Instance
- g5.xlarge
- Hardware
- GPU
- AWS / month
- ≈ $750
- Compute / hour
- $1.0060
What you get when you deploy Granite 3.1 8B Instruct
- Model node: a dedicated g5.xlarge (GPU) in the region you pick, serving granite-3.1-8b-instruct-Q4_K_M.gguf from bartowski/granite-3.1-8b-instruct-GGUF with llama.cpp.
- Chat API: POST /api/{projectId}/chat with a scoped x-api-key, returning chat.completion-shaped responses with source citations.
- Knowledge base (RAG): upload documents or sync GitHub, Confluence and Jira; they are stored and indexed inside your stack.
- LoRA fine-tuning on your own question/answer pairs, on the same node.
- Human handoff to Slack, Teams, Zendesk, Freshdesk, Intercom, email or a webhook when the model can’t answer. See chatbot integrations.
How to deploy it
- Create a ForgeStack account and pick Granite 3.1 8B Instruct from the catalog.
- Connect AWS with IAM Identity Center or a cross-account role. ForgeStack only gets a short-lived STS session.
- Set a system prompt and turn on retrieval.
- Review and download the generated CloudFormation template.
- Apply it. The stack goes live once the model node passes its health check.
What it costs
About $750 a month in us-east-1 for an always-on node, paid to AWS directly: the g5.xlarge instance, its root volume and one VPC endpoint. ForgeStack itself is free to prototype and adds no markup. Delete the stack and the charges stop.
Other GPU models
- Zephyr 7B β on g5.xlarge, ≈ $750/month
- Mistral 7B Instruct v0.2 on g5.xlarge, ≈ $750/month
- Phi-3 Mini 4K Instruct on g4dn.xlarge, ≈ $399/month
- Qwen 2.5 7B Instruct on g5.xlarge, ≈ $750/month
Deploy it into your own AWS account
Free to prototype. You pay AWS directly, and the CloudFormation template is yours to keep.