Guides

CloudFormation stacks named forgestack-*; IAM roles and instance profiles named fx-* (and only three AWS managed policies may be attached to them); SSM parameters under /forgestack/ plus read access to AWS's public Ubuntu AMI parameters; Lambda functions and S3 buckets named fx-*, including their objects; and, because EC2 and API Gateway cannot be scoped by name, the stack's own VPC, subnet, internet gateway, route table, VPC endpoints, security groups, instance and HTTP API.

Access-key connections: the last command should list every statement, including ReadPublicAmiParameters. NoSuchEntity means the policy was never created.

ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
ARN="arn:aws:iam::$ACCOUNT_ID:policy/forgestack-deploy-policy"
aws iam list-attached-user-policies --user-name forgestack-deploy \
  --query 'AttachedPolicies[].PolicyName'
V=$(aws iam get-policy --policy-arn "$ARN" --query Policy.DefaultVersionId --output text)
aws iam get-policy-version --policy-arn "$ARN" --version-id "$V" \
  --query 'PolicyVersion.Document.Statement[].Sid'

When the template gains a resource type, the policy gains the actions for it, and an older policy fails with "not authorized to perform". Access keys: paste the "Grant the deploy permissions" step again; it publishes a new version in place. Role ARN: update the bootstrap stack to the current template, as below. Your External ID and the forgestack-deployer role's ARN stay the same.

aws cloudformation update-stack --stack-name forgestack-bootstrap \
  --template-url "https://forgestack-mvp.vercel.app/api/aws/bootstrap-template" \
  --parameters ParameterKey=ExternalId,UsePreviousValue=true \
  --capabilities CAPABILITY_NAMED_IAM

The provisioning log shows CloudFormation's first failure — the resource and the reason — not the generic "resource(s) failed to create" line. The Infrastructure panel lists every resource in the template as Pending, Creating, Ready or Failed, so you can see how far the stack got.

Once CloudFormation finishes, the EC2 instance reports each boot stage itself — installing Docker, adding swap, fetching the runtime, building the image, starting the container, downloading and loading the model — as "Instance:" rows, with the download percentage. The log panel's Instance tab shows the last lines of its boot script and inference container, refreshed every 30 seconds. A deploy only fails here when the instance reports a failure, the model fails to load, or the instance goes silent for 12 minutes — not because it is slow.

# The same status and log, from CloudShell:
aws ssm get-parameter --name /forgestack/<your-stack-slug>/boot-status \
  --query Parameter.Value --output text
aws s3 cp s3://fx-<your-stack-slug>-kb/boot/boot.log - | tail -n 50

The policy is missing an action, almost always because it predates a template change. Update it as described under Deploy permissions, then apply again.

A failed apply does not roll back: everything that was already created is kept, and the next apply continues from there, building only what failed or never started. If the stack built but the model failed to start, only the instance is replaced. The kept resources stay in your account (and keep billing — the VPC endpoint and any instance) until you apply again or delete the stack. This prints CREATE_FAILED while a failed apply is waiting to be resumed.

aws cloudformation describe-stacks --stack-name forgestack-<your-stack-slug> \
  --query 'Stacks[0].StackStatus' --output text

Documents are not stored in ForgeStack. They are written to your own stack's S3 bucket (fx-<slug>-kb) and to the running node's index; ForgeStack keeps only the name, byte size and chunk count. So a document can only be added once there is a stack to receive it — before that, POST /documents answers 409 deployment_state_error, and every key-authenticated route answers 409 while the project status is anything but live.

Playground → Feed the index takes pasted text or any text-based file — .txt, .md, .csv, .json, .html, .xml, .log, source code. The file goes to your stack as-is: a Lambda there extracts its text (HTML and XML reduced to their text, script/style dropped), stores it under kb/ and indexes it. The cap is 2 MB per file. The panel on the right lists what is indexed and the chunk count per document.

Same limits, same pipeline. Send the file as multipart/form-data (a file field, optional name), or JSON with content as plain text — or as the file's bytes base64-encoded with "encoding":"base64". name defaults to the file name (or doc-<timestamp>.txt) and is truncated to 120 characters; keep its extension, which tells the extractor how to read it. Returns 201 for a new document, 200 with replaced:true when a document with that name already existed.

BASE="https://your-forgestack-host"   # where this dashboard runs
PROJECT_ID="your-project-id"           # the id in the dashboard URL
KEY="fsk_…"                            # from Keys & endpoints

curl -X POST "$BASE/api/$PROJECT_ID/documents" \
  -H "x-api-key: $KEY" \
  -H "content-type: application/json" \
  -d '{"name":"refund-policy.md","content":"Refunds are issued within 14 days…"}'

# Or upload a file — HTML/XML markup is stripped on the stack
curl -X POST "$BASE/api/$PROJECT_ID/documents" \
  -H "x-api-key: $KEY" \
  -F "file=@help-center/refunds.html"

ForgeStack runs no connectors — your own job does that, wherever it suits you (a scheduled Lambda, a CI step, a cron script), and pushes text here. Three rules keep a sync clean. (1) One stable name per source item, e.g. jira/ABC-123.md: pushing the same name again replaces that document in place, so re-running a sync never duplicates. (2) Text-based formats (HTML, XML, Markdown, JSON…) can be sent as they are; binary ones (PDF, Word) must be extracted to text in your job — the stack refuses them with 422. Put the title and link at the top so answers can cite where it came from. (3) Over 2 MB, split it and send each part under its own name (guide.p1.md, guide.p2.md). For items deleted upstream, DELETE the id this API returned for them.

// A scheduled Lambda: recently updated Jira issues → the stack's knowledge base.
const issues = await jiraSearch("updated >= -1h"); // your own Jira client
for (const issue of issues) {
  await fetch(BASE + "/api/" + PROJECT_ID + "/documents", {
    method: "POST",
    headers: { "x-api-key": KEY, "content-type": "application/json" },
    body: JSON.stringify({
      name: "jira/" + issue.key + ".md", // stable name: a re-sync replaces it
      content: "# " + issue.key + ": " + issue.summary + "\nSource: " + issue.url + "\n\n" + issue.text,
    }),
  });
}

chunks is how many retrievable pieces the text split into. note is a sentence written for a human. pending:true means the file was stored but text extraction was still running when the stack had to answer; it becomes searchable within a minute.

{
  "id": "…",
  "object": "kb.document",
  "bucket": "fx-<slug>-kb",
  "key": "kb/refund-policy.md",
  "size": 1284,
  "chunks": 3,
  "replaced": false,
  "synced": true,
  "pending": false,
  "note": "Text extracted and indexed in the stack's knowledge base — 3 retrievable chunks."
}

GET returns {object:"list", bucket, documents:[{id, key, size, chunks, created}]}. DELETE takes the id from that list, drops the object from the KB bucket and evicts it from the running node's live index — an S3 delete alone would never reach a chunk already loaded in memory. Returns {ok:true, note}.

curl -H "x-api-key: $KEY" "$BASE/api/$PROJECT_ID/documents"

curl -X DELETE -H "x-api-key: $KEY" \
  "$BASE/api/$PROJECT_ID/documents/<docId>"

400 invalid_request_error — malformed JSON or multipart body, no content or file, or an empty file. 413 invalid_request_error — over the 2 MB limit. 422 invalid_request_error — the stack found no usable text: a binary file (PDF, Office, image), base64 sent without "encoding":"base64", or under 20 characters once markup is stripped. 409 deployment_state_error — no deployed stack. 502 server_error — the stack could not be reached or failed. 404 — no document with that id on this project.

On a real AWS deploy the request goes through your stack's API Gateway router, which PUTs the text at kb/<name> in fx-<slug>-kb and simultaneously pushes it straight to the inference container so retrieval is current immediately, without waiting for a reindex.

Writing under kb/ fires fx-<slug>-kb-reindex (python3.12). It splits the document on blank lines, encodes each chunk with sentence-transformers (all-MiniLM-L6-v2 by default) and writes one artifact per document to indexes/ in the same bucket: {source, chunks, vectors}. Deleting the document deletes its artifact too, so a removed document cannot be resurrected by the next pull.

A background thread on the inference node lists indexes/, loads every artifact and holds the flattened {source, chunk, vector} list in memory. That in-memory list is what retrieval searches — the model itself is never re-trained by an upload.

The last user message is the query. The node scores every chunk against it and takes the top 3 that score above zero. Scoring today is term-overlap against the chunk text — the vectors are stored for the embedding-based scorer the node is built to swap in, and the code says so. Retrieval is skipped entirely when RETRIEVAL_ENABLED is false, which is the Configure screen's retrieval toggle.

The hits are joined as "[<source>] <chunk>" and appended to your configured system prompt under an instruction to answer from that context and to say it does not know when the answer is not in it. The whole thing is then formatted with the model's own chat template. This is the answer to "how does the knowledge base connect to the model": it does not change the weights — it is assembled into the prompt, per request, at answer time.

Every chunk that made it into the prompt is echoed on the response as {document, excerpt} — document being the kb/<name> key, excerpt the first 220 characters. An empty citations array means nothing scored, so the answer came from the model alone.

"citations": [
  { "document": "kb/refund-policy.md", "excerpt": "Refunds are issued within 14 days…" }
]

Retrieval changes what the model knows for one request. A LoRA fine-tune changes how it answers, permanently, for every request. Facts that change — policies, prices, docs — belong in the knowledge base. Tone, format and house phrasing are what training is for. They are independent: a trained adapter still retrieves, and retrieval still works untrained.

Three is the floor, enforced in two places — the dashboard's Start training button stays disabled below three filled rows, and the API answers 400 invalid_request_error. Rows with an empty question or answer are dropped before that count is taken.

epochs and lr are optional and best left out: the node sizes epochs, batch, gradient accumulation and sequence length from how many examples you sent, which beats any fixed default. Training runs in-process on the model node — there is no separate training instance — so chat is degraded while it runs.

curl -X POST "$BASE/api/$PROJECT_ID/train" \
  -H "x-api-key: $KEY" \
  -H "content-type: application/json" \
  -d '{"examples":[
        {"question":"How long do refunds take?","answer":"Up to 14 days."},
        {"question":"Do you ship to the EU?","answer":"Yes, from Rotterdam."},
        {"question":"What is the SLA?","answer":"99.9% monthly uptime."}
      ]}'

A started job answers {"status":"started","examples":3}. A refusal comes back as {"status":"error","error":"…"} — and because the node reports these as HTTP 200, you must check the status field, not just the status code. Refusals include a job already running, a base model still loading, and a preflight that judged this node too small to finish the fine-tune at all.

GET the same URL. status is idle, running, cancelling, cancelled, done or error, alongside epoch, totalEpochs, step, loss, durationS and error. The route answers {"status":"idle"} whenever there is no reachable node, so idle means "nothing running", not "never ran". The dashboard polls this every two seconds while a job is running.

curl -H "x-api-key: $KEY" "$BASE/api/$PROJECT_ID/train"

When training finishes the node rebuilds its serving pipeline on top of the trained adapter in place. The very next call to the chat endpoint is answered by the fine-tuned model — there is no redeploy, no restart and no separate model id to switch to. Cancelling a running job is available from the dashboard; the key-authenticated surface exposes only POST and GET.

Keys & endpoints → name it after the caller ("web-frontend") and press Issue a key. The table masks keys to their last four characters; the eye button reveals one in full and the copy button puts it on your clipboard, both fetched only when you ask so keys are not sitting in every page load. Revoking is instant and takes effect on the next request. Issue as many as you like — one per caller makes it obvious what to revoke when something leaks.

Three independent scopes, ticked at creation: chat (POST /chat), train (POST and GET /train), and documents (POST /documents, DELETE /documents/{id}). A key that has not been given a scope gets 403 permission_error on those routes, naming the scope it lacks. GET /health and GET /documents work with any valid key. Scopes are fixed at creation — to change them, issue a new key and revoke the old one. Keys issued before scopes existed carry all three.

# a chat-only key for a frontend service
curl -X POST "$BASE/api/$PROJECT_ID/train" -H "x-api-key: $CHAT_ONLY_KEY"
# {"error":{"message":"This API key does not have the \"train\" scope. …",
#           "type":"permission_error"}}   → 403

On every request, on every route below. Not Authorization, not a bearer token, not a query parameter. A key must begin with fsk_ to be looked up at all.

curl -H "x-api-key: $KEY" "$BASE/api/$PROJECT_ID/health"

All three authentication_error cases are 401: a missing or malformed header, a key that is unknown or revoked, and — easy to miss — a valid key used against a different project's URL. The key alone identifies the project; the id in the path is checked for agreement so a copy-paste mistake fails loudly instead of quietly answering for the wrong stack.

A perfectly good key still gets 409 deployment_state_error while the project is not live: "Solution is deploying. Endpoints activate when provisioning completes." while deploying, "Solution has not been deployed yet." before that. Treat 409 as retry-later, not as an auth problem.

Three spellings of the prompt are accepted and mean the same thing: a messages array of {role, content}, a bare question string, or a bare message string. With messages, the last entry whose role is user is the retrieval query.

BASE="https://your-forgestack-host"   # where this dashboard runs
PROJECT_ID="your-project-id"           # the id in the dashboard URL
KEY="fsk_…"                            # from Keys & endpoints

curl -X POST "$BASE/api/$PROJECT_ID/chat" \
  -H "x-api-key: $KEY" \
  -H "content-type: application/json" \
  -d '{"messages":[{"role":"user","content":"How long do refunds take?"}]}'

import requests

r = requests.post(
    f"{BASE}/api/{PROJECT_ID}/chat",
    headers={"x-api-key": KEY},
    json={"messages": [{"role": "user", "content": "How long do refunds take?"}]},
    timeout=200,
)
data = r.json()
print(data["choices"][0]["message"]["content"])
for c in data["citations"]:
    print(" ↳", c["document"])

const res = await fetch(`${BASE}/api/${PROJECT_ID}/chat`, {
  method: "POST",
  headers: { "x-api-key": KEY, "content-type": "application/json" },
  body: JSON.stringify({ messages: [{ role: "user", content: "How long do refunds take?" }] }),
});
const data = await res.json();
console.log(data.choices[0].message.content, data.citations);

messages / question / message — the prompt, one of the three. max_tokens — cap on the completion, default 512. Temperature is NOT a request field: the value from the Configure screen is applied server-side, so the same project answers with the same sampling for every caller. Streaming is not supported; a response arrives whole.

Every conversation is saved to your stack's own S3 bucket (chatlogs/, one folder per source) and shows on the dashboard's Chat logs screen. Three optional fields shape it. source names the app the chat comes from ("website", "ios-app"); without it, the calling website's Origin names it, and failing that it is "api". user is your own id for your end user, so the owner can see whose chat it was. conversation_id groups turns: each response returns one — send it back on the next call to continue the same conversation, or leave it out to start a new one. History is kept for a month unless the owner picks another retention for that source.

{
  "messages": [{ "role": "user", "content": "And for digital goods?" }],
  "source": "website",
  "user": "customer-8812",
  "conversation_id": "cmv80e2tq22pm35"
}

choices[0].message.content is the answer. citations lists the KB chunks that were put in the prompt. usage counts tokens as the node's own tokenizer counts them. latency_ms is the node's generation time, so it excludes network time to and from you.

{
  "id": "chatcmpl-…",
  "object": "chat.completion",
  "model": "…",
  "choices": [
    { "message": { "role": "assistant", "content": "Up to 14 days." },
      "finish_reason": "stop" }
  ],
  "citations": [{ "document": "kb/refund-policy.md", "excerpt": "…" }],
  "usage": { "prompt_tokens": 214, "completion_tokens": 12, "total_tokens": 226 },
  "latency_ms": 1840,
  "conversation_id": "cmv80e2tq22pm35"
}

A large model on a CPU-only instance answers in minutes, not seconds. ForgeStack waits up to 180 seconds for the node by default before giving up with 504, so any client timeout shorter than that will fire first and hide the real outcome.

error.type classifies it; error.code, when present, is the stable machine-readable reason. Branch on code, never on the message — messages are written to be read by people and are rewritten when they read badly.

{ "error": { "message": "…", "type": "deployment_state_error", "code": "model_loading" } }

The stack is up but the weights are still downloading onto the node. When the node reports progress, a download object rides along on the response as {downloadedBytes, totalBytes, percent}. Poll GET /health, or just retry with backoff; chat starts working when the node reports warm.

The node did not answer within the inference timeout. A shorter prompt, a lower max_tokens or a GPU instance type are the three things that actually change this.

unreachable — the node did not accept a connection. bad_response — something answered but not with a completion, usually a stack running an older router, fixed by redeploying. model_error — the model failed to load. unauthorized — the stack rejected ForgeStack's own internal node key, which means the stack was deployed against a key that has since been rotated; redeploy. None of these are about your API key.

"The model node is unreachable. Redeploy or reconnect the stack, then try again." There is no fallback path: your documents live in your stack's bucket, not here, so there is nothing for ForgeStack to answer from on its own.

Unparseable JSON, or no prompt under any of the three accepted names. The message names all three so you do not have to come back here to find out which.

Grounded chat completion against your model and knowledge base.

Add a knowledge-base document, or list the indexed ones. See the Knowledge base section above for bodies and limits.

Remove one document by the id that GET returned — from the bucket and from the live index.

Start a LoRA fine-tune on your own examples, or poll the running job.

Stack, model and knowledge-base status. Useful as a readiness probe: it reports the stack name and region, the model id and node instance type, the KB bucket with its document and byte counts, the indexing pipeline, and uptime_since — null until a deploy has finished.

curl -H "x-api-key: $KEY" "$BASE/api/$PROJECT_ID/health"

Anything under /api/projects/… is the dashboard's own API. It authenticates with your browser session, not with an API key, and is not a public interface — do not build against it.