Inference

Node Data serves its own metered inference API. The wire format is OpenAI-compatible, so any OpenAI client pointed at https://www.nodedata.ai/api/v1 works — the SDK just saves you a second dependency.

Model catalog

Node model ids are stable. The upstream weights they're served on are an implementation detail and can change without a version bump — you bill against the Node id and the Node price.

Fetch it live rather than hardcoding prices: await nd.inference.models().

A single completion

const completion = await nd.inference.create({
  model: "node-reason-70b",
  messages: [
    { role: "system", content: "You analyse robot trajectory logs." },
    { role: "user", content: log },
  ],
  max_tokens: 512,
  temperature: 0.2,
});
 
console.log(completion.choices[0].message.content);
console.log(completion.usage); // prompt/completion/total tokens

Streaming

Total time is roughly the same; perceived latency collapses from seconds to about a hundred milliseconds. Use it for anything a human watches.

for await (const chunk of nd.inference.stream({
  model: "node-fast-8b",
  messages: [{ role: "user", content: "Write a ROS 2 launch file." }],
})) {
  process.stdout.write(chunk.choices[0]?.delta.content ?? "");
}
 
// Or collect the whole thing while still rendering as it arrives
const text = await nd.inference.streamToText(
  { model: "node-fast-8b", messages },
  { onToken: (token) => render(token) },
);

The raw format is SSE. Chunk boundaries do not respect event boundaries, so a hand-rolled parser must buffer until a blank line and stop on data: [DONE].

data: {"id":"chatcmpl_1","choices":[{"delta":{"role":"assistant"}}]}
 
data: {"id":"chatcmpl_1","choices":[{"delta":{"content":"The"}}]}
 
data: {"id":"chatcmpl_1","choices":[{"delta":{"content":" arm"}}]}
 
data: [DONE]

Structured output

const completion = await nd.inference.create({
  model: "node-fast-8b",
  messages: [
    { role: "system", content: 'Reply with {"severity": 1-5, "summary": string}.' },
    { role: "user", content: report },
  ],
  response_format: { type: "json_object" },
  temperature: 0,
});
 
// JSON mode constrains the format, not the schema — still validate.
const parsed = ReportSchema.parse(
  JSON.parse(completion.choices[0].message.content),
);

Controlling cost

• Route by difficulty. node-fast-8b handles classification and extraction at roughly a twelfth the price of the flagship. This is usually the largest single win.
• Always set max_tokens. It's the only hard ceiling on a runaway generation.
• Cache identical prompts. Deterministic prompts at temperature 0 are trivially cacheable.
• Trim context. Input tokens are billed too; a 50k-token prompt for a yes/no answer is pure waste.

Estimate before you ship with the cost calculator, and check the real numbers afterwards:

const usage = await nd.account.usage({ days: 30 });
 
console.log(usage.summary.cost_usd, usage.summary.total_tokens);
for (const model of usage.by_model) {
  console.log(model.model, model.calls, model.cost_usd);
}

Costs are reported in USD micros (millionths) as well as dollars, so the integer field is the one to store in a ledger.