Inference

Node Data serves its own metered inference API. The wire format is OpenAI-compatible, so any OpenAI client pointed at https://www.nodedata.ai/api/v1 works — the SDK just saves you a second dependency.

Model catalog

Node model ids are stable. The upstream weights they're served on are an implementation detail and can change without a version bump — you bill against the Node id and the Node price.

Fetch it live rather than hardcoding prices: await nd.inference.models().

A single completion

const completion = await nd.inference.create({
  model: "node-reason-70b",
  messages: [
    { role: "system", content: "You analyse robot trajectory logs." },
    { role: "user", content: log },
  ],
  max_tokens: 512,
  temperature: 0.2,
});
 
console.log(completion.choices[0].message.content);
console.log(completion.usage); // prompt/completion/total tokens

Streaming

Total time is roughly the same; perceived latency collapses from seconds to about a hundred milliseconds. Use it for anything a human watches.

for await (const chunk of nd.inference.stream({
  model: "node-fast-8b",
  messages: [{ role: "user", content: "Write a ROS 2 launch file." }],
})) {
  process.stdout.write(chunk.choices[0]?.delta.content ?? "");
}
 
// Or collect the whole thing while still rendering as it arrives
const text = await nd.inference.streamToText(
  { model: "node-fast-8b", messages },
  { onToken: (token) => render(token) },
);

The raw format is SSE. Chunk boundaries do not respect event boundaries, so a hand-rolled parser must buffer until a blank line and stop on data: [DONE].

data: {"id":"chatcmpl_1","choices":[{"delta":{"role":"assistant"}}]}
 
data: {"id":"chatcmpl_1","choices":[{"delta":{"content":"The"}}]}
 
data: {"id":"chatcmpl_1","choices":[{"delta":{"content":" arm"}}]}
 
data: [DONE]

Structured output

const completion = await nd.inference.create({
  model: "node-fast-8b",
  messages: [
    { role: "system", content: 'Reply with {"severity": 1-5, "summary": string}.' },
    { role: "user", content: report },
  ],
  response_format: { type: "json_object" },
  temperature: 0,
});
 
// JSON mode constrains the format, not the schema — still validate.
const parsed = ReportSchema.parse(
  JSON.parse(completion.choices[0].message.content),
);

Tool calling

Describe your functions with JSON Schema and the model returns the calls it wants made. Nothing is executed for you — a tool call is a request, and running it is your side of the loop.

const tools = [
  {
    type: "function" as const,
    function: {
      name: "get_battery",
      description: "Current battery percentage for a robot.",
      parameters: {
        type: "object",
        properties: { robot_id: { type: "string" } },
        required: ["robot_id"],
      },
    },
  },
];
 
const messages = [{ role: "user", content: "Is arm-7 charged enough to run?" }];
 
const first = await nd.inference.create({
  model: "node-reason-70b",
  messages,
  tools,
});
 
// finish_reason is "tool_calls" when the model wants one or more calls.
for (const call of first.choices[0].message.tool_calls ?? []) {
  const args = JSON.parse(call.function.arguments); // always a JSON *string*
  const result = await getBattery(args.robot_id);
 
  messages.push(first.choices[0].message);      // the turn that asked
  messages.push({
    role: "tool",
    tool_call_id: call.id,                      // must match the call
    content: JSON.stringify(result),
  });
}
 
// Send the results back for the model to answer with.
const second = await nd.inference.create({
  model: "node-reason-70b",
  messages,
  tools,
});

Both round-trips are billed — the tool schemas and the results are input tokens like any other part of the prompt. tool_choice takes "auto", "none", "required", or { type: "function", function: { name } } to force a specific call.

Controlling cost

• Route by difficulty. node-fast-8b handles classification and extraction at roughly a twelfth the price of the flagship. This is usually the largest single win.
• Always set max_tokens. It's the only hard ceiling on a runaway generation.
• Cache identical prompts. Deterministic prompts at temperature 0 are trivially cacheable.
• Trim context. Input tokens are billed too; a 50k-token prompt for a yes/no answer is pure waste.

Estimate before you ship with the cost calculator, and check the real numbers afterwards:

const usage = await nd.account.usage({ days: 30 });
 
console.log(usage.summary.cost_usd, usage.summary.total_tokens);
for (const model of usage.by_model) {
  console.log(model.model, model.calls, model.cost_usd);
}

Costs are reported in USD micros (millionths) as well as dollars, so the integer field is the one to store in a ledger.

The credit limit

Inference is post-paid: calls run first and accrue an outstanding balance you settle by card. That balance has a ceiling — $25 by default — and while it is reached, completions return 402 spend_limit_reached instead of running.

It is a rolling limit, not a quota. Settling the balance at Billing restores the full allowance immediately, and nothing is charged for a refused call. Two things trip it:

• The balance reached the limit. Settle to continue.
• This request could carry it past. Checked before the call against an upper bound from your prompt and max_tokens, so lowering max_tokens or trimming context gets you through without paying anything.

Every answered call reports the headroom in response headers, so a proxy or dashboard can watch it without a second request:

x-nodedata-spend-limit-micros: 25000000
x-nodedata-spend-outstanding-micros: 2891422
x-nodedata-spend-remaining-micros: 22108578

The numbers are measured before the call they accompany, so they lag by one request. nd.account.usage() returns the same figures under balance, including limit_reached. In the SDK the refusal is a SpendLimitError — a subclass of PaymentRequiredError, so you can catch it specifically to trigger a settle flow rather than a blind retry.

import { SpendLimitError } from "@node-data/sdk";
 
try {
  await nd.inference.create({ model: "node-fast-8b", messages });
} catch (err) {
  if (err instanceof SpendLimitError) {
    // Nothing was charged. Retrying without settling fails the same way.
    await notifyBillingOwner(err.message);
    return;
  }
  throw err;
}