Network Programming in .NET

When developers first encounter a large-scale AI classification job — say, two million records that each need to be sent to an LLM for analysis — the instinct is immediately familiar: spin up threads, parallelise the work, saturate the API. It’s the same pattern that works for database processing, file I/O, HTTP scraping. More threads, more throughput.

With LLM APIs, that instinct leads you straight into a wall. And the wall has a name: TPM.

The Problem with Multithreading LLM Calls

Most LLM APIs — OpenAI included — impose a Tokens Per Minute (TPM) limit. This is a rolling window, not a per-request limit. Every token you send in a prompt, and every token the model returns, counts against it.

The naive multithreaded approach burns through this budget in a way that’s both wasteful and hard to control:

The system prompt repeats on every request. If your prompt is 700 tokens and you’re running 20 threads firing one request each, you’re spending 14,000 tokens per second just on prompt overhead — before the model has classified a single record. With a 200,000 TPM limit, you’ve consumed 4.2 minutes of budget in one second.

Burst behaviour triggers rate limits unpredictably. The TPM limit is a rolling window. Twenty threads firing simultaneously create a spike that can exceed the per-minute budget in seconds, even if your average rate would be well within limits. The API returns 429 errors, your retry logic kicks in, those retries themselves consume tokens, and the situation compounds.

Thread count is a blunt instrument. Dialling concurrency up and down doesn’t map cleanly to token consumption because request latency varies. A batch that takes 500ms doesn’t consume the same tokens as one that takes 1,500ms, but both hold a thread slot for their duration.

The Better Model: Semantic Batching

The insight that changes everything is this: the system prompt is a fixed overhead, and you should amortise it across as many classifications as possible per API call.

Instead of:

Thread 1: [system prompt 700 tokens] + [address 1: 15 tokens] → [result: 15 tokens]
Thread 2: [system prompt 700 tokens] + [address 2: 15 tokens] → [result: 15 tokens]
...× 20 threads
Total: 14,000 tokens for 20 classifications

You send:

[system prompt 700 tokens] + [addresses 1-20: 300 tokens] → [results 1-20: 100 tokens]
Total: 1,100 tokens for 20 classifications

That’s a 12× reduction in token consumption for the same work. Suddenly your 200,000 TPM budget — which could only sustain ~270 single-record requests per minute — supports ~3,600 classifications per minute. No extra threads needed.

Key Implementation Details

1. Include an ID in Both Request and Response

The most important correctness detail in batch processing is never rely on positional alignment.

If you send 20 addresses and ask the model to return 20 results, it might return 19. Now you don’t know which one it dropped. If you’re matching by position, records from item 7 onwards get silently misclassified.

The fix is to include a unique identifier in both directions:

User message:
id=548033: product X
id=548034: product Y
...
System prompt format instruction:
Reply ONLY with a JSON array. Format: [{"id":548033,"c":"E"}, ...]

Now you build a dictionary from the response keyed on id, and match each input item explicitly. A missing id means that specific record gets skipped and retried on the next run. Everything else classifies correctly regardless of what the model dropped.

2. Resolve Labels Locally

The model doesn’t need to return the full label text. "Prime City Professionals" costs tokens on every response item. A single letter costs one token.

Keep a static dictionary in your code:

csharp

private static readonly Dictionary<string, string> Labels = new()
{
    { "A", "Prime Product"   },
    { "B", "Budget Product"        },
    // ...
};

The model returns "c":"A", you look up the label locally. This also eliminates a class of hallucination errors where the model invents a label name slightly different from your taxonomy.

Note: even "category" vs "c" matters at scale. In the OpenAI tokenizer, "category" is 3 tokens; "c" is 1. Across 100,000 batch calls, that’s 200,000 tokens — small but free.

3. Track TPM with a Rolling Window, Not Concurrency

Rather than trying to infer safe concurrency from trial and error, measure what you’re actually consuming and throttle directly on that signal.

csharp

// On each successful response, record tokens used with a timestamp
tokenWindow.Enqueue((DateTime.UtcNow, inputTokens + outputTokens));
// Before each request, prune entries older than 60 seconds and sum the rest
var cutoff = DateTime.UtcNow.AddSeconds(-60);
while (window.Peek().t < cutoff) window.Dequeue();
long tpmUsed = window.Sum(x => x.tok);
// Throttle graduated to usage
if (tpmUsed > tpmLimit * 0.98) Thread.Sleep(2000);
else if (tpmUsed > tpmLimit * 0.95) Thread.Sleep(800);
else if (tpmUsed > tpmLimit * 0.85) Thread.Sleep(300);

This gives you automatic, self-correcting throttling that responds to real consumption rather than guessing from thread counts. If a batch of records happens to have longer addresses, the window fills faster and the delay kicks in sooner. No manual tuning required.

4. Resumability via Cursor Pagination

For a job that takes hours or days, stopping and restarting must be safe and cheap. The key is two things working together:

Write results immediately after each batch, not at the end of a page. If you crash mid-page, you’ve lost one batch (20 records), not a thousand.

Use a NULL-check filter combined with cursor pagination. The query for unclassified records looks like:

sql

WHERE segment_category IS NULL AND id > {lastId} ORDER BY id LIMIT 1000

On restart, lastId resets to 0, but the IS NULL filter automatically skips everything already classified. The cursor (id > lastId) keeps the query fast on large tables — OFFSET pagination slows to a crawl at millions of rows because the database still has to scan all preceding rows to find the offset position.

5. Handle Partial Batches Gracefully with Skip vs Error

Not all failures are equal. Distinguish between:

• Error: something went wrong that warrants logging (HTTP 500, persistent 429 after retries, DB connection failure). These need attention.
• Skip: the record wasn’t returned in this batch response. Leave it NULL in the database, it will be picked up automatically on the next run. No log noise needed.

This distinction keeps your error output meaningful. If every missing batch item logs as an error, a run with 0.1% skip rate produces thousands of error lines that mask real problems.

The Result

What started as a job estimated at 16–67 days with a naive multithreaded approach settled to around 7 hours using semantic batching — processing two million records through a rate-limited API without a single configuration change to the API account.

The throughput improvement didn’t come from more concurrency. It came from being smarter about what gets sent in each request.

The general principle applies beyond LLM classification: whenever you have a fixed overhead per API call (authentication, context, schema), the correct optimisation is to amortise that overhead across as much work as possible per call, not to fire more calls in parallel.

Summary of Patterns

The instinct to parallelise is correct in principle — you want to keep the API busy. But with token-limited LLM APIs, the right parallelism is within a single request, not across many simultaneous ones.

In today’s AI-driven world, connecting specialized APIs to large language models opens up powerful possibilities. One particularly useful integration is connecting vehicle registration lookup services to OpenAI’s custom GPTs through Actions. In this tutorial, we’ll walk through how to integrate the RegCheck API with OpenAI Actions, enabling your custom GPT to look up vehicle information from over 30 countries.

What is RegCheck?

RegCheck is a comprehensive vehicle data API that provides detailed information about vehicles based on their registration numbers (license plates). With support for countries including the UK, USA, Australia, and most of Europe, it’s an invaluable tool for automotive businesses, insurance companies, and vehicle marketplace platforms.

Why Integrate with OpenAI Actions?

OpenAI Actions allow custom GPTs to interact with external APIs, extending their capabilities beyond text generation. By integrating RegCheck, you can create a GPT assistant that:

• Instantly looks up vehicle specifications for customers
• Provides insurance quotes based on real vehicle data
• Assists with vehicle valuations and sales listings
• Answers detailed questions about specific vehicles

Prerequisites

Before you begin, you’ll need:

• An OpenAI Plus subscription (for creating custom GPTs)
• A RegCheck API account with credentials
• Basic familiarity with OpenAPI specifications

Step-by-Step Integration Guide

Step 1: Create Your Custom GPT

Navigate to OpenAI’s platform and create a new custom GPT. Give it a name like “Vehicle Lookup Assistant” and configure its instructions to handle vehicle-related queries.

Step 2: Add the OpenAPI Schema

In your GPT configuration, navigate to the “Actions” section and add the following OpenAPI specification:

yaml

openapi: 3.0.0
info:
  title: RegCheck Vehicle Lookup API
  version: 1.0.0
  description: API for looking up vehicle registration information across multiple countries
servers:
  - url: https://www.regcheck.org.uk/api/json.aspx

paths:
  /Check/{registration}:
    get:
      operationId: checkUKVehicle
      summary: Get details for a vehicle in the UK
      parameters:
        - name: registration
          in: path
          required: true
          schema:
            type: string
          description: UK vehicle registration number
      responses:
        '200':
          description: Successful response
          content:
            application/json:
              schema:
                type: object

  /CheckSpain/{registration}:
    get:
      operationId: checkSpainVehicle
      summary: Get details for a vehicle in Spain
      parameters:
        - name: registration
          in: path
          required: true
          schema:
            type: string
          description: Spanish vehicle registration number
      responses:
        '200':
          description: Successful response
          content:
            application/json:
              schema:
                type: object

  /CheckFrance/{registration}:
    get:
      operationId: checkFranceVehicle
      summary: Get details for a vehicle in France
      parameters:
        - name: registration
          in: path
          required: true
          schema:
            type: string
          description: French vehicle registration number
      responses:
        '200':
          description: Successful response
          content:
            application/json:
              schema:
                type: object

  /VinCheck/{vin}:
    get:
      operationId: checkVehicleByVin
      summary: Get details for a vehicle by VIN number
      parameters:
        - name: vin
          in: path
          required: true
          schema:
            type: string
          description: Vehicle Identification Number
      responses:
        '200':
          description: Successful response
          content:
            application/json:
              schema:
                type: object

Note: You can expand this schema to include additional endpoints for other countries as needed. The RegCheck API supports over 30 countries.

Step 3: Configure Authentication

1. In the Authentication section, select Basic authentication
2. Enter your RegCheck API username
3. Enter your RegCheck API password
4. OpenAI will securely encrypt and store these credentials

The authentication header will be automatically included in all API requests made by your GPT.

Step 4: Test Your Integration

Use the built-in test feature in the Actions panel to verify the connection:

1. Select the checkUKVehicle operation
2. Enter a test registration like YYO7XHH
3. Click “Test” to see the response

You should receive a JSON response with vehicle details including make, model, year, engine size, and more.

Step 5: Configure GPT Instructions

Update your GPT’s instructions to effectively use the new Actions:

You are a vehicle information assistant. When users provide a vehicle 
registration number, use the appropriate CheckVehicle action based on 
the country. Present the information in a clear, user-friendly format.

Always ask which country the registration is from if not specified.
Provide helpful context about the vehicle data returned.

Example Use Cases

Once integrated, your GPT can handle queries like:

User: “What can you tell me about UK registration YYO7XHH?”

GPT: [Calls checkUKVehicle action] “This is a 2007 Peugeot 307 X-line with a 1.4L petrol engine. It’s a 5-door manual transmission vehicle with right-hand drive…”

User: “Look up Spanish plate 0075LTJ”

GPT: [Calls checkSpainVehicle action] “Here’s the information for that Spanish vehicle…”

Best Practices and Considerations

API Limitations

• The RegCheck API is currently in BETA and may change without notice
• Consider implementing error handling in your GPT instructions
• Be aware of rate limits on your API account

Privacy and Security

• Never expose API credentials in your GPT’s instructions or responses
• Inform users that vehicle lookups are being performed
• Comply with data protection regulations in your jurisdiction

Optimizing Performance

• Cache frequently requested vehicle information where appropriate
• Use the most specific endpoint (e.g., CheckSpain vs. generic Check)
• Consider implementing fallback behavior for failed API calls

Expanding the Integration

The RegCheck API offers many more endpoints you can integrate:

• UKMOT: Access MOT test history for UK vehicles
• WheelSize: Get wheel and tire specifications
• CarSpecifications: Retrieve detailed specs by make/model/year
• Country-specific checks: Add support for Australia, USA, and 25+ other countries

Simply add these endpoints to your OpenAPI schema following the same pattern.

Troubleshooting Common Issues

Authentication Errors: Double-check your username and password are correct in the Authentication settings.

404 Not Found: Verify the registration format matches the country’s standard format.

Empty Responses: Some vehicles may not have complete data in the RegCheck database.

Conclusion

Integrating the RegCheck API with OpenAI Actions transforms a standard GPT into a powerful vehicle information assistant. Whether you’re building tools for automotive dealerships, insurance platforms, or customer service applications, this integration provides instant access to comprehensive vehicle data from around the world.

The combination of AI’s natural language understanding with RegCheck’s extensive vehicle database creates a seamless user experience that would have required significant custom development just a few years ago.

Ready to get started? Create your RegCheck account, set up your custom GPT, and start building your vehicle lookup assistant today!