Archive
The Static HttpClient That Wouldn’t Rotate: A Tale of Pooled Connections
The symptom
A production .NET service had been running fine for months. It made outbound HTTP calls through a rotating proxy provider — the kind that promises a new exit IP for each request. Then one day, requests started timing out. Not failing cleanly with a 4xx or 5xx. Timing out. Each one sat there for thirty seconds or more before giving up.
The fix that worked, every time, was restarting IIS.
That’s an interesting fix. It tells you the problem isn’t with the code’s logic — the code is the same after the restart as before. It tells you the problem is with state that the application is accumulating over its lifetime. And it tells you that whatever that state is, it lives somewhere inside the application’s process, not on disk or in a database.
The setup
Here’s the shape of the code:
csharp
private static readonly HttpClient _httpClient = new HttpClient(new HttpClientHandler{ Proxy = _rotatingProxy, UseProxy = true});
A static HttpClient. A rotating proxy. By every piece of Microsoft guidance written in the last decade, this is correct. “Don’t create a new HttpClient per request — it’ll exhaust your sockets.” So a static instance it is.
The intent: every outbound request goes through the proxy, the proxy assigns a different exit IP each time, and the upstream service sees a varied stream of source addresses rather than a single hammering client.
The reality, as we’ll see, was very different.
Anti-pattern #1: assuming HttpClient opens a new connection per request
This is the foundational misunderstanding. HttpClient does not open a new TCP connection for each request. It maintains a connection pool, keyed by destination host, and reuses pooled connections wherever possible. This is HTTP keep-alive doing its job — and it’s a good thing for the general case, because TCP and TLS handshakes are expensive.
But here’s the consequence for a rotating proxy: rotation happens per TCP connection, not per HTTP request. If your HttpClient opens one TCP connection on the first request and then reuses that connection for every subsequent request, you get one exit IP, forever. Your “rotating” proxy is effectively a sticky proxy, because nothing is ever telling it you want a new connection.
The static HttpClient makes this worse, because the connection pool lives for the lifetime of the process. There is no natural moment at which the pooled connection gets replaced. It just sits there, being reused, until something forces it to close.
Anti-pattern #2: no timeout configured
HttpClient.Timeout defaults to 100 seconds. Most developers never set it. For most APIs this is fine — if the call is going to succeed, it’ll succeed in well under 100 seconds.
But “100 seconds” becomes a serious problem when something on the network silently fails. A pooled TCP connection that’s been killed by an intermediate firewall, a load balancer that’s stopped routing your traffic, a remote server that’s decided to tarpit you — none of these produce a clean error. They produce a hang. And your code will sit in that hang for the full timeout duration.
The combination of “no timeout” + “pooled connection that’s gone bad” is the textbook recipe for the mysterious thirty-second-plus pause that I described at the top. The connection looks fine to your code. The pool hands it out. The request goes out. Nothing comes back. You wait.
Anti-pattern #3: silent bans look identical to network glitches
Modern anti-abuse systems rarely respond to suspected scraping or hammering with a clean 403 Forbidden. A 403 is a gift to the attacker — it tells them immediately that they need to rotate. Far more effective is to do nothing: accept the TCP connection, accept the request, and then never send a response. The attacker’s client hangs. Their thread is consumed. Their queue backs up. Their logs fill with timeouts that look indistinguishable from random network failures.
If you’re not expecting this behaviour, you’ll spend a lot of time chasing phantom network problems, blaming your cloud provider, blaming your proxy, blaming DNS. The truth — that you’ve been quietly added to a deny list — is invisible from the symptoms alone.
When you combine this with the pooled-connection problem, you get a particularly nasty failure mode: you get one exit IP from your proxy, you hammer the upstream API from that single IP, the upstream API decides you’re abusive and starts tarpitting that IP, and now every single request hangs because they’re all going through the same pooled connection to the same banned IP. The only way out is to break the connection — which an IIS restart conveniently does as a side effect.
What’s actually going on
Putting these three pieces together, the mechanism is:
- The application starts and opens its first outbound connection through the proxy. The proxy assigns exit IP
X. - .NET pools that connection and reuses it for every subsequent request.
- Every request to the upstream service comes from IP
X. - Upstream eventually notices the volume from IP
Xand starts silently dropping requests. - Now every request hangs for the default 100-second timeout, or however long the underlying TCP layer takes to give up.
- Restarting the application destroys the connection pool, forces a new TCP connection on the next request, and the proxy assigns a new exit IP. Service is restored — until the cycle repeats.
The root cause is not the proxy. The proxy is doing what it was told. The root cause is that the application is never asking it for a new connection.
The pattern: disable keep-alive when you actually want rotation
The fix is one line:
csharp
client.DefaultRequestHeaders.ConnectionClose = true;
This sets the Connection: close header on every outgoing request. It tells .NET’s connection pool not to reuse the underlying TCP socket after the response. The next request opens a fresh connection. The proxy assigns a fresh exit IP. Rotation now works the way you expected it to work all along.
The complete pattern looks like this:
csharp
private static readonly HttpClient _httpClient = BuildHttpClient();private static HttpClient BuildHttpClient(){ var handler = new HttpClientHandler { Proxy = _rotatingProxy, UseProxy = true }; var client = new HttpClient(handler) { Timeout = TimeSpan.FromSeconds(15) }; client.DefaultRequestHeaders.ConnectionClose = true; return client;}
Three things to notice:
The HttpClient is still static. We haven’t abandoned the standard guidance. The object itself is long-lived; it’s just configured not to pool connections internally.
The timeout is now sane. Fifteen seconds is appropriate for most API calls. If something goes wrong — a tarpit, a network glitch, anything — we fail fast and try again, rather than blocking a thread for a minute and a half.
There’s no manual connection management. No tracking of connection age, no recycling logic, no locks around pool rebuilds. The framework handles it because we’ve told it to handle it the right way.
The cost, and when not to do this
Disabling keep-alive isn’t free. Every request now pays the full cost of a TCP handshake and a TLS negotiation. Through a proxy, that’s typically 400–800ms of overhead per request, on top of whatever the upstream service itself takes.
If you’re making thousands of requests per second to the same endpoint and you want a stable identity from the upstream service’s perspective, do not do this. Keep-alive exists for excellent reasons. The pattern in this post is specifically for the case where you have a rotating proxy and you actively want to defeat connection reuse so that rotation happens.
For high-volume scenarios where you want both rotation and performance, the right answer is usually session tokens — most rotating proxy providers let you encode a session identifier in the proxy username, and you can vary that token per request to force rotation without giving up keep-alive on individual sessions. That’s a more involved pattern and a topic for another post.
The broader lesson
The interesting thing about this bug is how each of the three anti-patterns is, in isolation, considered best practice or at least defensible. Static HttpClient is recommended. Default timeouts are what you get if you don’t think about them. And nobody designs their code around the possibility of being silently banned by an upstream service.
It’s the interaction between these three things, plus the presence of a rotating proxy, that produces the failure. And the failure mode — a hang, not an error — is the worst possible signal, because it tells you nothing useful and points you in completely the wrong direction.
The takeaway isn’t “always disable keep-alive” or “always set short timeouts.” The takeaway is that when your code interacts with infrastructure that has its own behaviours — proxies, load balancers, anti-abuse layers — the default settings of your HTTP client may not match the assumptions you’re making about how that infrastructure works. Take a moment to ask: does my code’s connection lifecycle match what I actually want to happen at the network level? If you have a rotating proxy and you’ve never thought about connection pooling, the answer is almost certainly no.
The Misleading IndexOutOfRangeException That Means “Your List Isn’t Thread-Safe”
If you’ve ever seen a stack trace like this in a .NET application:
System.IndexOutOfRangeException: Index was outside the bounds of the array. at System.Collections.Generic.List`1.Enumerator.MoveNext() at System.Linq.Enumerable.WhereListIterator`1.MoveNext() at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source) at YourCode.SomeMethod(...)
…and you stared at the offending line — something innocuous like myList.Where(x => x.IsActive).ToList() — wondering how on earth a LINQ query could be indexing outside an array, you’ve run into one of the most misleadingly-named exceptions in the framework.
The message says “index out of bounds.” The actual problem is a race condition.
What the stack trace is telling you
Read the frames from the bottom up. Your code called ToList(). ToList() calls the List<T> constructor that takes an IEnumerable<T>. That constructor enumerates the Where iterator, which in turn enumerates the underlying List<T> via its Enumerator. And it’s Enumerator.MoveNext() that throws.
So the iterator is walking your list, and at some point it tries to read element N, and N is past the end of the internal array.
How can that happen? The list got smaller — or its internal buffer got swapped — while the enumerator was mid-walk.
List<T> is implemented as a wrapper around a T[] array plus a _size field. The enumerator captures a reference to the list at construction time, then on each MoveNext() it increments an index and reads list._items[index]. If another thread calls Remove, Clear, or triggers a resize via Add between two MoveNext() calls, the array your enumerator is reading from may have been replaced with a smaller one, or the items may have been shuffled. The result: an index that was valid when you started iterating is no longer valid, and you get IndexOutOfRangeException.
Why you don’t get the “nice” exception
List<T> does have a version-check mechanism. Every mutation increments an internal _version field, and the enumerator records the version it started with. If MoveNext() notices the version has changed, it throws InvalidOperationException: Collection was modified; enumeration operation may not execute — the exception most .NET developers have seen at least once and immediately recognise as a concurrency or mid-loop-mutation bug.
The catch: that version check happens after the index increment and array access. If the racing thread mutates the list in a way that shrinks the array between those two operations, you hit the raw IndexOutOfRangeException before the version check ever runs. Same root cause, much worse error message.
This is also why the bug is so frustrating to reproduce. It depends on precise interleaving of two threads down to the instruction level. You can hammer it in a test loop and not see it for a million iterations, then it triggers twice in five minutes in production.
The minimal reproduction
var list = new List<int>();for (int i = 0; i < 1000; i++) list.Add(i);// Thread A — repeatedly enumerateTask.Run(() =>{ while (true) { var copy = list.Where(x => x > 0).ToList(); }});// Thread B — repeatedly mutateTask.Run(() =>{ var rng = new Random(); while (true) { if (list.Count > 0) list.RemoveAt(rng.Next(list.Count)); list.Add(rng.Next()); }});
Run that for a few seconds and you’ll get either InvalidOperationException or IndexOutOfRangeException — sometimes both, on different runs. The exception you get is essentially a coin flip determined by exactly when the second thread’s mutation lands relative to the first thread’s bounds check.
The diagnostic giveaway
The single most useful signal in the stack trace is this frame:
at System.Collections.Generic.List`1.Enumerator.MoveNext()
IndexOutOfRangeException originating from List<T>.Enumerator.MoveNext almost always means concurrent modification. The enumerator’s own code is straightforward — there’s no realistic way for it to compute a bad index on its own. Something outside the enumerator changed the list while it was looking away.
If your stack trace shows that frame, stop looking for off-by-one errors in your LINQ predicate. Start looking for which other thread is writing to the same list.
Fixing it
The fix depends on the access pattern. In rough order of preference:
Use the right collection type. System.Collections.Concurrent has thread-safe equivalents tuned for different shapes of access:
ConcurrentBag<T>— many writers, occasional bulk drain. No keyed lookup or removal by item.ConcurrentQueue<T>/ConcurrentStack<T>— FIFO / LIFO producer-consumer pipelines.ConcurrentDictionary<TKey, TValue>— by far the most generally useful. Supports thread-safe add, remove, lookup, and snapshot enumeration via.Keysand.Values. If you’re keying items by an ID, this is almost always what you want.
Crucially: .Values on a ConcurrentDictionary returns a snapshot, so iterating it is safe even if other threads are mutating the dictionary at the same time. No exceptions, no locks.
Lock around access. If you’re stuck with List<T> — maybe the API surface is fixed, or the contention is low enough that the overhead doesn’t matter — wrap every read and every write in a lock on a dedicated private object. Every read and every write. Missing one is enough to bring the bug back.
private static readonly object _lock = new object();private static readonly List<Thing> _items = new List<Thing>();public static List<Thing> GetActiveThings(){ lock (_lock) { return _items.Where(t => t.IsActive).ToList(); }}public static void AddThing(Thing t){ lock (_lock) { _items.Add(t); }}
The ToList() inside the lock is deliberate — it forces the enumeration to complete before the lock is released, so the returned list is a safe, isolated copy the caller can work with at leisure.
Snapshot, then iterate. A halfway measure for read-mostly workloads:
var snapshot = Volatile.Read(ref _items).ToArray();
…paired with copy-on-write semantics for the writers. This is the pattern behind ImmutableList<T> from System.Collections.Immutable, which is worth knowing about for scenarios with many readers and rare writers.
What I’d take away from this
Two things.
First: when you see IndexOutOfRangeException coming out of a LINQ chain on a List<T>, your first hypothesis should be “another thread is writing to this list,” not “my predicate has a bug.” The stack trace looks like a logic error and it almost never is.
Second: List<T> being non-thread-safe is one of those facts every .NET developer knows in the abstract and still trips over in practice, because the framework gives you no help at all until something explodes. There’s no ThrowIfShared mode, no Roslyn analyzer that flags static List<T> fields, no runtime check at write time. The only feedback you get is a confusing exception from deep inside an enumerator, possibly weeks after deployment.
The fix is almost always “use the right type from System.Collections.Concurrent.” It costs you nothing in code clarity and saves you from a class of bug that’s genuinely painful to track down once it’s loose in production.
The dangers of Parallel.ForEach(… , async (item)) in IIS
A single, trivial exception — one that your code already has a catch block for — shouldn’t be able to bring down your entire IIS web server. But it can, and it will, if you combine Parallel.ForEach with an async lambda. This post explains exactly why it happens, how to spot it in the Windows Event Log, and how to fix it permanently.
The Setup
You have a method that needs to perform the same async operation against multiple items — calling a set of external APIs, processing a batch of records, sending a collection of requests. You reach for Parallel.ForEach because it sounds like the right tool: parallel work, multiple items, run them all at once. You even add a try/catch inside the lambda because you’re being responsible. It looks like this:
Parallel.ForEach(items, async (item) =>{ try { var result = await ProcessItemAsync(item); lock (results) { results.Add(result); } } catch (ItemNotFoundException) { // item not found - fine, skip it } catch (Exception ex) { lock (errors) { errors.Add(ex); } }});
This looks safe. It has error handling. It uses async/await. It compiles without a single warning. And it will crash your IIS worker process (w3wp.exe) the moment any exception is thrown after an await.
Why It Crashes: The async void Trap
Parallel.ForEach was designed before async/await existed in C#. It expects a synchronous Action<T> delegate. When you pass it an async lambda, something subtle and dangerous happens: the compiler silently treats the lambda as returning void rather than Task.
This is the async void anti-pattern, and it has one devastating property: any exception thrown inside it cannot be caught by any caller. It escapes directly to the thread’s synchronisation context — and on a raw ThreadPool thread, that means it goes completely unhandled.
Here is the exact sequence of events that kills your server:
Parallel.ForEachfires the lambda for each item in the collection- Each lambda hits the first
awaitand suspends, returning control immediately Parallel.ForEachsees each lambda return (as void) and considers its job doneParallel.ForEachexits — the method returns to its caller — everything looks fine- Milliseconds later, the awaited operations complete and the continuations resume on raw ThreadPool threads
- An exception is thrown inside one of those continuations
- The
try/catchinside the lambda? It only catches exceptions thrown before the firstawait. After theawait, the lambda has already returned as far asParallel.ForEachis concerned - The exception has no owner, no observer, no catch block — it propagates to the ThreadPool itself
- In .NET 4.0 and later, an unhandled exception on a ThreadPool thread terminates the process
w3wp.execrashes. IIS restarts the application pool. All in-flight requests are lost
The particularly insidious part is that the try/catch gives you a false sense of security. You can see it right there in the code. But it doesn’t work the way you expect once an await is involved.
A Minimal Reproduction
You don’t need a complex codebase to reproduce this. The following is all it takes:
public static void CrashIIS(){ Parallel.ForEach(new[] { 1, 2, 3 }, async (item) => { await Task.Delay(100); // simulate any async I/O throw new Exception("This kills w3wp.exe"); // After the await, this runs on an orphaned ThreadPool thread // The process terminates }); // Parallel.ForEach has already returned here // The crash happens 100ms later}
Call that from any ASP.NET request handler — a controller action, an HttpHandler, anywhere — and your application pool will crash within moments. The caller gets no exception. The HTTP response may even succeed before the crash occurs. The next user to make any request gets a 503.
Even wrapping the call in a try/catch at the call site doesn’t help:
try{ Parallel.ForEach(new[] { 1, 2, 3 }, async (item) => { await Task.Delay(100); throw new Exception("Crash"); });}catch (Exception ex){ // This NEVER fires. // The exception doesn't happen until after Parallel.ForEach // has already exited this try block entirely. Log(ex);}
The catch block is long gone by the time the exception is thrown. This is what makes the pattern so dangerous — it looks exception-safe at every level, and isn’t.
How It Appears in the Windows Event Log
When this crash occurs, it leaves a very specific fingerprint in the Windows Event Log. Open Event Viewer → Windows Logs → Application and look for two entries appearing within seconds of each other.
Entry 1: .NET Runtime — Unhandled Exception
Source: .NET Runtime
Event ID: 1026
Application: w3wp.exeFramework Version: v4.0.30319Description: The process was terminated due to an unhandled exception.Exception Info: YourNamespace.YourException at YourClass.YourMethod() at SomeClass+<SomeMethod>d__3.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task) at SomeClass+<>c__DisplayClass4_0+<<YourParallelMethod>b__0>d.MoveNext() at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Threading.ExecutionContext.RunInternal(...) at System.Threading.ExecutionContext.Run(...) at System.Threading.QueueUserWorkItemCallback.ExecuteWorkItem() at System.Threading.ThreadPoolWorkQueue.Dispatch()
The key indicators are at the bottom of the stack trace:
QueueUserWorkItemCallback.ExecuteWorkItem()ThreadPoolWorkQueue.Dispatch()
These tell you the exception surfaced on a raw ThreadPool work item with no managed owner — the classic signature of an orphaned async continuation. You will also see compiler-generated state machine names like <YourParallelMethod>b__0>d.MoveNext() in the trace, confirming the exception came from inside an async lambda. The angle brackets and the b__ notation are the C# compiler’s naming convention for anonymous methods and lambdas.
Entry 2: Application Error — w3wp.exe Fault
Source: Application Error
Event ID: 1000
Faulting application name: w3wp.exeFaulting module name: KERNELBASE.dllException code: 0xe0434352
Exception code 0xe0434352 is the Windows error code for a managed (.NET) exception that has escaped to the Win32 layer. It’s the OS-level record of a .NET exception killing a process. When you see this code combined with KERNELBASE.dll as the faulting module, a .NET unhandled exception is the cause.
What to Look For — Summary
| Signal | Where | What it means |
|---|---|---|
ThreadPoolWorkQueue.Dispatch() at bottom of stack | Event ID 1026, .NET Runtime | Exception from orphaned async continuation |
Compiler-generated names like b__0>d.MoveNext() | Event ID 1026, .NET Runtime | Exception came from inside an async lambda |
Exception code 0xe0434352 | Event ID 1000, Application Error | .NET exception killed the process |
Faulting module: KERNELBASE.dll | Event ID 1000, Application Error | Managed exception, not a native crash |
| Both entries within seconds of each other | Application log | Single event caused immediate process termination |
The Effect on IIS
When w3wp.exe terminates due to an unhandled exception, IIS detects the process death and marks the application pool as faulted. Depending on your Rapid Fail Protection settings (found in IIS Manager → Application Pools → Advanced Settings), IIS will either:
- Restart the worker process automatically — users experience a brief outage and then service resumes, with the first request after restart being slow due to application warm-up
- Disable the application pool if failures occur too frequently within the Rapid Fail Protection window (default: 5 failures in 5 minutes) — this results in a persistent 503 until an administrator manually starts the pool again
This is worth understanding because thread exhaustion and this crash pattern look identical from the outside — both produce 503 errors — but they behave very differently. Thread exhaustion self-recovers when load drops. A crashed application pool requires either automatic restart (if Rapid Fail Protection hasn’t tripped) or manual intervention. If your team is regularly performing IISResets to recover from outages, a crash like this is a more likely culprit than thread exhaustion.
The Fix
The correct replacement for Parallel.ForEach with async work is Task.WhenAll, which is async-native and properly propagates exceptions back to the awaiting caller:
public static async Task<IReadOnlyList<Result>> ProcessAllAsync(IEnumerable<Item> items){ var tasks = items.Select(async item => { try { return await ProcessItemAsync(item); } catch (ItemNotFoundException) { return Result.Empty; } }); // All items processed in parallel. // Exceptions surface here, as AggregateException, to a proper awaiter. return await Task.WhenAll(tasks);}
With Task.WhenAll:
- All items are processed in parallel — no performance regression
- Every async continuation is properly tracked by the Task infrastructure
- Exceptions are collected and re-thrown as
AggregateExceptionwhen awaited — to a caller that can handle them - The process does not terminate
As an immediate safety net while refactoring, you can also add a global handler in Global.asax that prevents process termination from unobserved task exceptions:
// In Application_Start (Global.asax)TaskScheduler.UnobservedTaskException += (sender, args) =>{ Logger.Error("Unobserved task exception", args.Exception); args.SetObserved(); // prevents process termination};
This is a safety net, not a fix — the underlying orphaned tasks still exist and their results are still lost. But it prevents a single unhandled background exception from taking down your entire server while you work through a proper refactor.
The Rule
The rule to remember is simple: never pass an async lambda to Parallel.ForEach. The two are fundamentally incompatible. Parallel.ForEach has no understanding of Task, does not await the work it fires, and any exception thrown after the first await inside your lambda will be orphaned on the ThreadPool. In .NET 4.0 and later, that means process termination.
The pattern is particularly easy to introduce because it compiles cleanly, looks reasonable, and even appears to have proper error handling. The only sign something is wrong is your server going down.
When you need parallel async work, use Task.WhenAll. It was designed for exactly this purpose.
Found this useful? If you’re diagnosing IIS instability, check your application pool’s Rapid Fail Protection settings and review Event Viewer’s Application log for Event ID 1026 with ThreadPoolWorkQueue.Dispatch() at the bottom of the stack trace — that’s the fingerprint that points directly to this pattern.
Migrating Google Cloud Run to Scaleway: Bringing Your Cloud Infrastructure Back to Europe
Introduction: Why European Cloud Sovereignty Matters Now More Than Ever
In an era of increasing geopolitical tensions, data sovereignty concerns, and evolving international relations, European companies are reconsidering their dependence on US-based cloud providers. The EU’s growing emphasis on digital sovereignty, combined with uncertainties around US data access laws like the CLOUD Act and recent political developments, has made many businesses uncomfortable with storing sensitive data on American infrastructure.
For EU-based companies running containerized workloads on Google Cloud Run, there’s good news: migrating to European alternatives like Scaleway is surprisingly straightforward. This guide will walk you through the technical process of moving your Cloud Run services to Scaleway’s Serverless Containers—keeping your applications running while bringing your infrastructure back under European jurisdiction.
Why Scaleway?
Scaleway, a French cloud provider founded in 1999, offers a compelling alternative to Google Cloud Run:
- 🇪🇺 100% European: All data centers located in France, Netherlands, and Poland
- 📜 GDPR Native: Built from the ground up with European data protection in mind
- 💰 Transparent Pricing: No hidden costs, generous free tiers, and competitive rates
- 🔒 Data Sovereignty: Your data never leaves EU jurisdiction
- ⚡ Scale-to-Zero: Just like Cloud Run, pay only for actual usage
- 🌱 Environmental Leadership: Strong commitment to sustainable cloud infrastructure
Most importantly: Scaleway Serverless Containers are technically equivalent to Google Cloud Run. Both are built on Knative, meaning your containers will run identically on both platforms.
Prerequisites
Before starting, ensure you have:
- An existing Google Cloud Run service
- Windows machine with PowerShell
gcloudCLI installed and authenticated- A Scaleway account (free to create)
- Skopeo installed (we’ll cover this)
Understanding the Architecture
Both Google Cloud Run and Scaleway Serverless Containers work the same way:
- You provide a container image
- The platform runs it on-demand via HTTPS endpoints
- It scales automatically (including to zero when idle)
- You pay only for execution time
The migration process is simply:
- Copy your container image from Google’s registry to Scaleway’s registry
- Deploy it as a Scaleway Serverless Container
- Update your DNS/endpoints
No code changes required—your existing .NET, Node.js, Python, Go, or any other containerized application works as-is.
Step 1: Install Skopeo (Lightweight Docker Alternative)
Since we’re on Windows and don’t want to run full Docker Desktop, we’ll use Skopeo—a lightweight tool designed specifically for copying container images between registries.
Install via winget:
powershell
winget install RedHat.Skopeo
Or download directly from: https://github.com/containers/skopeo/releases
Why Skopeo?
- No daemon required: No background services consuming resources
- Direct registry-to-registry transfer: Images never touch your local disk
- Minimal footprint: ~50MB vs. several GB for Docker Desktop
- Perfect for CI/CD: Designed for automation and registry operations
Configure Skopeo’s Trust Policy
Skopeo requires a policy file to determine which registries to trust. Create it:
powershell
# Create the config directoryNew-Item -ItemType Directory -Force -Path "$env:USERPROFILE\.config\containers"# Create a permissive policy that trusts all registries@"{ "default": [ { "type": "insecureAcceptAnything" } ], "transports": { "docker-daemon": { "": [{"type": "insecureAcceptAnything"}] } }}"@ | Out-File -FilePath "$env:USERPROFILE\.config\containers\policy.json" -Encoding utf8
For production environments, you might want a more restrictive policy that only trusts specific registries:
powershell
@"{ "default": [{"type": "reject"}], "transports": { "docker": { "gcr.io": [{"type": "insecureAcceptAnything"}], "europe-west2-docker.pkg.dev": [{"type": "insecureAcceptAnything"}], "rg.fr-par.scw.cloud": [{"type": "insecureAcceptAnything"}] } }}"@ | Out-File -FilePath "$env:USERPROFILE\.config\containers\policy.json" -Encoding utf8
Step 2: Find Your Cloud Run Container Image
Your Cloud Run service uses a specific container image. To find it:
Via gcloud CLI (recommended):
bash
gcloud run services describe YOUR-SERVICE-NAME \ --region=YOUR-REGION \ --project=YOUR-PROJECT \ --format='value(spec.template.spec.containers[0].image)'```This returns the full image URL, something like:```europe-west2-docker.pkg.dev/your-project/cloud-run-source-deploy/your-service@sha256:abc123...
Via Google Cloud Console:
- Navigate to Cloud Run in the console
- Click your service
- Go to the “Revisions” tab
- Look for “Container image URL”
The @sha256:... digest is important—it ensures you’re copying the exact image currently running in production.
Step 3: Set Up Scaleway Container Registry
Create a Scaleway Account
- Sign up at https://console.scaleway.com/
- Complete email verification
- Navigate to the console
Create a Container Registry Namespace
- Go to Containers → Container Registry
- Click Create namespace
- Choose a region (Paris, Amsterdam, or Warsaw)
- Important: Choose the same region where you’ll deploy your containers
- Enter a namespace name (e.g.,
my-containers,production)- Must be unique within that region
- Lowercase, numbers, and hyphens only
- Set Privacy to Private
- Click Create
Your registry URL will be: rg.fr-par.scw.cloud/your-namespace
Create API Credentials
- Click your profile → API Keys (or visit https://console.scaleway.com/iam/api-keys)
- Click Generate API Key
- Give it a name (e.g., “container-migration”)
- Save the Secret Key securely—it’s only shown once
- Note both the Access Key and Secret Key
Step 4: Copy Your Container Image
Now comes the magic—copying your container directly from Google to Scaleway without downloading it locally.
Authenticate and Copy:
powershell
# Set your Scaleway secret key as environment variable (more secure)$env:SCW_SECRET_KEY = "your-scaleway-secret-key-here"# Copy the image directly between registriesskopeo copy ` --src-creds="oauth2accesstoken:$(gcloud auth print-access-token)" ` --dest-creds="nologin:$env:SCW_SECRET_KEY" ` docker://europe-west2-docker.pkg.dev/your-project/cloud-run-source-deploy/your-service@sha256:abc123... ` docker://rg.fr-par.scw.cloud/your-namespace/your-service:latest```### What's Happening:- `--src-creds`: Authenticates with Google using your gcloud session- `--dest-creds`: Authenticates with Scaleway using your API key- Source URL: Your Google Artifact Registry image- Destination URL: Your Scaleway Container RegistryThe transfer happens directly between registries—your Windows machine just orchestrates it. Even a multi-GB container copies in minutes.### Verify the Copy:1. Go to https://console.scaleway.com/registry/namespaces2. Click your namespace3. You should see your service image listed with the `latest` tag## Step 5: Deploy to Scaleway Serverless Containers### Create a Serverless Container Namespace:1. Navigate to **Containers** → **Serverless Containers**2. Click **Create namespace**3. Choose the **same region** as your Container Registry4. Give it a name (e.g., `production-services`)5. Click **Create**### Deploy Your Container:1. Click **Create container**2. **Image source**: Select "Scaleway Container Registry"3. Choose your namespace and image4. **Configuration**: - **Port**: Set to the port your app listens on (usually 8080 for Cloud Run apps) - **Environment variables**: Copy any env vars from Cloud Run - **Resources**: - Memory: Start with what you used in Cloud Run - vCPU: 0.5-1 vCPU is typical - **Scaling**: - **Min scale**: `0` (enables scale-to-zero, just like Cloud Run) - **Max scale**: Set based on expected traffic (e.g., 10)5. Click **Deploy container**### Get Your Endpoint:After deployment (1-2 minutes), you'll receive an HTTPS endpoint:```https://your-container-namespace-xxxxx.functions.fnc.fr-par.scw.cloud
This is your public API endpoint—no API Gateway needed, SSL included for free.
Step 6: Test Your Service
powershell
# Test the endpointInvoke-WebRequest -Uri "https://your-container-url.functions.fnc.fr-par.scw.cloud/your-endpoint"
Your application should respond identically to how it did on Cloud Run.
Understanding the Cost Comparison
Google Cloud Run Pricing (Typical):
- vCPU: $0.00002400/vCPU-second
- Memory: $0.00000250/GB-second
- Requests: $0.40 per million
- Plus: API Gateway, Load Balancer, or other routing costs
Scaleway Serverless Containers:
- vCPU: €0.00001/vCPU-second (€1.00 per 100k vCPU-s)
- Memory: €0.000001/GB-second (€0.10 per 100k GB-s)
- Requests: Free (no per-request charges)
- HTTPS endpoint: Free (included)
- Free Tier: 200k vCPU-seconds + 400k GB-seconds per month
Example Calculation:
For an API handling 1 million requests/month, 200ms average response time, 1 vCPU, 2GB memory:
Google Cloud Run:
- vCPU: 1M × 0.2s × $0.000024 = $4.80
- Memory: 1M × 0.2s × 2GB × $0.0000025 = $1.00
- Requests: 1M × $0.0000004 = $0.40
- Total: ~$6.20/month
Scaleway:
- vCPU: 200k vCPU-s → Free (within free tier)
- Memory: 400k GB-s → Free (within free tier)
- Total: €0.00/month
Even beyond free tiers, Scaleway is typically 30-50% cheaper, with no surprise charges.
Key Differences to Be Aware Of
Similarities (Good News):
✅ Both use Knative under the hood ✅ Both support HTTP, HTTP/2, WebSocket, gRPC ✅ Both scale to zero automatically ✅ Both provide HTTPS endpoints ✅ Both support custom domains ✅ Both integrate with monitoring/logging
Differences:
- Cold start: Scaleway takes ~2-5 seconds (similar to Cloud Run)
- Idle timeout: Scaleway scales to zero after 15 minutes (vs. Cloud Run’s varies)
- Regions: Limited to EU (Paris, Amsterdam, Warsaw) vs. Google’s global presence
- Ecosystem: Smaller ecosystem than GCP (but rapidly growing)
When Scaleway Makes Sense:
- ✅ Your primary users/customers are in Europe
- ✅ GDPR compliance is critical
- ✅ You want to avoid US jurisdiction over your data
- ✅ You prefer transparent, predictable pricing
- ✅ You don’t need GCP-specific services (BigQuery, etc.)
When to Consider Carefully:
- ⚠️ You need global edge distribution (though you can use CDN)
- ⚠️ You’re heavily integrated with other GCP services
- ⚠️ You need GCP’s machine learning services
- ⚠️ Your customers are primarily in Asia/Americas
Additional Migration Considerations
Environment Variables and Secrets:
Scaleway offers Secret Manager integration. Copy your Cloud Run secrets:
- Go to Secret Manager in Scaleway
- Create secrets matching your Cloud Run environment variables
- Reference them in your container configuration
Custom Domains:
Both platforms support custom domains. In Scaleway:
- Go to your container settings
- Add custom domain
- Update your DNS CNAME to point to Scaleway’s endpoint
- SSL is handled automatically
Databases and Storage:
If you’re using Cloud SQL or Cloud Storage:
- Databases: Consider Scaleway’s Managed PostgreSQL/MySQL or Serverless SQL Database
- Object Storage: Scaleway Object Storage is S3-compatible
- Or: Keep using GCP services (cross-cloud is possible, but adds latency)
Monitoring and Logging:
Scaleway provides Cockpit (based on Grafana):
- Automatic logging for all Serverless Containers
- Pre-built dashboards
- Integration with alerts and metrics
- Similar to Cloud Logging/Monitoring
The Broader Picture: European Digital Sovereignty
This migration isn’t just about cost savings or technical features—it’s about control.
Why EU Companies Are Moving:
- Legal Protection: GDPR protections are stronger when data never leaves EU jurisdiction
- Political Risk: Reduces exposure to US government data requests under CLOUD Act
- Supply Chain Resilience: Diversification away from Big Tech dependency
- Supporting European Tech: Strengthens the European cloud ecosystem
- Future-Proofing: As digital sovereignty regulations increase, early movers are better positioned
The Economic Argument:
Every euro spent with European cloud providers:
- Stays in the European economy
- Supports European jobs and innovation
- Builds alternatives to US/Chinese tech dominance
- Strengthens Europe’s strategic autonomy
Conclusion: A Straightforward Path to Sovereignty
Migrating from Google Cloud Run to Scaleway Serverless Containers is technically simple—often taking just a few hours for a typical service. The containers are identical, the pricing is competitive, and the operational model is the same.
But beyond the technical benefits, there’s a strategic argument: as a European company, every infrastructure decision is a choice about where your data lives, who has access to it, and which ecosystem you’re supporting.
Scaleway (and other European cloud providers) aren’t perfect replacements for every GCP use case. But for containerized APIs and web services—which represent the majority of Cloud Run workloads—they’re absolutely production-ready alternatives that keep your infrastructure firmly within European jurisdiction.
In 2026’s geopolitical landscape, that’s not just a nice-to-have—it’s increasingly essential.
Resources
- Scaleway Serverless Containers: https://www.scaleway.com/en/serverless-containers/
- Scaleway Documentation: https://www.scaleway.com/en/docs/
- Skopeo Documentation: https://github.com/containers/skopeo
- European Cloud Providers: Research Scaleway, OVHcloud, Hetzner, and others
- EU Digital Sovereignty: European Commission digital strategy resources
Have you migrated your infrastructure back to Europe? Share your experience in the comments below.
Controlling Remote Chrome Instances with C# and the Chrome DevTools Protocol
If you’ve ever needed to programmatically interact with a Chrome browser running on a remote server—whether for web scraping, automated testing, or debugging—you’ve probably discovered that it’s not as straightforward as it might seem. In this post, I’ll walk you through how to connect to a remote Chrome instance using C# and the Chrome DevTools Protocol (CDP), with a practical example of retrieving all cookies, including those pesky HttpOnly cookies that JavaScript can’t touch.
Why Remote Chrome Control?
There are several scenarios where controlling a remote Chrome instance becomes invaluable:
- Server-side web scraping where you need JavaScript rendering but want to keep your scraping infrastructure separate from your application servers
- Cross-platform testing where you’re developing on Windows but testing on Linux environments
- Distributed automation where multiple test runners need to interact with centralized browser instances
- Debugging production issues where you need to inspect cookies, local storage, or network traffic on a live system
The Chrome DevTools Protocol gives us low-level access to everything Chrome can do—and I mean everything. Unlike browser automation tools that work through the DOM, CDP operates at the browser level, giving you access to cookies (including HttpOnly), network traffic, performance metrics, and much more.
The Challenge: Making Chrome Accessible Remotely
Chrome’s remote debugging feature is powerful, but getting it to work remotely involves some Linux networking quirks that aren’t immediately obvious. Let me break down the problem and solution.
The Problem
When you launch Chrome with the --remote-debugging-port flag, even if you specify --remote-debugging-address=0.0.0.0, Chrome often binds only to 127.0.0.1 (localhost). This means you can’t connect to it from another machine.
You can verify this by checking what Chrome is actually listening on:
netstat -tlnp | grep 9222
tcp 0 0 127.0.0.1:9222 0.0.0.0:* LISTEN 1891/chrome
See that 127.0.0.1? That’s the problem. It should be 0.0.0.0 to accept connections from any interface.
The Solution: socat to the Rescue
The elegant solution is to use socat (SOcket CAT) to proxy connections. We run Chrome on one port (localhost only), and use socat to forward a public-facing port to Chrome’s localhost port.
Here’s the setup on your Linux server:
# Start Chrome on localhost:9223
google-chrome \
--headless=new \
--no-sandbox \
--disable-gpu \
--remote-debugging-port=9223 \
--user-data-dir=/tmp/chrome-remote-debug &
# Use socat to proxy external 9222 to internal 9223
socat TCP-LISTEN:9222,fork,bind=0.0.0.0,reuseaddr TCP:127.0.0.1:9223 &
Now verify it’s working:
netstat -tlnp | grep 9222
tcp 0 0 0.0.0.0:9222 0.0.0.0:* LISTEN 2103/socat
netstat -tlnp | grep 9223
tcp 0 0 127.0.0.1:9223 0.0.0.0:* LISTEN 2098/chrome
Perfect! Chrome is safely listening on localhost only, while socat provides the public interface. This is actually more secure than having Chrome directly exposed.
Understanding the Chrome DevTools Protocol
Before we dive into code, let’s understand how CDP works. When Chrome runs with remote debugging enabled, it exposes two types of endpoints:
1. HTTP Endpoints (for discovery)
# Get browser version and WebSocket URL
curl http://your-server:9222/json/version
# Get list of all open pages/targets
curl http://your-server:9222/json
The /json/version endpoint returns something like:
{
"Browser": "Chrome/143.0.7499.169",
"Protocol-Version": "1.3",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36...",
"V8-Version": "14.3.127.17",
"WebKit-Version": "537.36...",
"webSocketDebuggerUrl": "ws://your-server:9222/devtools/browser/14706e92-5202-4651-aa97-a72d683bf88e"
}
2. WebSocket Endpoint (for control)
The webSocketDebuggerUrl is what we use to actually control Chrome. All CDP commands flow through this WebSocket connection using a JSON-RPC-like protocol.
Enter PuppeteerSharp
While you could manually handle WebSocket connections and craft CDP commands by hand (and I’ve done that with libraries like MasterDevs.ChromeDevTools), there’s an easier way: PuppeteerSharp.
PuppeteerSharp is a .NET port of Google’s Puppeteer library, providing a high-level API over CDP. The beauty is that it handles all the WebSocket plumbing, message routing, and protocol intricacies for you.
Here’s our complete C# application:
using System;
using System.Net.Http;
using System.Text.Json;
using System.Threading.Tasks;
using PuppeteerSharp;
namespace ChromeRemoteDebugDemo
{
class Program
{
static async Task Main(string[] args)
{
// Configuration
string remoteDebugHost = "xxxx.xxx.xxx.xxx";
int remoteDebugPort = 9222;
Console.WriteLine("=== Chrome Remote Debug - Cookie Retrieval Demo ===\n");
try
{
// Step 1: Get the WebSocket URL from Chrome
Console.WriteLine($"Connecting to http://{remoteDebugHost}:{remoteDebugPort}/json/version");
using var httpClient = new HttpClient();
string versionUrl = $"http://{remoteDebugHost}:{remoteDebugPort}/json/version";
string jsonResponse = await httpClient.GetStringAsync(versionUrl);
// Parse JSON to get webSocketDebuggerUrl
using JsonDocument doc = JsonDocument.Parse(jsonResponse);
JsonElement root = doc.RootElement;
string webSocketUrl = root.GetProperty("webSocketDebuggerUrl").GetString();
Console.WriteLine($"WebSocket URL: {webSocketUrl}\n");
// Step 2: Connect to Chrome using PuppeteerSharp
Console.WriteLine("Connecting to Chrome via WebSocket...");
var connectOptions = new ConnectOptions
{
BrowserWSEndpoint = webSocketUrl
};
var browser = await Puppeteer.ConnectAsync(connectOptions);
Console.WriteLine("Successfully connected!\n");
// Step 3: Get or create a page
var pages = await browser.PagesAsync();
IPage page;
if (pages.Length > 0)
{
page = pages[0];
Console.WriteLine($"Using existing page: {page.Url}");
}
else
{
page = await browser.NewPageAsync();
await page.GoToAsync("https://example.com");
}
// Step 4: Get ALL cookies (including HttpOnly!)
Console.WriteLine("\nRetrieving all cookies...\n");
var cookies = await page.GetCookiesAsync();
Console.WriteLine($"Found {cookies.Length} cookie(s):\n");
foreach (var cookie in cookies)
{
Console.WriteLine($"Name: {cookie.Name}");
Console.WriteLine($"Value: {cookie.Value}");
Console.WriteLine($"Domain: {cookie.Domain}");
Console.WriteLine($"Path: {cookie.Path}");
Console.WriteLine($"Secure: {cookie.Secure}");
Console.WriteLine($"HttpOnly: {cookie.HttpOnly}"); // ← This is the magic!
Console.WriteLine($"SameSite: {cookie.SameSite}");
Console.WriteLine($"Expires: {(cookie.Expires == -1 ? "Session" : DateTimeOffset.FromUnixTimeSeconds((long)cookie.Expires).ToString())}");
Console.WriteLine(new string('-', 80));
}
await browser.DisconnectAsync();
Console.WriteLine("\nDisconnected successfully.");
}
catch (Exception ex)
{
Console.WriteLine($"\n❌ ERROR: {ex.Message}");
}
}
}
}
The Key Insight: HttpOnly Cookies
Here’s what makes this approach powerful: page.GetCookiesAsync() returns ALL cookies, including HttpOnly ones.
In a normal web page, JavaScript cannot access HttpOnly cookies—that’s the whole point of the HttpOnly flag. It’s a security feature that prevents XSS attacks from stealing session tokens. But when you’re operating at the CDP level, you’re not bound by JavaScript’s restrictions. You’re talking directly to Chrome’s internals.
This is incredibly useful for:
- Session management in automation: You can extract session cookies from one browser session and inject them into another
- Security testing: Verify that sensitive cookies are properly marked HttpOnly
- Debugging authentication issues: See exactly what cookies are being set by your backend
- Web scraping: Maintain authenticated sessions across multiple scraper instances
Setting Up the Project
Create a new console application:
dotnet new console -n ChromeRemoteDebugDemo
cd ChromeRemoteDebugDemo
Add PuppeteerSharp:
dotnet add package PuppeteerSharp
Your .csproj should look like:
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net8.0</TargetFramework>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="PuppeteerSharp" Version="20.2.4" />
</ItemGroup>
</Project>
Running the Demo
On your Linux server:
# Install socat if needed
apt-get install socat -y
# Start Chrome on internal port 9223
google-chrome \
--headless=new \
--no-sandbox \
--disable-gpu \
--remote-debugging-port=9223 \
--user-data-dir=/tmp/chrome-remote-debug &
# Proxy external 9222 to internal 9223
socat TCP-LISTEN:9222,fork,bind=0.0.0.0,reuseaddr TCP:127.0.0.1:9223 &
On your Windows development machine:
dotnet run
You should see output like:
=== Chrome Remote Debug - Cookie Retrieval Demo ===
Connecting to http://xxxx.xxx.xxx.xxx:9222/json/version
WebSocket URL: ws://xxxx.xxx.xxx.xxx:9222/devtools/browser/14706e92-5202-4651-aa97-a72d683bf88e
Connecting to Chrome via WebSocket...
Successfully connected!
Using existing page: https://example.com
Retrieving all cookies...
Found 2 cookie(s):
Name: _ga
Value: GA1.2.123456789.1234567890
Domain: .example.com
Path: /
Secure: True
HttpOnly: False
SameSite: Lax
Expires: 2026-12-27 10:30:45
--------------------------------------------------------------------------------
Name: session_id
Value: abc123xyz456
Domain: example.com
Path: /
Secure: True
HttpOnly: True ← Notice this!
SameSite: Strict
Expires: Session
--------------------------------------------------------------------------------
Disconnected successfully.
Security Considerations
Before you deploy this in production, consider these security implications:
1. Firewall Configuration
Only expose port 9222 to trusted networks. If you’re running this on a cloud server:
# Allow only your specific IP
sudo ufw allow from YOUR.IP.ADDRESS to any port 9222
Or better yet, use an SSH tunnel and don’t expose the port at all:
# On Windows, create a tunnel
ssh -N -L 9222:localhost:9222 user@remote-server
# Then connect to localhost:9222 in your code
2. Authentication
The Chrome DevTools Protocol has no built-in authentication. Anyone who can connect to the debugging port has complete control over Chrome. This includes:
- Reading all page content
- Executing arbitrary JavaScript
- Accessing all cookies (as we’ve demonstrated)
- Intercepting and modifying network requests
In production, you should:
- Use SSH tunnels instead of exposing the port
- Run Chrome in a sandboxed environment
- Use short-lived debugging sessions
- Monitor for unauthorized connections
3. Resource Limits
A runaway Chrome instance can consume significant resources. Consider:
# Limit Chrome's memory usage
google-chrome --headless=new \
--max-old-space-size=512 \
--remote-debugging-port=9223 \
--user-data-dir=/tmp/chrome-remote-debug
Beyond Cookies: What Else Can You Do?
The Chrome DevTools Protocol is incredibly powerful. Here are some other things you can do with this same setup:
Take Screenshots
await page.ScreenshotAsync("/path/to/screenshot.png");
Monitor Network Traffic
await page.SetRequestInterceptionAsync(true);
page.Request += (sender, e) =>
{
Console.WriteLine($"Request: {e.Request.Url}");
e.Request.ContinueAsync();
};
Execute JavaScript
var title = await page.EvaluateExpressionAsync<string>("document.title");
Modify Cookies
await page.SetCookieAsync(new CookieParam
{
Name = "test",
Value = "123",
Domain = "example.com",
HttpOnly = true, // Can set HttpOnly from CDP!
Secure = true
});
Emulate Mobile Devices
await page.EmulateAsync(new DeviceDescriptorOptions
{
Viewport = new ViewPortOptions { Width = 375, Height = 667 },
UserAgent = "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0 like Mac OS X)"
});
Comparing Approaches
You might be wondering how this compares to other approaches:
PuppeteerSharp vs. Selenium
Selenium uses the WebDriver protocol, which is a W3C standard but higher-level and more abstracted. PuppeteerSharp/CDP gives you lower-level access to Chrome specifically.
- Selenium: Better for cross-browser testing, more stable API
- PuppeteerSharp: More powerful Chrome-specific features, faster, lighter weight
PuppeteerSharp vs. Raw CDP Libraries
You could use libraries like MasterDevs.ChromeDevTools or ChromeProtocol for more direct CDP access:
// With MasterDevs.ChromeDevTools
var session = new ChromeSession(webSocketUrl);
var cookies = await session.SendAsync(new GetCookiesCommand());
Low-level CDP libraries:
- Pros: More control, can use experimental CDP features
- Cons: More verbose, have to handle protocol details
PuppeteerSharp:
- Pros: High-level API, actively maintained, comprehensive documentation
- Cons: Abstracts away some CDP features
For most use cases, PuppeteerSharp hits the sweet spot between power and ease of use.
Troubleshooting Common Issues
“Could not connect to Chrome debugging endpoint”
Check firewall:
sudo ufw status
sudo iptables -L -n | grep 9222
Verify Chrome is running:
ps aux | grep chrome
netstat -tlnp | grep 9222
Test locally first:
curl http://localhost:9222/json/version
“No cookies found”
This is normal if the page hasn’t set any cookies. Navigate to a site that does:
await page.GoToAsync("https://github.com");
var cookies = await page.GetCookiesAsync();
Chrome crashes or hangs
Add more stability flags:
google-chrome \
--headless=new \
--no-sandbox \
--disable-gpu \
--disable-dev-shm-usage \
--disable-setuid-sandbox \
--remote-debugging-port=9223 \
--user-data-dir=/tmp/chrome-remote-debug
Real-World Use Case: Session Management
Here’s a practical example of how I’ve used this in production—managing authenticated sessions for web scraping:
public class SessionManager
{
private readonly string _remoteChrome;
public async Task<CookieParam[]> LoginAndGetSession(string username, string password)
{
var browser = await ConnectToRemoteChrome();
var page = await browser.NewPageAsync();
// Perform login
await page.GoToAsync("https://example.com/login");
await page.TypeAsync("#username", username);
await page.TypeAsync("#password", password);
await page.ClickAsync("#login-button");
await page.WaitForNavigationAsync();
// Extract all cookies (including HttpOnly session tokens!)
var cookies = await page.GetCookiesAsync();
await browser.DisconnectAsync();
// Store these cookies for later use
return cookies;
}
public async Task ReuseSession(CookieParam[] cookies)
{
var browser = await ConnectToRemoteChrome();
var page = await browser.NewPageAsync();
// Inject the saved cookies
await page.SetCookieAsync(cookies);
// Now you're authenticated!
await page.GoToAsync("https://example.com/dashboard");
// Do your work...
}
}
This allows you to:
- Log in once in a “master” browser
- Extract the session cookies (including HttpOnly auth tokens)
- Distribute those cookies to multiple scraper instances
- All scrapers are now authenticated without re-logging in
Conclusion
The Chrome DevTools Protocol opens up a world of possibilities for browser automation and debugging. By combining it with PuppeteerSharp and a bit of Linux networking knowledge, you can:
- Control Chrome instances running anywhere on your network
- Access all browser data, including HttpOnly cookies
- Build powerful automation and testing tools
- Debug production issues remotely
The key takeaways:
- Use socat to proxy Chrome’s localhost debugging port to external interfaces
- PuppeteerSharp provides the easiest way to interact with CDP from C#
- CDP gives you superpowers that normal JavaScript can’t access
- Security matters—only expose debugging ports to trusted networks
The complete code from this post is available on GitHub (replace with your actual link). If you found this useful, consider giving it a star!
Have you used the Chrome DevTools Protocol in your projects? What creative uses have you found for it? Drop a comment below—I’d love to hear your experiences!
Further Reading
- Chrome DevTools Protocol Documentation
- PuppeteerSharp Documentation
- Puppeteer (Node.js) Guide
- socat Man Page
Tags: C#, Chrome, DevTools Protocol, PuppeteerSharp, Web Automation, Browser Automation, Linux, socat
How to Check All AWS Regions for Deprecated Python 3.9 Lambda Functions (PowerShell Guide)
If you’ve received an email from AWS notifying you that Python 3.9 is being deprecated for AWS Lambda, you’re not alone. As runtimes reach End-Of-Life, AWS sends warnings so you can update your Lambda functions before support officially ends.
The key question is:
How do you quickly check every AWS region to see where you’re still using Python 3.9?
AWS only gives you a single-region example in their email, but many teams have functions deployed globally. Fortunately, you can automate a full multi-region check using a simple PowerShell script.
This post shows you exactly how to do that.
🚨 Why You Received the Email
AWS is ending support for Python 3.9 in AWS Lambda.
After the deprecation dates:
- No more security patches
- No AWS technical support
- You won’t be able to create/update functions using Python 3.9
- Your functions will still run, but on an unsupported runtime
To avoid risk, you should upgrade these functions to Python 3.10, 3.11, or 3.12.
But first, you need to find all the functions using Python 3.9 — across all regions.
✔️ Prerequisites
Make sure you have:
- AWS CLI installed
- AWS credentials configured (via
aws configure) - Permissions to run:
lambda:ListFunctionsec2:DescribeRegions
🧪 Step 1 — Verify AWS CLI Access
Run this to confirm your CLI is working:
aws sts get-caller-identity --region eu-west-1
If it returns your AWS ARN, you’re good to go.
If you see “You must specify a region”, set a default region:
aws configure set region eu-west-1
📝 Step 2 — PowerShell Script to Check Python 3.9 in All Regions
Save this as aws-lambda-python39-check.ps1 (or any name you prefer):
# Get all AWS regions (forcing region so the call always works)
$regions = (aws ec2 describe-regions --region us-east-1 --query "Regions[].RegionName" --output text) -split "\s+"
foreach ($region in $regions) {
Write-Host "Checking region: $region ..."
$functions = aws lambda list-functions `
--region $region `
--query "Functions[?Runtime=='python3.9'].FunctionArn" `
--output text
if ($functions) {
Write-Host " → Found Python 3.9 functions:"
Write-Host " $functions"
} else {
Write-Host " → No Python 3.9 functions found."
}
}
This script does three things:
- Retrieves all AWS regions
- Loops through each region
- Prints any Lambda functions that still use Python 3.9
It handles the common AWS CLI error:
You must specify a region
by explicitly using --region us-east-1 when retrieving the region list.
▶️ Step 3 — Run the Script
Open PowerShell in the folder where your script is saved:
.\aws-lambda-python39-check.ps1
You’ll see output like:
Checking region: eu-west-1 ...
→ Found Python 3.9 functions:
arn:aws:lambda:eu-west-1:123456789012:function:my-old-function
Checking region: us-east-1 ...
→ No Python 3.9 functions found.
If no functions appear, you’re fully compliant.
🛠️ What to Do Next
For each function identified, update the runtime:
aws lambda update-function-configuration `
--function-name MyFunction `
--runtime python3.12
If you package dependencies manually (ZIP deployments), ensure you rebuild them using the new Python version.
🎉 Summary
AWS’s deprecation emails can be slightly alarming, but the fix is simple:
- Scan all regions
- Identify Python 3.9 Lambda functions
- Upgrade them in advance of the cutoff date
With the PowerShell script above, you can audit your entire AWS account in seconds.
Fixing .NET 8 HttpClient Permission Denied Errors on Google Cloud Run
If you’re deploying a .NET 8 application to Google Cloud Run and encountering a mysterious NetworkInformationException (13): Permission denied error when making HTTP requests, you’re not alone. This is a known issue that stems from how .NET’s HttpClient interacts with Cloud Run’s restricted container environment.
The Problem
When your .NET application makes HTTP requests using HttpClient, you might see an error like this:
System.Net.NetworkInformation.NetworkInformationException (13): Permission denied
at System.Net.NetworkInformation.NetworkChange.CreateSocket()
at System.Net.NetworkInformation.NetworkChange.add_NetworkAddressChanged(NetworkAddressChangedEventHandler value)
at System.Net.Http.HttpConnectionPoolManager.StartMonitoringNetworkChanges()
This error occurs because .NET’s HttpClient attempts to monitor network changes and handle advanced HTTP features like HTTP/3 and Alt-Svc (Alternative Services). To do this, it tries to create network monitoring sockets, which requires permissions that Cloud Run containers don’t have by default.
Cloud Run’s security model intentionally restricts certain system-level operations to maintain isolation and security. While this is great for security, it conflicts with .NET’s network monitoring behavior.
Why Does This Happen?
The .NET runtime includes sophisticated connection pooling and HTTP version negotiation features. When a server responds with an Alt-Svc header (suggesting alternative protocols or endpoints), .NET tries to:
- Monitor network interface changes
- Adapt connection strategies based on network conditions
- Support HTTP/3 where available
These features require low-level network access that Cloud Run’s sandboxed environment doesn’t permit.
The Solution
Fortunately, there’s a straightforward fix. You need to disable the features that require elevated network permissions by setting two environment variables:
Environment.SetEnvironmentVariable("DOTNET_SYSTEM_NET_DISABLEIPV6", "1");
Environment.SetEnvironmentVariable("DOTNET_SYSTEM_NET_HTTP_SOCKETSHTTPHANDLER_HTTP3SUPPORT", "false");
Place these lines at the very top of your Program.cs file, before any HTTP client initialization or web application builder creation.
What These Variables Do
- DOTNET_SYSTEM_NET_DISABLEIPV6: Disables IPv6 support, which also disables the network change monitoring that requires socket creation.
- DOTNET_SYSTEM_NET_HTTP_SOCKETSHTTPHANDLER_HTTP3SUPPORT: Explicitly disables HTTP/3 support, preventing .NET from trying to negotiate HTTP/3 connections.
Alternative Approaches
Option 1: Set in Dockerfile
You can bake these settings into your container image:
FROM mcr.microsoft.com/dotnet/aspnet:8.0
WORKDIR /app
# Disable network monitoring features
ENV DOTNET_SYSTEM_NET_DISABLEIPV6=1
ENV DOTNET_SYSTEM_NET_HTTP_SOCKETSHTTPHANDLER_HTTP3SUPPORT=false
COPY publish/ .
ENTRYPOINT ["dotnet", "YourApp.dll"]
Option 2: Set via Cloud Run Configuration
You can configure these as environment variables in your Cloud Run deployment:
gcloud run deploy your-service \
--image gcr.io/your-project/your-image \
--set-env-vars DOTNET_SYSTEM_NET_DISABLEIPV6=1,DOTNET_SYSTEM_NET_HTTP_SOCKETSHTTPHANDLER_HTTP3SUPPORT=false
Or through the Cloud Console when configuring your service’s environment variables.
Performance Impact
You might wonder if disabling these features affects performance. In practice:
- HTTP/3 isn’t widely used yet, and most services work perfectly fine with HTTP/2 or HTTP/1.1
- Network change monitoring is primarily useful for long-running desktop applications that move between networks (like a laptop switching from WiFi to cellular)
- In a Cloud Run container with a stable network environment, these features provide minimal benefit
The performance impact is negligible, and the tradeoff is well worth it for a working application.
Why It Works Locally But Fails in Cloud Run
This issue often surprises developers because their code works perfectly on their development machine. That’s because:
- Local development environments typically run with full system permissions
- Your local machine isn’t running in a restricted container
- Cloud Run’s security sandbox is much more restrictive than a typical development environment
This is a classic example of environment-specific behavior where security constraints in production expose issues that don’t appear during development.
Conclusion
The Permission denied error when using HttpClient in .NET 8 on Google Cloud Run is caused by the runtime’s attempt to use network monitoring features that aren’t available in Cloud Run’s restricted environment. The fix is simple: disable these features using environment variables.
This solution is officially recognized by the .NET team as the recommended workaround for containerized environments with restricted permissions, so you can use it with confidence in production.
Related Resources
Have you encountered other .NET deployment issues on Cloud Run? Feel free to share your experiences in the comments below.
Enhanced Italian Vehicle #API: VIN Numbers Now Available for Motorcycles
We’re excited to announce a significant enhancement to the Italian vehicle data API available through Targa.co.it. Starting today, our API responses now include Vehicle Identification Numbers (VIN) for motorcycle lookups, providing developers and businesses with more comprehensive vehicle data than ever before.
What’s New
The Italian vehicle API has been upgraded to return VIN numbers alongside existing motorcycle data. This enhancement brings motorcycle data parity with our car lookup service, ensuring consistent and complete vehicle information across all vehicle types.
Sample Response Structure
Here’s what you can expect from the enhanced API response for a motorcycle lookup:
json
{
"Description": "Yamaha XT 1200 Z Super Ténéré",
"RegistrationYear": "2016",
"CarMake": {
"CurrentTextValue": "Yamaha"
},
"CarModel": {
"CurrentTextValue": "XT 1200 Z Super Ténéré"
},
"EngineSize": {
"CurrentTextValue": "1199"
},
"FuelType": {
"CurrentTextValue": ""
},
"MakeDescription": {
"CurrentTextValue": "Yamaha"
},
"ModelDescription": {
"CurrentTextValue": "XT 1200 Z Super Ténéré"
},
"Immobiliser": {
"CurrentTextValue": ""
},
"Version": "ABS (2014-2016) 1199cc",
"ABS": "",
"AirBag": "",
"Vin": "JYADP041000002470",
"KType": "",
"PowerCV": "",
"PowerKW": "",
"PowerFiscal": "",
"ImageUrl": "http://www.targa.co.it/image.aspx/@WWFtYWhhIFhUIDEyMDAgWiBTdXBlciBUw6luw6lyw6l8bW90b3JjeWNsZQ=="
}
Why VIN Numbers Matter
Vehicle Identification Numbers serve as unique fingerprints for every vehicle, providing several key benefits:
Enhanced Vehicle Verification: VINs offer the most reliable method to verify a vehicle’s authenticity and specifications, reducing fraud in motorcycle transactions.
Complete Vehicle History: Access to VIN enables comprehensive history checks, insurance verification, and recall information lookup.
Improved Business Applications: Insurance companies, dealerships, and fleet management services can now build more robust motorcycle-focused applications with complete vehicle identification.
Regulatory Compliance: Many automotive business processes require VIN verification for legal and regulatory compliance.
Technical Implementation
The VIN field has been seamlessly integrated into existing API responses without breaking changes. The new "Vin" field appears alongside existing motorcycle data, maintaining backward compatibility while extending functionality.
Key Features:
- No Breaking Changes: Existing integrations continue to work unchanged
- Consistent Data Structure: Same JSON structure across all vehicle types
- Comprehensive Coverage: VIN data available for motorcycles registered in the Italian vehicle database
- Real-time Updates: VIN information reflects the most current data from official Italian vehicle registries
Getting Started
Developers can immediately begin utilizing VIN data in their applications. The API endpoint remains unchanged, and VIN information is automatically included in all motorcycle lookup responses where available.
For businesses already integrated with our Italian vehicle API, this enhancement provides immediate additional value without requiring any code changes. New integrations can take full advantage of complete motorcycle identification data from day one.
Use Cases
This enhancement opens up new possibilities for motorcycle-focused applications:
- Insurance Platforms: Accurate risk assessment and policy management
- Marketplace Applications: Enhanced listing verification and buyer confidence
- Fleet Management: Complete motorcycle inventory tracking
- Service Centers: Precise parts identification and service history management
- Regulatory Reporting: Compliance with Italian vehicle registration requirements
Looking Forward
This VIN integration for motorcycles represents our continued commitment to providing comprehensive Italian vehicle data. We’re constantly working to enhance our API capabilities and expand data coverage to better serve the automotive technology ecosystem.
The addition of VIN numbers to motorcycle data brings our Italian API to feature parity with leading international vehicle data providers, while maintaining the accuracy and reliability that Italian businesses have come to expect from Targa.co.it.
Ready to integrate enhanced motorcycle data into your application? Visit Targa.co.it to explore our Italian vehicle API documentation and get started with VIN-enabled motorcycle lookups today.
Porting a PHP OAuth Spotler Client to C#: Lessons Learned
Recently I had to integrate with Spotler’s REST API from a .NET application. Spotler provides a powerful marketing automation platform, and their API uses OAuth 1.0 HMAC-SHA1 signatures for authentication.
They provided a working PHP client, but I needed to port this to C#. Here’s what I learned (and how you can avoid some common pitfalls).
🚀 The Goal
We started with a PHP class that:
✅ Initializes with:
consumerKeyconsumerSecret- optional SSL certificate verification
✅ Creates properly signed OAuth 1.0 requests
✅ Makes HTTP requests with cURL and parses the JSON responses.
I needed to replicate this in C# so we could use it inside a modern .NET microservice.
🛠 The Port to C#
🔑 The tricky part: OAuth 1.0 signatures
Spotler’s API requires a specific signature format. It’s critical to:
- Build the signature base string by concatenating:
- The uppercase HTTP method (e.g.,
GET), - The URL-encoded endpoint,
- And the URL-encoded, sorted OAuth parameters.
- The uppercase HTTP method (e.g.,
- Sign it using HMAC-SHA1 with the
consumerSecretfollowed by&. - Base64 encode the HMAC hash.
This looks simple on paper, but tiny differences in escaping or parameter order will cause 401 Unauthorized.
💻 The final C# solution
We used HttpClient for HTTP requests, and HMACSHA1 from System.Security.Cryptography for signatures. Here’s what our C# SpotlerClient does:
✅ Generates the OAuth parameters (consumer_key, nonce, timestamp, etc).
✅ Creates the exact signature base string, matching the PHP implementation character-for-character.
✅ Computes the HMAC-SHA1 signature and Base64 encodes it.
✅ Builds the Authorization header.
✅ Sends the HTTP request, with JSON bodies if needed.
We also added better exception handling: if the API returns an error (like 401), we throw an exception that includes the full response body. This made debugging much faster.
🐛 Debugging tips for OAuth 1.0
- Print the signature base string.
It needs to match exactly what Spotler expects. Any stray spaces or wrong escaping will fail. - Double-check timestamp and nonce generation.
OAuth requires these to prevent replay attacks. - Compare with the PHP implementation.
We literally copied the signature generation line-by-line from PHP into C#, carefully mappingrawurlencodetoUri.EscapeDataString. - Turn off SSL validation carefully.
During development, you might disable certificate checks (ServerCertificateCustomValidationCallback), but never do this in production.
using System.Security.Cryptography;
using System.Text;
namespace SpotlerClient
{
public class SpotlerClient
{
private readonly string _consumerKey;
private readonly string _consumerSecret;
private readonly string _baseUrl = "https://restapi.mailplus.nl";
private readonly HttpClient _httpClient;
public SpotlerClient(string consumerKey, string consumerSecret, bool verifyCertificate = true)
{
_consumerKey = consumerKey;
_consumerSecret = consumerSecret;
var handler = new HttpClientHandler();
if (!verifyCertificate)
{
handler.ServerCertificateCustomValidationCallback = (sender, cert, chain, sslPolicyErrors) => true;
}
_httpClient = new HttpClient(handler);
}
public async Task<string> ExecuteAsync(string endpoint, HttpMethod method, string jsonData = null)
{
var request = new HttpRequestMessage(method, $"{_baseUrl}/{endpoint}");
var authHeader = CreateAuthorizationHeader(method.Method, endpoint);
request.Headers.Add("Accept", "application/json");
request.Headers.Add("Authorization", authHeader);
if (jsonData != null)
{
request.Content = new StringContent(jsonData, Encoding.UTF8, "application/json");
}
var response = await _httpClient.SendAsync(request);
if (!response.IsSuccessStatusCode)
{
var body = await response.Content.ReadAsStringAsync();
return body;
}
return await response.Content.ReadAsStringAsync();
}
private string CreateAuthorizationHeader(string httpMethod, string endpoint)
{
var timestamp = DateTimeOffset.UtcNow.ToUnixTimeSeconds().ToString();
var nonce = Guid.NewGuid().ToString("N");
var paramString = "oauth_consumer_key=" + Uri.EscapeDataString(_consumerKey) +
"&oauth_nonce=" + Uri.EscapeDataString(nonce) +
"&oauth_signature_method=" + Uri.EscapeDataString("HMAC-SHA1") +
"&oauth_timestamp=" + Uri.EscapeDataString(timestamp) +
"&oauth_version=" + Uri.EscapeDataString("1.0");
var sigBase = httpMethod.ToUpper() + "&" +
Uri.EscapeDataString(_baseUrl + "/" + endpoint) + "&" +
Uri.EscapeDataString(paramString);
var sigKey = _consumerSecret + "&";
var signature = ComputeHmacSha1Signature(sigBase, sigKey);
var authHeader = $"OAuth oauth_consumer_key=\"{_consumerKey}\", " +
$"oauth_nonce=\"{nonce}\", " +
$"oauth_signature_method=\"HMAC-SHA1\", " +
$"oauth_timestamp=\"{timestamp}\", " +
$"oauth_version=\"1.0\", " +
$"oauth_signature=\"{Uri.EscapeDataString(signature)}\"";
return authHeader;
}
private string ComputeHmacSha1Signature(string data, string key)
{
using var hmac = new HMACSHA1(Encoding.UTF8.GetBytes(key));
var hash = hmac.ComputeHash(Encoding.UTF8.GetBytes(data));
return Convert.ToBase64String(hash);
}
}
}
✅ The payoff
Once the signature was constructed precisely, authentication errors disappeared. We could now use the Spotler REST API seamlessly from C#, including:
- importing contact lists,
- starting campaigns,
- and fetching campaign metrics.
📚 Sample usage
var client = new SpotlerClient(_consumerKey, _consumerSecret, false);
var endpoint = "integrationservice/contact/email@gmail.com";
var json = client.ExecuteAsync(endpoint, HttpMethod.Get).GetAwaiter().GetResult();
🎉 Conclusion
Porting from PHP to C# isn’t always as direct as it looks — especially when it comes to cryptographic signatures. But with careful attention to detail and lots of testing, we managed to build a robust, reusable client.
If you’re facing a similar integration, feel free to reach out or clone this approach. Happy coding!
🚫 Why AWS SDK for S3 No Longer Works Smoothly with .NET Framework 4.8 — and How to Fix It
In 2024, more .NET developers are finding themselves in a strange situation: suddenly, tried-and-tested .NET Framework 4.8 applications that interact with Amazon S3 start throwing cryptic build errors or runtime exceptions. The culprit? The AWS SDK for .NET has increasingly shifted toward support for .NET Core / .NET 6+, and full compatibility with .NET Framework is eroding.
In this post, we’ll explain:
- Why this happens
- What errors you might see
- And how to remove the AWS SDK altogether and replace it with pure .NET 4.8-compatible code for downloading (and uploading) files from S3 using Signature Version 4.
🧨 The Problem: AWS SDK & .NET Framework 4.8
The AWS SDK for .NET (like AWSSDK.S3) now depends on modern libraries like:
System.Text.JsonSystem.BuffersSystem.Runtime.CompilerServices.UnsafeMicrosoft.Bcl.AsyncInterfaces
These dependencies were designed for .NET Core and later versions — not .NET Framework. While it was once possible to work around this with binding redirects and careful version pinning, the situation has become unstable and error-prone.
❗ Common Symptoms
You may see errors like:
Could not load file or assembly ‘System.Text.Json, Version=6.0.0.11’
Or:
Could not load file or assembly ‘System.Buffers, Version=4.0.5.0’
Or during build:
Warning: Unable to update auto-refresh reference ‘system.text.json.dll’
Even if you install the correct packages, you may end up needing to fight bindingRedirect hell, and still not get a working application.
✅ The Solution: Remove the AWS SDK
Fortunately, you don’t need the SDK to use S3. All AWS S3 requires is a properly signed HTTP request using AWS Signature Version 4, and you can create that yourself using standard .NET 4.8 libraries.
🔐 Downloading from S3 Without the AWS SDK
Here’s how you can download a file from S3 using HttpWebRequest and Signature Version 4.
✔️ The Key Points:
- You must include the
x-amz-content-sha256header (even for GETs!) - You sign the request using your AWS secret key
- No external packages required — works on plain .NET 4.8
🧩 Code Snippet
public static byte[] DownloadFromS3(string bucketName, string objectKey, string region, string accessKey, string secretKey)
{
var method = "GET";
var service = "s3";
var host = $"{bucketName}.s3.{region}.amazonaws.com";
var uri = $"https://{host}/{objectKey}";
var requestDate = DateTime.UtcNow;
var amzDate = requestDate.ToString("yyyyMMddTHHmmssZ");
var dateStamp = requestDate.ToString("yyyyMMdd");
var canonicalUri = "/" + objectKey;
var signedHeaders = "host;x-amz-content-sha256;x-amz-date";
var payloadHash = HashSHA256(string.Empty); // Required even for GET
var canonicalRequest = $"{method}\n{canonicalUri}\n\nhost:{host}\nx-amz-content-sha256:{payloadHash}\nx-amz-date:{amzDate}\n\n{signedHeaders}\n{payloadHash}";
var credentialScope = $"{dateStamp}/{region}/{service}/aws4_request";
var stringToSign = $"AWS4-HMAC-SHA256\n{amzDate}\n{credentialScope}\n{HashSHA256(canonicalRequest)}";
var signingKey = GetSignatureKey(secretKey, dateStamp, region, service);
var signature = ToHexString(HmacSHA256(signingKey, stringToSign));
var authorizationHeader = $"AWS4-HMAC-SHA256 Credential={accessKey}/{credentialScope}, SignedHeaders={signedHeaders}, Signature={signature}";
var request = (HttpWebRequest)WebRequest.Create(uri);
request.Method = method;
request.Headers["Authorization"] = authorizationHeader;
request.Headers["x-amz-date"] = amzDate;
request.Headers["x-amz-content-sha256"] = payloadHash;
try
{
using (var response = (HttpWebResponse)request.GetResponse())
using (var responseStream = response.GetResponseStream())
using (var memoryStream = new MemoryStream())
{
responseStream.CopyTo(memoryStream);
return memoryStream.ToArray();
}
}
catch (WebException ex)
{
using (var errorResponse = (HttpWebResponse)ex.Response)
using (var reader = new StreamReader(errorResponse.GetResponseStream()))
{
var errorText = reader.ReadToEnd();
throw new Exception($"S3 request failed: {errorText}", ex);
}
}
}
🔧 Supporting Methods
private static string HashSHA256(string text)
{
using (var sha256 = SHA256.Create())
{
return ToHexString(sha256.ComputeHash(Encoding.UTF8.GetBytes(text)));
}
}private static byte[] HmacSHA256(byte[] key, string data)
{
using (var hmac = new HMACSHA256(key))
{
return hmac.ComputeHash(Encoding.UTF8.GetBytes(data));
}
}private static byte[] GetSignatureKey(string secretKey, string dateStamp, string region, string service)
{
var kSecret = Encoding.UTF8.GetBytes("AWS4" + secretKey);
var kDate = HmacSHA256(kSecret, dateStamp);
var kRegion = HmacSHA256(kDate, region);
var kService = HmacSHA256(kRegion, service);
return HmacSHA256(kService, "aws4_request");
}private static string ToHexString(byte[] bytes)
{
return BitConverter.ToString(bytes).Replace("-", "").ToLowerInvariant();
}
📝 Uploading to S3 Without the AWS SDK
You can extend the same technique for PUT requests. The only differences are:
You calculate the SHA-256 hash of the file content
You include a Content-Type and Content-Length header
You use PUT instead of GET
Let me know in the comments if you’d like the full upload version — it follows the same Signature V4 pattern.
✅ Summary
Feature AWS SDK for .NET Manual Signature V4
.NET Framework 4.8 support ❌ Increasingly broken ✅ Fully supported
Heavy NuGet dependencies ✅ ❌ Minimal
Simple download/upload ✅ ✅ (with more code)
Presigned URLs ✅ 🟡 Manual support
Final Thoughts
If you’re stuck on .NET Framework 4.8 and running into weird AWS SDK issues — you’re not alone. But you’re not stuck either. Dropping the SDK and using HTTP + Signature V4 is entirely viable, especially for simple tasks like uploading/downloading S3 files.
Let me know if you’d like a full upload example, presigned URL generator, or if you’re considering migrating to .NET 6+.
Network Programming in .NET 