Add hooks to debug OpenSSL memory allocations

[rzikm]

This ressurects #101626. CC: @wfurt.

Changes since his PR:

• Moved some code around, the managed part now lives in Interop.Crypto.
• Added ReaderWriterLock to prevent program crashes due to AV

Open questions:

• Should the functionality (or some part of it) be under #if DEBUG?
• Does OpenSSL have requirements on memory alignment?

We had several cases when users complained about large memory use. For than native it is quite difficult to figure out where the memory goes. This PR aims to make that somewhat easier.

OpenSSL provides hooks for memory function so this PR adds switch to optimally hook into that.
The only one caveat that the CRYPTO_set_mem_functions works only if called before any allocations e.g. it needs to be done very early in the process. So I end up putting into initialization process.

The simple use pattern is something like

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Net.Http;
using System.Net.Security;
using System.Reflection;
using System.Runtime.InteropServices;

// Environment variable needs to be set before launching the process, as otherwise the native layer will
// not see the environment variable
// Environment.SetEnvironmentVariable("DOTNET_SYSTEM_NET_SECURITY_OPENSSL_MEMORY_DEBUG", "1");
//
// Once enabled, the functionality can be accessed by following methods on the Interop.Crypto class:
// - GetOpenSslAllocatedMemory - Gets the total amount of memory allocated by OpenSSL
// - GetOpenSslAllocationCount - Gets the number of allocations made by OpenSSL
// - EnableTracking/DisableTracking - toggles tracking of individual (outstanding) allocations via internal dictionary. This is rather expensive and may affect performance.
// - GetIncrementalAllocations - Gets a list of all allocations since the last EnableTracking call. Warning: Internal lock will prevent OpenSSL from allocating from other threads.

var ci = typeof(SslStream).Assembly.GetTypes().First(t => t.Name == "Crypto");
ci.InvokeMember("EnableTracking", BindingFlags.InvokeMethod | BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Static, null, null, null);

HttpClient client = new HttpClient();
await client.GetAsync("https://www.google.com");

using var process = Process.GetCurrentProcess();
Console.WriteLine($"Bytes known to GC [{GC.GetTotalMemory(false)}], process working set [{process.WorkingSet64}]");
Console.WriteLine("OpenSSL memory {0}", ci.InvokeMember("GetOpenSslAllocatedMemory", BindingFlags.InvokeMethod | BindingFlags.Public | BindingFlags.Static, null, null, null));

(UIntPtr, int, string)[] allAllocations = ((UIntPtr, int, string)[])ci.InvokeMember("GetIncrementalAllocations", BindingFlags.InvokeMethod | BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Static | BindingFlags.Instance, null, null, null);
Dictionary<string, int> allocationSizes = new Dictionary<string, int>();
for (int j = 0; j < allAllocations.Length; j++)
{
    var (ptr, size, name) = allAllocations[j];
    CollectionsMarshal.GetValueRefOrAddDefault(allocationSizes, name, out _) += size;
}

System.Console.WriteLine("Total allocated OpenSSL memory by location");
foreach (var (name, total) in allocationSizes.OrderByDescending(kvp => kvp.Value))
{
    Console.WriteLine($"{total,10} {name}");
}

Dumping large allocation data set is slow and expensive. It is done under local so it blocks all other OpenSSL allocations. I feel this is ok for now but it should be used with caution. I also feel that access through Reflection is OK since this is only last resort debug hook e.g. it does not need stable API and convenient access.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[Copilot]

Copilot reviewed 2 out of 8 changed files in this pull request and generated 1 comment.

Files not reviewed (6)

• src/native/libs/System.Security.Cryptography.Native/apibridge_30.h: Language not supported
• src/native/libs/System.Security.Cryptography.Native/entrypoints.c: Language not supported
• src/native/libs/System.Security.Cryptography.Native/openssl.c: Language not supported
• src/native/libs/System.Security.Cryptography.Native/openssl.h: Language not supported
• src/native/libs/System.Security.Cryptography.Native/opensslshim.h: Language not supported
• src/native/libs/System.Security.Cryptography.Native/pal_ssl.c: Language not supported

[jkotas]

Calling managed code from the free method is going to have reliability issues. The problem is that OpenSSL can call free very late during thread destruction after the .NET runtime cleaned up its own per-thread data structures. It will cause the per-thread .NET runtime data structures to be partially reinitialized and stuck in a weird state. This weird state will manifest as memory leak and a hard to diagnose crash later.

Take a look at #110442 to see the crash caused by calling back managed code from OpenSSL malloc/free and what it took to diagnose it.

[wfurt]

is there some way to know if runtime or thread is going down @jkotas? I don't envision this to be enabled often or in production and we can also document the dragons ... or keep it hidden and make it debug hook for us.

[jkotas]

I do not think there is a reliable way to detect that the thread is in the middle of being shutdown.

[jkoritzinsky]

Yeah I'd recommend (if you want to get some sort of memory tracking in) to instead write any memory tracking code in native code. It's not as nice as managed code, but as @jkotas points out, there's no reliable way to do this in managed code.