DisposeAsync hangs within DistributedApplicationTestingBuilder

[afscrome]

I've been encountering a number of cases where our Tests appear to be done, yet the tests are still running in the test runner. Further investigation suggests this hanging is happening when disposing the app built through DistributedApplicationTestingBuilder. I've sometimes seen delays of several minutes here.

The criteria seem to be

• Running within DistributedApplicationTestingBuilder (doesn't replicate using plain DistributedApplication.CreateBuilder)
• Needs to have a resource with waits for another resource AND has it's own health check

A minimal repro for this is below . I expect the time between the two beeps to be negligible, however it is 10+ seconds.

AppHost:

var builder = DistributedApplication.CreateBuilder();

var failsToStart = builder.AddContainer("failsToStart", "does/not/exist");

var dependency = builder.AddExecutable("app", "pwsh", ".")
    .WaitFor(failsToStart)
    .WithHttpEndpoint()
    .WithHttpHealthCheck();

using var app = builder.Build();
app.Run();

Test:

    [Test]
    public async Task Test()
    {
        var appHost = await DistributedApplicationTestingBuilder.CreateAsync<Projects.AspireApp2_AppHost>();

        await using var app = await appHost.BuildAsync();

        try
        {
            await app.StartAsync().WaitAsync(TimeSpan.FromSeconds(1));
        }
        catch
        {
            Console.Beep();
            await app.DisposeAsync();
            Console.Beep();

        }
    }

Looking at console output from the test and you can see there is a ~13 second gap between Dependency resource 'initial' failed to start. and Failed to create resource app. Given the 1 second timeout in the above code, I'd expect this to be no more than that.

info: Aspire.Hosting.DistributedApplication[0]
      Aspire version: 9.0.0+01ed51919f8df692ececce51048a140615dc759d
info: Aspire.Hosting.DistributedApplication[0]
      Distributed application starting.
info: Aspire.Hosting.DistributedApplication[0]
      Application host directory is: S:\AspireApp2\AspireApp2.AppHost
fail: Aspire.Hosting.Dcp.dcpctrl.ContainerReconciler[0]
      could not create the container	{"Container": {"name":"initial-danpsbmy-621867ae"}, "Reconciliation": 3, "error": "object not found\ncontainer not found\ndocker command 'CreateContainer' returned with non-zero exit code 1: command output: Stdout: '' Stderr: 'invalid reference format\n'"}

info: AspireApp2.AppHost.Resources.app[0]
      1: 2025-01-17T13:22:01.1537805Z Waiting for resource 'initial' to enter the 'Running' state.
fail: AspireApp2.AppHost.Resources.app[0]
      2: 2025-01-17T13:22:14.9408991Z Dependency resource 'initial' failed to start.
fail: AspireApp2.AppHost.Resources.app[0]
      3: 2025-01-17T13:22:14.9432384Z Failed to create resource app
Aspire.Hosting.DistributedApplicationException: Dependency resource 'initial' failed to start.
         at Aspire.Hosting.ApplicationModel.ResourceNotificationService.WaitUntilHealthyAsync(IResource resource, IResource dependency, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/ApplicationModel/ResourceNotificationService.cs:line 139
         at Aspire.Hosting.ApplicationModel.ResourceNotificationService.WaitForDependenciesAsync(IResource resource, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/ApplicationModel/ResourceNotificationService.cs:line 257
         at Aspire.Hosting.DistributedApplicationBuilder.<>c.<<-ctor>b__32_0>d.MoveNext() in /_/src/Aspire.Hosting/DistributedApplicationBuilder.cs:line 174
      --- End of stack trace from previous location ---
         at Aspire.Hosting.Eventing.DistributedApplicationEventing.<>c__DisplayClass4_0`1.<<Subscribe>b__0>d.MoveNext() in /_/src/Aspire.Hosting/Eventing/DistributedApplicationEventing.cs:line 82
      --- End of stack trace from previous location ---
         at Aspire.Hosting.Eventing.DistributedApplicationEventing.PublishAsync[T](T event, EventDispatchBehavior dispatchBehavior, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Eventing/DistributedApplicationEventing.cs:line 69
         at Aspire.Hosting.Dcp.ApplicationExecutor.<CreateExecutablesAsync>g__CreateResourceExecutablesAsyncCore|62_0(IResource resource, IEnumerable`1 executables, CancellationToken cancellationToken) in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 1216
info: Aspire.Hosting.DistributedApplication[0]
      Distributed application started. Press Ctrl+C to shut down.

Interestingly this only happens through the test host - replace the app.Run() call in Program.cs with the following, and DisposeAsync is near instant

try
{
    await app.StartAsync().WaitAsync(TimeSpan.FromSeconds(1));
}
catch
{
    Console.Beep();
    await app.DisposeAsync();
    Console.Beep();
}

May also be some relationship with #7009 as they both share the criteria of mixing health checks with waitfor, although this issue does only reproduce under the test host.