Feature request: keep all InProc dependencies via sampling overrides

[trask]

For example, to support keeping all Inproc dependencies (INTERNAL spans), but dropping all other dependencies (CLIENT spans).

An option to support this would be to introduce a new telemetryType of dependency/inproc, e.g.

{
  "sampling": {
    "percentage": 100,
    "overrides": [
      {
        "telemetryType": "dependency/inproc",
        "percentage": 100
      },
      {
        "telemetryType": "dependency",
        "percentage": 0
      }
    ]
  }
}

[progxaker]

Background

Application Insights allows to blacklist dependencies - spans with the INTERNAL, PRODUCER, CLIENT and so on types, i.e. exclude the unnecessary ones and keep the necessary ones. There is no problem with this.

{
  "sampling": {
    "percentage": 100,
    "overrides": [
      {
        "telemetryType": "dependency",
        "attributes": [
          {
            "key": "db.system",
            "value": "redis",
            "matchType": "strict"
          }
        ],
        "percentage": 0
      }
    ]
  }
}

If the client wants to maintain a whitelist of dependencies, there may be a problem with span with the INTERNAL kind (InProc type). They can be created using the Span class or the @WithSpan annotation and initially have no attributes that can be used to select and preserve them.

For example, when using the following configuration:

{
  "preview": {
    "captureControllerSpans": true // https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-config#autocollect-inproc-dependencies-preview
  },
  "sampling": {
    "percentage": 0,
    "overrides": [
      {
        "telemetryType": "dependency",
        "attributes": [
          {
            "key": "db.system",
            "value": "redis",
            "matchType": "strict"
          }
        ],
        "percentage": 100
      }
    ]
  }
}

Redis dependencies will be preserved, all others including custom/user (InProc) will be captured (captureControllerSpans) but filtered by sampling. To preserve them, it's necessary to apply an attribute to each custom spans, which can be overhead in a complex product.

P.S. The following configuration will save all spans, not just InProc, which also does not satisfy the whitelist definition.

{
  "preview": {
    "captureControllerSpans": true // https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-config#autocollect-inproc-dependencies-preview
  },
  "sampling": {
    "percentage": 0,
    "overrides": [
      {
        "telemetryType": "dependency",
        "percentage": 100
      }
    ]
  }
}

[progxaker]

I think sampling on a percentage basis is not a good solution, as you also pointed out in #3654 (comment).

if you sample out the InProc SecretClient.getSecretSync, it will cascade down and sample out the underlying HTTP request as well, which I don't believe is what you want.

I think it's better to provide a switch, like "sampling.keepInProcDependencies" or similar, that allows to toggle keep or exclude InProc dependencies. But in this case, it's necessary to make sure that if the user sets keepInProcDependecies to false and sets the override with the product=test attribute, INTERNAL spans with the specified attribute are preserved.

{
  "sampling": {
    "percentage": 0,
    "keepInProcDependencies": false,
    "overrides": [
      {
        "telemetryType": "dependency",
        "attributes": [
          {
            "key": "product",
            "value": "test",
            "matchType": "strict"
          }
        ],
        "percentage": 100
      }
    ]
  }
}

[trask]

I think it's better to provide a switch, like "sampling.keepInProcDependencies" or similar, that allows to toggle keep or exclude InProc dependencies.

I didn't follow how this is different from providing a new telemetryType of dependencies/inproc that can be configured with either percentage 0 or 100 depending on your need? thanks

[progxaker]

how this is different

If we set the percentage to 50%, only A - - - D will remain from trace A -> B -> C -> D, which looks illogical. I think the user wants to keep all InProc spans within the trace or exclude them (in the case of App Insights). I think being able to set the percentage would cause a burden on the support team to explain why B and C disappeared.

[trask]

not totally sure I'm following, can you confirm this what you're saying:

with the configuration below, only A - - - D will remain from trace A -> B -> C -> D? (resulting in broken trace)

{
  "sampling": {
    "percentage": 100,
    "overrides": [
      {
        "telemetryType": "dependency/inproc",
        "percentage": 0
      },
      {
        "telemetryType": "dependency",
        "percentage": 100
      }
    ]
  }
}

[progxaker]

In general yes, but with the specified configuration there will be no InProc spans as their percentage is 0. I meant that if the user sets a value greater than 0 and less than 100, the result will be a broken trace.

{
  "sampling": {
    "percentage": 100,
    "overrides": [
      {
        "telemetryType": "dependency/inproc",
        "percentage": 73
      },
      {
        "telemetryType": "dependency",
        "percentage": 100
      }
    ]
  }
}

[trask]

broken traces are (unfortunately) a "feature" of filtering out intermediate spans via sampling (i'm not sure how can be avoided?)

see open-telemetry/opentelemetry-specification#3867 for a possible future alternative