Monitor multiple applicationsΒΆ

In this guide we will modify the plugin described in extending plugin with error handling to make it possible to monitor more then one application (or "process group", as they are called in Dynatrace).

The plugin will monitor both of the test applications used so far. We will run the applications on different TCP ports and expect them to bind to the local IP address 127.0.0.1. We will start the demo applications listening on ports 8991 and 8992 respectively:

python -m oneagent_sdk.demo_app  -p 8991
python -m oneagent_sdk.demo_app_auth -p 8992

To begin, let's look at the entire plugin source. Then we'll go over the modifications that need to be performed. Here's the plugin source code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
'''
Demo plugin which shows how to monitor multiple process groups with single plugin.
'''
import requests
import requests.exceptions
import json
import logging
from ruxit.api.base_plugin import BasePlugin
from ruxit.api.snapshot import parse_port_bindings
from ruxit.api.exceptions import AuthException, ConfigException
from ruxit.api.snapshot import pgi_name

class DemoPluginMulti(BasePlugin):
    '''
    Class implementing plugin which shows how to monitor multiple process groups with single plugin.
    The group is to be monitored if its name starts with predefined string.
    '''
    _APPL_PREFIX='oneagent_sdk.demo_app'

    def query(self, **kwargs):
        '''
        Scan process snapshot for groups with name starting with given prefix, and
        for all matching this prefix find ports on which they are listening. Then use this port
        for building URL to query with http. The result is json with two variables which are
        put into results with entity id of given process group.
        '''
        config = kwargs["config"]
        user = config["user"]
        password = config["password"]
        # search process snapshot using criteria defined by lambda expression
        pgi_list = self.find_all_process_groups( lambda entry: entry.group_name.startswith(self._APPL_PREFIX))
        for pgi in pgi_list:
            pgi_id = pgi.group_instance_id
            self.logger.info( "Demo pgid=%x application=%s" % (pgi_id,pgi.group_name ))
            port = None
            for process in pgi.processes:
               port = process.properties.get("ListeningPorts", None)
               break
            if port is None:
                raise ValueError( "no port definition for process group with id=%d" % pgi_id )
            # build URL for quering
            url = "http://127.0.0.1:" + port
            self.logger.info( "using url %s for pgid=%x app=%s" % (url,pgi_id,pgi.group_name ))

            try:
                # query URL for results
                response = requests.get(url, auth=(user, password))
                if response.status_code == 401:
                    raise AuthException(response)
                stats = json.loads(response.content.decode())
            except requests.exceptions.ConnectTimeout as ex:
                raise ConfigException('Timeout on connecting with "%s"' % url) from ex
            except requests.exceptions.RequestException as ex:
                raise ConfigException('Unable to connect to "%s"' % url) from ex
            except json.JSONDecodeError as ex:
                raise ConfigException('Server response from %s is not json' % url) from ex

            # save received results
            self.results_builder.absolute(key='random', value=stats['random'], entity_id=pgi_id)
            self.results_builder.relative(key='counter', value=stats['counter'], entity_id=pgi_id)

And the corresponding plugin.json is almost identical. The only modified lines are:

{
  "version": "1.0",
  "requiredAgentVersion": "1.90",
  "name": "custom.python.demo_plugin_multi",
  "type": "python",
  "entity": "PROCESS_GROUP_INSTANCE",
  "processTypeNames": ["PYTHON"],
  "source": {
    "package": "demo_plugin_multi",
    "className": "DemoPluginMulti",
    "install_requires": ["requests>=2.6.0"],
    "activation": "Singleton"
  },
  "metrics": [
    {
      "timeseries": {
        "key": "random",
        "unit": "Count",
        "dimensions": []
      }
    },
    {
      "timeseries": {
        "key": "counter",
        "unit": "Count",
        "dimensions": []
      }
    }
  ],
  "configUI": {
    "displayName": "custom.python.demo_plugin_multi"
  },
  "properties": [
    {
      "key": "user",
      "type": "String"
    },
    {
      "key": "password",
      "type": "Password"
    }
  ]
}

Please refer to plugin.json reference for further information regarding properties and configUI elements, including more descriptive field names, ordering of fields, and more.

In our plugin we want to monitor all applications discovered by OneAgent that begin with the prefix oneagent_sdk.demo_app. After we start the demo applications, as explained above, web UI shows them as:

../_images/demo_03_applications.png

This means we want to monitor applications oneagent_sdk.demo_app and oneagent_sdk.demo_app_auth. Below is the fragment that searches for them in current processes snapshot:

        pgi_list = self.find_all_process_groups( lambda entry: entry.group_name.startswith(self._APPL_PREFIX))

In our case, the snapshot appears as follows (entries of interest are highlighted):

[
ProcessSnapshotEntry(
  group_id=12668946693571282165, 
  node_id=0, 
  group_instance_id=18165771568562771184, 
  process_type=29, 
  group_name='oneagent_sdk.demo_app_auth', 
  processes=[
    ProcessInfo(
      pid=10426, 
      process_name='python3.5', 
      properties={
        'CmdLine': '-m oneagent_sdk.demo_app_auth -p 8992', 
        'WorkDir': '/home/jj/ruxit/trunk/core/product/agent', 
        'PortBindings': '127.0.0.1_8992', 
        'ListeningPorts': '8992'
      }
    )
  ], 
  properties={}
), 
ProcessSnapshotEntry(
  group_id=17388020258730275021, 
  node_id=0, 
  group_instance_id=5122137151859959739, 
  process_type=29, 
  group_name='oneagent_sdk.demo_app', 
  processes=[
    ProcessInfo(
      pid=6586, 
      process_name='python3.5', 
      properties={
        'CmdLine': '-m oneagent_sdk.demo_app -p 8991', 
        'WorkDir': '/home/jj/ruxit/trunk/core/product/agent', 
        'PortBindings': '127.0.0.1_8991', 
        'ListeningPorts': '8991'
      }
    )
  ], 
  properties={}
)
]

Then, based on all found applications, we read from the snapshot the TCP port on which the given application is listening. Information about listening ports opened by applications is collected by OneAgent and made available for plugin agent in the process snapshot. We use this to create a URL that we will query with HTTP for counters of interest:

            for process in pgi.processes:
               port = process.properties.get("ListeningPorts", None)
               break
            if port is None:
                raise ValueError( "no port definition for process group with id=%d" % pgi_id )
            # build URL for quering
            url = "http://127.0.0.1:" + port

To learn more, please explore these additional resources: