Describe the bug: ... Events getting reported to APM-Server are randomly getting rejected. It appears that every so often the event getting reported is being met with a RemoteDisconnected exception.

To Reproduce

1. Setup a django project as per directions
2. Hit the homepage of the project
3. Wait for several events to hit the APM server
4. One will eventually cause the exception

Expected behavior: ... No Exception

Environment (please complete the following information)

• OS: Mojave but running under python:3 docker container
• Python version: 3.7.2
• Framework and version: Django version 2.1.5
• APM Server version: apm-server version 6.6.0 (amd64), libbeat 6.6.0
• Agent version: 4.1.0

Additional context

downgrading to 4.0.3 remedies the issue.

Directly related: https://discuss.elastic.co/t/unable-to-reach-apm-server-connection-aborted-remotedisconnected-remote-end-closed-connection-without-response/166500

• Agent config options

  Click to expand

  ELASTIC_APM = {
      'SERVICE_NAME': 'web-backend',
      'SECRET_TOKEN': '8149e743-2524-2524-2524-94aadd3fcde7',
      'SERVER_URL': 'http://192.168.16.100:8200',
      'DEBUG': True,
  }
  
  

• requirements.txt:

  Click to expand

  django-allauth
  django-guardian
  django-qr-code
  django-icons
  django-phonenumber-field
  django-crispy-forms
  django-autocomplete-light
  django-money
  django-anymail
  django-debug-toolbar
  django-extensions
  django-countries
  djangorestframework
  psycopg2-binary
  stripe
  geoip2
  redis
  celery
  sentry-sdk
  googlemaps
  django
  pillow
  phonenumbers
  gunicorn
  whitenoise
  elastic-apm
  

web_1      | Failed to submit message: "Unable to reach APM Server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) (url: http://192.168.16.100:8200/intake/v2/events)"
web_1      | Traceback (most recent call last):
web_1      |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
web_1      |     chunked=chunked)
web_1      |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
web_1      |     six.raise_from(e, None)
web_1      |   File "<string>", line 2, in raise_from
web_1      |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
web_1      |     httplib_response = conn.getresponse()
web_1      |   File "/usr/local/lib/python3.7/site-packages/sentry_sdk/integrations/stdlib.py", line 48, in getresponse
web_1      |     rv = real_getresponse(self, *args, **kwargs)
web_1      |   File "/usr/local/lib/python3.7/http/client.py", line 1321, in getresponse
web_1      |     response.begin()
web_1      |   File "/usr/local/lib/python3.7/http/client.py", line 296, in begin
web_1      |     version, status, reason = self._read_status()
web_1      |   File "/usr/local/lib/python3.7/http/client.py", line 265, in _read_status
web_1      |     raise RemoteDisconnected("Remote end closed connection without"
web_1      | http.client.RemoteDisconnected: Remote end closed connection without response
web_1      | 
web_1      | During handling of the above exception, another exception occurred:
web_1      | 
web_1      | Traceback (most recent call last):
web_1      |   File "/usr/local/lib/python3.7/site-packages/elasticapm/transport/http.py", line 40, in send
web_1      |     "POST", url, body=data, headers=self._headers, timeout=self._timeout, preload_content=False
web_1      |   File "/usr/local/lib/python3.7/site-packages/urllib3/poolmanager.py", line 323, in urlopen
web_1      |     response = conn.urlopen(method, u.request_uri, **kw)
web_1      |   File "/usr/local/lib/python3.7/site-packages/elasticapm/instrumentation/packages/base.py", line 106, in call_if_sampling
web_1      |     return wrapped(*args, **kwargs)
web_1      |   File "/usr/local/lib/python3.7/site-packages/elasticapm/instrumentation/packages/base.py", line 106, in call_if_sampling
web_1      |     return wrapped(*args, **kwargs)
web_1      |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
web_1      |     _stacktrace=sys.exc_info()[2])
web_1      |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 367, in increment
web_1      |     raise six.reraise(type(error), error, _stacktrace)
web_1      |   File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 685, in reraise
web_1      |     raise value.with_traceback(tb)
web_1      |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
web_1      |     chunked=chunked)
web_1      |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 384, in _make_request
web_1      |     six.raise_from(e, None)
web_1      |   File "<string>", line 2, in raise_from
web_1      |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 380, in _make_request
web_1      |     httplib_response = conn.getresponse()
web_1      |   File "/usr/local/lib/python3.7/site-packages/sentry_sdk/integrations/stdlib.py", line 48, in getresponse
web_1      |     rv = real_getresponse(self, *args, **kwargs)
web_1      |   File "/usr/local/lib/python3.7/http/client.py", line 1321, in getresponse
web_1      |     response.begin()
web_1      |   File "/usr/local/lib/python3.7/http/client.py", line 296, in begin
web_1      |     version, status, reason = self._read_status()
web_1      |   File "/usr/local/lib/python3.7/http/client.py", line 265, in _read_status
web_1      |     raise RemoteDisconnected("Remote end closed connection without"
web_1      | urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
web_1      | 
web_1      | During handling of the above exception, another exception occurred:
web_1      | 
web_1      | Traceback (most recent call last):
web_1      |   File "/usr/local/lib/python3.7/site-packages/elasticapm/transport/base.py", line 184, in send_sync
web_1      |     self.sync_transport.send(self, data)
web_1      |   File "/usr/local/lib/python3.7/site-packages/elasticapm/transport/http.py", line 53, in send
web_1      |     raise TransportException(message, data, print_trace=print_trace)
web_1      | elasticapm.transport.base.TransportException: Unable to reach APM Server: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) (url: http://192.168.16.100:8200/intake/v2/events)

What does this pull request do?

Replacement for https://github.com/elastic/apm-agent-python/pull/522

Related issues

closes elastic/apm-agent-python#521

Todo

• [x] Unit Tests
• [x] Minor Refactoring
• [x] Documentation

What does this pull request do?

Adds support for the asyncpg database interface library for Python and PostgreSQL by implementing corresponding instrumentation in the APM agent.

Related issues

https://github.com/elastic/apm-agent-python/issues/755

Closes issues

https://github.com/elastic/apm-agent-python/issues/755

I'm not sure how to reproduce but I've had this issue in production for several months. The crash happens randomly during the day, 20-50 times (we run 400k+ tasks per day)

The crash is:

TransportException: HTTP 400: {"accepted":0,"errors":[{"message":"gzip: invalid header"}]}

  File "elasticapm/transport/base.py", line 208, in _flush
    self.send(data)
  File "elasticapm/transport/http.py", line 102, in send
    raise TransportException(message, data, print_trace=print_trace)

Environment (please complete the following information)

• OS: Ubuntu 18.04
• Python version: 3.6
• Framework and version: Django 2.2 Celery 4.4.2
• APM Server version: 7.6.0
• Agent version: 5.3.2

What does this pull request do?

Makes SANITIZE_FIELD_NAMES a configurable option such that users can specify custom fields they want masked.

From the testing code, it looks like the base sanitize code already handles some wildcard-like features (i.e, passing some- as a field to be sanitized would mask fields that match *some-*).

Example usage:

ELASTIC_APM = {
    'APP_NAME': '<APP-NAME>',
    'SECRET_TOKEN': '<SECRET-TOKEN>',
    'PROCESSORS': (
        'path.to.my_processor',
        'elasticapm.processors.sanitize_stacktrace_locals',
        'elasticapm.processors.sanitize_http_request_cookies',
        'elasticapm.processors.sanitize_http_headers',
        'elasticapm.processors.sanitize_http_wsgi_env',
        'elasticapm.processors.sanitize_http_request_querystring',
        'elasticapm.processors.sanitize_http_request_body',
    ),
   'SANITIZE_FIELD_NAMES': (
        'some-sensitive-field-1',
        'should-be-hidden'
    ),
}

I wanted to get a review before updating the docs. I had some time and was peeking around the code and decided to take a crack at this and see if it would work out. Please do let me know if this approach is overly simplistic or if I'm missing some other context.

Related issues

closes #841

What does this pull request do?

Is a cleanup of @harshanarayana PR (https://github.com/elastic/apm-agent-python/pull/1056) to add compat with Sanic v21.9

Related issues

closes #521

Describe the bug: ...

CONTRIBUTING.md contains the following:

To run the full test suite, you need to install several databases (Elasticsearch, PostgreSQL, MySQL, Cassandra, Redis). This can be quite a hassle, so we recommend to use our dockerized test setup. See Running tests for detailed instructions.

README.md also points to Running tests

Running tests contains the following:

To run tests locally you can make use of the docker images also used when running the whole test suite with Jenkins. Running the full test suite first does some linting and then runs the actual tests with different versions of Python and different web frameworks. For a full overview of the test matrix and supported versions have a look at Jenkins Configuration.

That document does not however detail how to build/start the docker images. It includes instructions on running ./tests/scripts/docker/run_tests.sh, but as near as I can tell this is all run entirely on a host machine, not using a docker container. It also requires Python and several pip installs locally.

I would expect something in one of the docs of the form:

docker compose build...
docker build -t ...  && docker run ...
etc

Otherwise locally a clearer picture of exactly what resources I need to start up and how to configure them instead of simply "This can be quite a hassle".

Also, CONTRIBUTING.md contains the following:

However, for running local unit tests, you can install the relevant requirements files and then run py.test from the project root.

The root of the project does not contain a py.test file, nor does the test directory or the test/scripts directory.

Basically I'm trying to put together a pull request, and I'm really not sure how to build/test this repo.

What does this pull request do?

OpenTracing API allows to explicitly set start and end timestamps of spans but the implementation in this project does not allow that. That patch fixes it.

Closes #1126

What does this pull request do?

Adds additional tests for windows on different versions of python

Why is it important?

Resolves potential flakiness from Appveyor tests by deprecating them

Related issues

closes

Describe the bug: We have fastapi framework, and we add apm_client for starlette to the application, we want to generate trace.id to correlate logs when an exception happens. I have structlog package for the logging and import elasticapm.handlers.structlog.structlog_processor. We set exception handlers in fastapi application for our excpected handlers and also one for Exception class to capture the ones that we do not expect. I can get trace.id in child exceptions but not in the exception handler that is comparing to root.

My goal was to have log Exception types and have log correlation in them.

I am not sure if it is bug or feature request🤷‍♂️ To Reproduce

1. create apm agent and fastapi app, add the apm agent to application

from fastapi import FastAPI
from elasticapm.contrib.starlette import make_apm_client
from elasticapm.contrib.starlette import ElasticAPM
apm = make_apm_client(config=config)
app = FastAPI()
app.add_middleware(ElasticAPM, client=apm)

2. configure structlog

import structlog
from elasticapm.handlers.structlog import structlog_processor
def processor(logger, level, log_dict):
    return json.dumps(log_dict)

structlog.configure(processors=[structlog_processor, processor])

3. create an child exceptions and root and child exception

class AuthException(Exception):
    def __ini__(self):
        super().__init__()


@app.exception_handler(Exception)
async def fallback_handler(request , exc: Exception):
    getLogger().info("Uncaught exception during request handling happend")
    return JSONResponse({"detail": "ROOT EXCEPTION"}, status_code=500)

@app.exception_handler(AuthException)
async def validation_exception_handler(request, exc: AuthException):
    getLogger().info(f"Request Validation exception happend{sys.exc_info()}")
    return JSONResponse({"detail": "CHILD EXCEPTION"}, status_code=404)

4. create an endpoint to send request to see if logging formatter/processor works or not

@app.post("/")
async def post(body: Any = Body(...)):
    if body == 1: raise AuthException("child exception")
    if body == 2: raise ZeroDivisionError("root exception")
    return JSONResponse({"Response": "test SUCCESS"}, status_code=200)

Environment (please complete the following information)

• OS: Win and Linux
• Python version: 3.7
• Framework and version [e.g. Django 2.1]: fastapi 0.63
• APM Server version: _
• Agent version: 6.*

What does this pull request do?

b4edea3ca7885f5cbbad202a8f12325d131225ce has errors, if body size has more than 10000 bytes, it writes 10000 bytes message to elastic and call await self.app(scope, receive, wrapped_send) with same receive and invalid body,

Comment:

I separate receivers and now, if request has body more than 10000 bytes -> it will send to elastic only 10000 bytes and continue handle as normal request without cutting

Is your feature request related to a problem? Please describe.

Duo's client isn't instrumented because they use http.client.HTTPConnection.

Describe the solution you'd like

Add instrumentation for http.client.HTTPConnection.

Describe alternatives you've considered

Manually creating spans around Duo client calls, which is cumbersome and easy to forget.

Additional context

The urllib3 instrumentation almost works for http.client.HTTPConnection. It successfully instrumented Duo's client when I made these changes:

diff --git a/elasticapm/instrumentation/packages/urllib3.py b/elasticapm/instrumentation/packages/urllib3.py
index cce2c17c..65eba2b4 100644
--- a/elasticapm/instrumentation/packages/urllib3.py
+++ b/elasticapm/instrumentation/packages/urllib3.py
@@ -85,6 +85,7 @@ class Urllib3Instrumentation(AbstractInstrumentedModule):
         # packages that vendor or vendored urllib3 in the past
         ("requests.packages.urllib3.connectionpool", "HTTPConnectionPool.urlopen"),
         ("botocore.vendored.requests.packages.urllib3.connectionpool", "HTTPConnectionPool.urlopen"),
+        ("http.client", "HTTPConnection.request"),
     ]
 
     def call(self, module, method, wrapped, instance, args, kwargs):
@@ -95,8 +96,8 @@ class Urllib3Instrumentation(AbstractInstrumentedModule):
 
         host = instance.host
 
-        if instance.port != default_ports.get(instance.scheme):
-            host += ":" + str(instance.port)
+        # if instance.port != default_ports.get(instance.scheme):
+        #     host += ":" + str(instance.port)
 
         if "url" in kwargs:
             url = kwargs["url"]
@@ -105,8 +106,8 @@ class Urllib3Instrumentation(AbstractInstrumentedModule):
 
         signature = method.upper() + " " + host
 
-        if url.startswith("/"):
-            url = "%s://%s%s" % (instance.scheme, host, url)
+        # if url.startswith("/"):
+        #     url = "%s://%s%s" % (instance.scheme, host, url)
 
         transaction = execution_context.get_transaction()

Note how the instance.scheme stuff is commented out. There isn't a scheme attribute on http.client.HTTPConnection.

Describe the bug: ... In python 3.11, aioredis is broken, may we should use redis-py which include aioredis code instead of aioredis To Reproduce Just import this package and config APM middleware:

app.add_middleware(ElasticAPM, client=apm_client)

Error:

...
  File "/root/.cache/pypoetry/virtualenvs/app-iZOiGJ6Y-py3.11/lib/python3.11/site-packages/elasticapm/contrib/starlette/__init__.py", line 121, in __init__
    elasticapm.instrumentation.control.instrument()
  File "/root/.cache/pypoetry/virtualenvs/app-iZOiGJ6Y-py3.11/lib/python3.11/site-packages/elasticapm/instrumentation/control.py", line 45, in instrument
    obj.instrument()
  File "/root/.cache/pypoetry/virtualenvs/app-iZOiGJ6Y-py3.11/lib/python3.11/site-packages/elasticapm/instrumentation/packages/base.py", line 144, in instrument
    parent, attribute, original = wrapt.resolve_path(module, method)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/pypoetry/virtualenvs/app-iZOiGJ6Y-py3.11/lib/python3.11/site-packages/wrapt/wrappers.py", line 773, in resolve_path
    __import__(module)
  File "/root/.cache/pypoetry/virtualenvs/app-iZOiGJ6Y-py3.11/lib/python3.11/site-packages/aioredis/__init__.py", line 1, in <module>
    from aioredis.client import Redis, StrictRedis
  File "/root/.cache/pypoetry/virtualenvs/app-iZOiGJ6Y-py3.11/lib/python3.11/site-packages/aioredis/client.py", line 32, in <module>
    from aioredis.connection import (
  File "/root/.cache/pypoetry/virtualenvs/app-iZOiGJ6Y-py3.11/lib/python3.11/site-packages/aioredis/connection.py", line 33, in <module>
    from .exceptions import (
  File "/root/.cache/pypoetry/virtualenvs/app-iZOiGJ6Y-py3.11/lib/python3.11/site-packages/aioredis/exceptions.py", line 14, in <module>
    class TimeoutError(asyncio.TimeoutError, builtins.TimeoutError, RedisError):
TypeError: duplicate base class TimeoutError

Environment (please complete the following information)

• OS: [e.g. Linux] linux
• Python version: 3.11
• Framework and version [e.g. Django 2.1]:
• APM Server version: None
• Agent version: 6.13.2

Additional context

Add any other context about the problem here.

https://github.com/aio-libs/aioredis-py/issues/1443

• Agent config options

  Click to expand

  replace this line with your agent config options
  remember to mask any sensitive fields like tokens
  

• requirements.txt:

  Click to expand

  replace this line with your `requirements.txt`
  

What does this pull request do?

Adds the wrapper script to automate instrumentation without code changes

I expect I'll add at least Flask support and some docs labeled "experimental" before I actually merge this, but I wanted to open a PR for feedback.

Related issues

Closes #1019

I believe ElasticAPM will hang the lambda function when no more events exist in the SQS queue.

To Reproduce

• Setup a lambda function to process events from an SQS queue
• Send events to an SQS queue
• Set ELASTIC_APM_LOG_LEVEL: debug

Environment (please complete the following information)

• OS: debian:bullseye-slim (python:3.9-slim container image) x86
• Python version: 3.9
• Framework and version [e.g. Django 2.1]:
• APM Server version: 8.4.2
• Agent version: 6.13.2

Additional context Cloudwatch logs (can dive deeper if needed) after execution is completed