We recently started noticing that a lot of our queries are running slowly over bolt from the python driver when they open up new sessions. At first we thought that it was because we were setting up the sessions wrong and so they weren't correctly pulling from the connection pool and it would keep having to start up new connections with each request. However during our debugging and testing we ran into this strange case:

def tf(f):
     start = time()
     f()
     end = time()
     return (end - start)*1000

This function is just here to time how long each query takes.

with driver.new_session() as session:
    for i in range(0, 500):
        val = tf(lambda: session.run("match (n) return count(n)"))
        if val > 2:
            print i, ":", val

Then this creates one session and uses it to run the same query over and over. Usually the queries take under a ms, but this prints out the ones that didn't and this is what we got:

0 : 3.43799591064
2 : 2.49195098877
6 : 2.96497344971
14 : 4.04977798462
30 : 6.24489784241
62 : 12.0089054108
132 : 23.7801074982
266 : 17.529964447
432 : 41.0959720612

(This was running on my local graph, we see the same thing in our prod environment but with 10x larger slowdowns) There's a pattern where the one's that take ~10x time is roughly following f(0) = 0, f(n) = f(n-1)+ 2^n. I've run this a few times and its always the same ones that are slow.

Any ideas what could be going on here and why these specific queries are so slow?

raise ServiceUnavailable("Failed to establish connection to {!r} (reason {})".format(resolved_address, error))\nneo4j.exceptions.ServiceUnavailable: Failed to establish connection to IPv4Address(('0.0.0.0', 7687)) (reason [Errno 111] Connection refused)","statusCode":430

when is use neo4j4.2.3 on tencent cloud function use bolt drive find Failed to establish connection to IPv4Address(('0.0.0.0', 7687) when use 127.0.0.1 Failed to establish connection to IPv4Address(('127.0.0.1', 7687)

why? how to solve it? thank you!

This is occurring in a databricks Python notebook when using the neo4j python driver: from neo4j import GraphDatabase I receive the following error:

---------------------------------------------------------------------------
ServiceUnavailable                        Traceback (most recent call last)
<command-3411714849431767> in <module>
      2
      3 # create dataset node with metadata and write to neo4j
----> 4 neo4j_write(generate_graph_dataset_query(params, spark_df))
 
<command-3411714849431763> in neo4j_write(query)
     31     """Write data to the graph database with a custom query"""
     32
---> 33     session = connect_to_graph()
     34     session.run(query)
     35     session.close()
 
<command-3411714849431763> in connect_to_graph(neo_4j_url, ws_username, ws_password, database)
     21 #         print(e)
     22 #         return None
---> 23     connection = GraphDatabase.driver(neo_4j_url,
     24         auth=(ws_username, ws_password))
     25     session = connection.session(database=database)
 
/databricks/python/lib/python3.8/site-packages/neo4j/__init__.py in driver(cls, uri, auth, **config)
    184         elif driver_type == DRIVER_NEO4j:
    185             routing_context = parse_routing_context(parsed.query)
--> 186             return cls.neo4j_driver(parsed.netloc, auth=auth, routing_context=routing_context, **config)
    187
    188     @classmethod
 
/databricks/python/lib/python3.8/site-packages/neo4j/__init__.py in neo4j_driver(cls, auth, routing_context, *targets, **config)
    207
    208         try:
--> 209             return Neo4jDriver.open(*targets, auth=auth, routing_context=routing_context, **config)
    210         except (BoltHandshakeError, BoltSecurityError) as error:
    211             from neo4j.exceptions import ServiceUnavailable
 
/databricks/python/lib/python3.8/site-packages/neo4j/__init__.py in open(cls, auth, routing_context, *targets, **config)
    408         addresses = cls.parse_targets(*targets)
    409         pool_config, default_workspace_config = Config.consume_chain(config, PoolConfig, WorkspaceConfig)
--> 410         pool = Neo4jPool.open(*addresses, auth=auth, routing_context=routing_context, pool_config=pool_config, workspace_config=default_workspace_config)
    411         return cls(pool, default_workspace_config)
    412
 
/databricks/python/lib/python3.8/site-packages/neo4j/io/__init__.py in open(cls, auth, pool_config, workspace_config, routing_context, *addresses)
    579
    580         try:
--> 581             pool.update_routing_table(database=workspace_config.database)
    582         except Exception:
    583             pool.close()
 
/databricks/python/lib/python3.8/site-packages/neo4j/io/__init__.py in update_routing_table(self, database)
    802         # None of the routers have been successful, so just fail
    803         log.error("Unable to retrieve routing information")
--> 804         raise ServiceUnavailable("Unable to retrieve routing information")
    805
    806     def update_connection_pool(self, *, database):
 
ServiceUnavailable: Unable to retrieve routing information

Here is my code for connecting to neo4j:

def connect_to_graph(neo_4j_url=URL, ws_username=USER, ws_password=PWD, database=DB):
   
    """ Connect to the neo4j graph DB """

    connection = GraphDatabase.driver(neo_4j_url,
        auth=(ws_username, ws_password))
    session = connection.session(database=database)

    return session

This code successfully writes data to a different neo4j database when tested

The C library function on windows 10 does not support times below a certain value.

https://github.com/neo4j/neo4j-python-driver/blob/4.0/neo4j/time/init.py#L239

    @classmethod
    def local_offset(cls):
        """ The offset from UTC for local time read from this clock.
        """
        return ClockTime(-int(mktime(gmtime(0))))

python time module

Although this module is always available, not all functions are available on all platforms. Most of the functions defined in this module call platform C library functions with the same name. It may sometimes be helpful to consult the platform documentation, because the semantics of these functions varies among platforms.

The epoch is the point where the time starts, and is platform dependent. For Unix, the epoch is January 1, 1970, 00:00:00 (UTC). To find out what the epoch is on a given platform, look at time.gmtime(0).

https://docs.python.org/3/library/time.html

>>> time.gmtime(0)
time.struct_time(tm_year=1970, tm_mon=1, tm_mday=1, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=3, tm_yday=1, tm_isdst=0)

>>> list((ix for ix in time.gmtime(0)))
[1970, 1, 1, 0, 0, 0, 3, 1, 0]

>>> time.mktime(time.gmtime(0))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: mktime argument out of range

I encountered a new issue with the Async driver version 5.0.0a2.

The error states:

ERROR in app: Exception on request POST /recommend_contact
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/quart/app.py", line 1489, in handle_request
    return await self.full_dispatch_request(request_context)
  File "/usr/local/lib/python3.7/site-packages/quart/app.py", line 1514, in full_dispatch_request
    result = await self.handle_user_exception(error)
  File "/usr/local/lib/python3.7/site-packages/quart/app.py", line 964, in handle_user_exception
    raise error
  File "/usr/local/lib/python3.7/site-packages/quart/app.py", line 1512, in full_dispatch_request
    result = await self.dispatch_request(request_context)
  File "/usr/local/lib/python3.7/site-packages/quart/app.py", line 1557, in dispatch_request
    return await self.ensure_async(handler)(**request_.view_args)
  File "/usr/local/lib/python3.7/site-packages/quart/views.py", line 57, in view
    return await current_app.ensure_async(self.dispatch_request)(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/quart_openapi/resource.py", line 71, in dispatch_request
    return await handler(*args, **kwargs)
  File "/api/app.py", line 119, in post
    recommendations = await asyncio.gather(*tasks)
  File "/api/app.py", line 108, in recommendation_task
    el['offsite'])
  File "/usr/local/lib/python3.7/site-packages/recommendation_engine-0.0.1-py3.7.egg/recommendation_engine/recommendation/recommender.py", line 32, in get_popular_contact
    ret = await self.graph_handler.run_query_async(graph_query, contact_id=contact_id, p=p, offsite=offsite)
  File "/usr/local/lib/python3.7/site-packages/recommendation_engine-0.0.1-py3.7.egg/recommendation_engine/graph_handler.py", line 636, in run_query_async
    return await session.read_transaction(self.run_transaction, query, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/neo4j/_meta.py", line 73, in inner
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/neo4j/_async/work/session.py", line 656, in read_transaction
    READ_ACCESS, transaction_function, *args, **kwargs
  File "/usr/local/lib/python3.7/site-packages/neo4j/_async/work/session.py", line 478, in _run_transaction
    timeout=timeout
  File "/usr/local/lib/python3.7/site-packages/neo4j/_async/work/session.py", line 392, in _open_transaction
    await self._connect(access_mode=access_mode)
  File "/usr/local/lib/python3.7/site-packages/neo4j/_async/work/session.py", line 122, in _connect
    await super()._connect(access_mode, **access_kwargs)
  File "/usr/local/lib/python3.7/site-packages/neo4j/_async/work/workspace.py", line 194, in _connect
    self._connection = await self._pool.acquire(**acquire_kwargs_)
  File "/usr/local/lib/python3.7/site-packages/neo4j/_async/io/_pool.py", line 403, in acquire
    self.address, deadline, liveness_check_timeout
  File "/usr/local/lib/python3.7/site-packages/neo4j/_async/io/_pool.py", line 216, in _acquire
    "{!r}s (timeout)".format(deadline.original_timeout)
neo4j.exceptions.ClientError: {code: None} {message: None}

I suspected it is the issue with connections being kept alive or being alive for a prolonged period of time, as when I checked "/usr/local/lib/python3.7/site-packages/neo4j/_async/io/_pool.py" I noticed the error is thrown when the pool is full and there are no avaliable connections. Also the code and message are shown as None, so I am not sure if that is a bug also.

We are running Neo4j Version 4.4.10 Community edition as a single instance.

Neo4j Version: 4.1.0 Enterprise
Neo4j Mode: Cluster
Driver version: Python driver 4.0.2
Operating System: Docker

The network has three neo4j database. Cluster1 is the leader, Cluster0 and Cluster2 are the follower. And my code is like:

createDriver function:

class Graph(object):

    def __init__(self, config):
        uri = config["neo4jUri"]
        addresses = config["neo4jAddress"]
        user = config["neo4jUser"]
        password = config["neo4jPass"]
        self._driver = self.createDriver(uri=uri, user=user, password=password, addresses=addresses)
    
    def createDriver(self, uri, user, password, addresses):
        def resolver(address):
            for address in addresses:
                yield address, 7687
        return GraphDatabase.driver(uri, auth=(user, password), resolver=resolver)

    def run_query(self, func, arg):
        with self._driver.session() as session:
            response = None
            try:
                response = session.write_transaction(func, arg)
            except ConstraintError:
                print("ConstraintError")
                return response
            except Exception as e:
                print("Exception: {0}".format(e))
            finally:
                return response

And when I only use Cluster1, it works just fine.

But when I use all three addresses, it won't work, and tell:

Exception: Should only attempt to acquire lease when leader.

Does driver 1.7.6 support neo4j4.0?

  File "/home/dba/db_neo4j/adapters/neo4j.py", line 24, in initialize_neo4j_pool
    password=Neo4jConfig.password))
  File "/home/dba/db_neo4j/python3/lib/python3.7/neo4j/__init__.py", line 120, in driver
    return Driver(uri, **config)
  File "/home/dba/db_neo4j/python3/lib/python3.7/neo4j/__init__.py", line 161, in __new__
    return subclass(uri, **config)
  File "/home/dba/db_neo4j/python3/lib/python3.7/neo4j/__init__.py", line 235, in __new__
    pool.release(pool.acquire())
  File "/home/dba/db_neo4j/python3/lib/python3.7/neobolt/direct.py", line 715, in acquire
    return self.acquire_direct(self.address)
  File "/home/dba/db_neo4j/python3/lib/python3.7/neobolt/direct.py", line 608, in acquire_direct
    connection = self.connector(address, error_handler=self.connection_error_handler)
  File "/home/dba/db_neo4j/python3/lib/python3.7/neo4j/__init__.py", line 232, in connector
    return connect(address, **dict(config, **kwargs))
  File "/home/dba/db_neo4j/python3/lib/python3.7/neobolt/direct.py", line 972, in connect
    raise last_error
  File "/home/dba/db_neo4j/python3/lib/python3.7/neobolt/direct.py", line 963, in connect
    s, der_encoded_server_certificate = _secure(s, host, security_plan.ssl_context, **config)
  File "/home/dba/db_neo4j/python3/lib/python3.7/neobolt/direct.py", line 854, in _secure
    s = ssl_context.wrap_socket(s, server_hostname=host if HAS_SNI and host else None)
  File "/home/dba/db_neo4j/python3/lib/python3.7/ssl.py", line 412, in wrap_socket
    session=session
  File "/home/dba/db_neo4j/python3/lib/python3.7/ssl.py", line 850, in _create
    self.do_handshake()
  File "/home/dba/db_neo4j/python3/lib/python3.7/ssl.py", line 1108, in do_handshake
    self._sslobj.do_handshake()
OSError: [Errno 0] Error

The neo4j driver will give the error neobolt.exceptions.ServiceUnavailable: Failed to read from defunct connection Address(host='<remote neo4j bolt>', port=7687) if certain amount of time spent between the creation of neo4j client and creation of the session. However, if I start the session the second time, the error is gone.

from neo4j import GraphDatabase
import time

neo4j_url = "bolt://<remote_neo4j_url>:7687"
user_name = "neo4j"
password = ""
neo4j_client = GraphDatabase.driver(neo4j_url, auth=(user_name, password))

time.sleep(300) # 5 minutes
query = '''MATCH (n) RETURN n LIMIT 1'''

# with error
try:
    with neo4j_client.session() as session:
        query_result = session.run(query)
except Exception as e:
    print(str(e))

# no error
with neo4j_client.session() as session:
    query_result = session.run(query)

Environment: python 3.6 neo4j==1.7.2 neobolt==1.7.4 neotime==1.7.4 mac 10.14.4

I got NotALeaderError in write_transaction() when running an application with neo4j-driver for about 1 day on average. This happens almost every day since it is runnning 24/7. The app runs a transaction every 5 minutes if necessary. After the first error is thrown no more transactions will succeed until application restart.

This might be related: https://github.com/neo4j-contrib/neomodel/issues/335

Neo4j Version: 3.4.9 Enterprise
Neo4j Mode: Causal cluster with 3 core Driver version: Python driver 1.7.1
Operating System: Docker base image python:2.7-slim Packaging Tool: Pipenv

Steps to reproduce

1. Start Neo4j on self-hosted VM
2. Start Application on Kubernetes
3. Let it run for a day

Expected behavior

The application keeps being able to run write_transaction() on Neo4j cluster

Actual behavior

After running for about 1 day the application stops being able to write. Traceback (modified to hide function/application names):

Traceback (most recent call last):\
 results = session.write_transaction(_unit_of_work, time_limit)\
 File \\"/usr/local/lib/python2.7/site-packages/neo4j/__init__.py\\", line 708, in write_transaction\
 return self._run_transaction(WRITE_ACCESS, unit_of_work, *args, **kwargs)\
 File \\"/usr/local/lib/python2.7/site-packages/neo4j/__init__.py\\", line 683, in _run_transaction\
 tx.close()\
 File \\"/usr/local/lib/python2.7/site-packages/neo4j/__init__.py\\", line 822, in close\
 self.sync()\
 File \\"/usr/local/lib/python2.7/site-packages/neo4j/__init__.py\\", line 787, in sync\
 self.session.sync()\
 File \\"/usr/local/lib/python2.7/site-packages/neo4j/__init__.py\\", line 538, in sync\
 detail_count, _ = self._connection.sync()\
 File \\"/usr/local/lib/python2.7/site-packages/neobolt/direct.py\\", line 506, in sync\
 detail_delta, summary_delta = self.fetch()\
 File \\"/usr/local/lib/python2.7/site-packages/neobolt/direct.py\\", line 413, in fetch\
 raise error\
NotALeaderError: No write operations are allowed directly on this database. Writes must pass through the leader. The role of this server is: FOLLOWER\

I tried to run the example given for python at https://neo4j.com/developer/ I get the following error

ConnectionRefusedError Traceback (most recent call last) /usr/local/lib/python3.7/site-packages/neo4j/bolt/connection.py in _connect(resolved_address, **config) 577 log_debug("~~ [CONNECT] %s", resolved_address) --> 578 s.connect(resolved_address) 579 s.settimeout(t)

ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

ServiceUnavailable Traceback (most recent call last) in () 1 from neo4j.v1 import GraphDatabase, basic_auth 2 ----> 3 session = GraphDatabase.driver("bolt://localhost", auth=basic_auth("neo4j", "")).session() 4 5 # Insert data

/usr/local/lib/python3.7/site-packages/neo4j/v1/api.py in driver(cls, uri, **config) 86 :class:.Driver subclass instance directly. 87 """ ---> 88 return Driver(uri, **config) 89 90

/usr/local/lib/python3.7/site-packages/neo4j/v1/api.py in new(cls, uri, **config) 125 for subclass in Driver.subclasses(): 126 if parsed.scheme == subclass.uri_scheme: --> 127 return subclass(uri, **config) 128 raise ValueError("URI scheme %r not supported" % parsed.scheme) 129

/usr/local/lib/python3.7/site-packages/neo4j/v1/direct.py in new(cls, uri, **config) 71 72 pool = DirectConnectionPool(connector, instance.address, **config) ---> 73 pool.release(pool.acquire()) 74 instance._pool = pool 75 instance._max_retry_time = config.get("max_retry_time", default_config["max_retry_time"])

/usr/local/lib/python3.7/site-packages/neo4j/v1/direct.py in acquire(self, access_mode) 42 43 def acquire(self, access_mode=None): ---> 44 return self.acquire_direct(self.address) 45 46

/usr/local/lib/python3.7/site-packages/neo4j/bolt/connection.py in acquire_direct(self, address) 451 if can_create_new_connection: 452 try: --> 453 connection = self.connector(address, self.connection_error_handler) 454 except ServiceUnavailable: 455 self.remove(address)

/usr/local/lib/python3.7/site-packages/neo4j/v1/direct.py in connector(address, error_handler) 68 69 def connector(address, error_handler): ---> 70 return connect(address, security_plan.ssl_context, error_handler, **config) 71 72 pool = DirectConnectionPool(connector, instance.address, **config)

/usr/local/lib/python3.7/site-packages/neo4j/bolt/connection.py in connect(address, ssl_context, error_handler, **config) 705 raise ServiceUnavailable("Failed to resolve addresses for %s" % address) 706 else: --> 707 raise last_error

/usr/local/lib/python3.7/site-packages/neo4j/bolt/connection.py in connect(address, ssl_context, error_handler, **config) 695 log_debug("~~ [RESOLVED] %s -> %s", address, resolved_address) 696 try: --> 697 s = _connect(resolved_address, **config) 698 s, der_encoded_server_certificate = _secure(s, address[0], ssl_context, **config) 699 connection = _handshake(s, resolved_address, der_encoded_server_certificate, error_handler, **config)

/usr/local/lib/python3.7/site-packages/neo4j/bolt/connection.py in _connect(resolved_address, **config) 585 _force_close(s) 586 if error.errno in (61, 99, 111, 10061): --> 587 raise ServiceUnavailable("Failed to establish connection to {!r} (reason {})".format(resolved_address, error.errno)) 588 else: 589 raise

ServiceUnavailable: Failed to establish connection to ('127.0.0.1', 7687) (reason 61)

when I run python start.py(python2.7, no errors with python3 ), got this error:

Exception AttributeError: AttributeError("'NoneType' object has no attribute 'close'",) in <bound method DirectDriver.__del__ of <neo4j.v1.direct.DirectDriver object at 0x108923ed0>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'close'" in <bound method Connection.__del__ of <neo4j.bolt.connection.Connection object at 0x109972b90>> ignored

my project structure:

neo4j_test
    start.py
    neo4j_test
        __init__.py
        db.py

start.py:

from neo4j_test import db

db.py:

from neo4j.v1 import GraphDatabase
neo4j_driver = GraphDatabase.driver('bolt://localhost:7687', auth=('neo4j', 'password'))

I found while calling close method from neo4j.bolt.connection.Connection, log_info is None.

my version info

• python 2.7.10
• neo4j 3.2.0
• neo4j-driver 1.3.1

A similar case (saved as test.py):

import logging
log = logging.getLogger("neo4j.bolt")
log_info = log.info
class Test(object):
    def __del__(self):
        self.close()
    def close(self):
        log_info("~~ [CLOSE]")
t = Test()

>>> python test.py
Exception TypeError: "'NoneType' object is not callable" in <bound method Test.__del__ of <__main__.Test object at 0x103cdb910>> ignored
>>> python3 test.py

Implement serialization support for

• python types: tuple
• pandas types:
  • all but period, interval, pyarrow
• numpy types:
  • all but void, complexfloating

pyarrow support was not implemented as it would either require more ifs in the recursive packing function, making it (driver's hot-path) slower for non-pyarrow use-cases. Or alternatively, transformers would have to be used making pyarrow type serialization rather slow.

To help us understand your issue, please specify important details, primarily:

• Neo4j version: AWS Neptune
• Neo4j Mode: N/A
• Driver version: All
• Operating system: Linux

We're using openCypher with AWS Neptune and want to use the neo4j driver with the bolt protocol. We are generating an AWS V4 signature and passing it in as a basic auth to the driver. The request is signed using temporary credentials that expires in 5 minutes.

Unfortunately, it seems that the authentication token is cached at connection creation time.

# Determine auth details
if not auth:
    self.auth_dict = {}
elif isinstance(auth, tuple) and 2 <= len(auth) <= 3:
    from neo4j import Auth
    self.auth_dict = vars(Auth("basic", *auth))
else:
    try:
        self.auth_dict = vars(auth)
    except (KeyError, TypeError):
        raise AuthError("Cannot determine auth details from %r" % auth)

Thus, if the max_connection_lifetime is set to less than 5 minutes, requests start failing after 5 minutes because the signature is no longer valid.

self = <neo4j.io._common.InitResponse object at 0x7f41eb9dc750>
metadata = {'code': '"BoltProtocol.unexpectedException"', 'message': '"Unexpected server exception \'Signature expired: 20221018T003343Z is now earlier than 20221018T003344Z (20221018T003844Z - 5 min.)\'"'}

    def on_failure(self, metadata):
        code = metadata.get("code")
        if code == "Neo.ClientError.Security.Unauthorized":
            raise Neo4jError.hydrate(**metadata)
        else:
            raise ServiceUnavailable(
>               metadata.get("message", "Connection initialisation failed")
            )
E           neo4j.exceptions.ServiceUnavailable: "Unexpected server exception 'Signature expired: 20221018T003343Z is now earlier than 20221018T003344Z (20221018T003844Z - 5 min.)'"

The Javascript driver also has the same issue.

But the Java driver provides a way to avoid this problem by exposing a toMap method, that is called to obtain the token. While the cached token is returned by default, this method can be overridden in a subclass as shown here.

We would like the Python driver to also provide an option not to cache the auth token and regenerate it on demand.

Exceptions raised during some cleanup steps in session.close could have left the session in an unclean state. In particular, the session wouldn't be marked as closed. Moreover, errors encountered when consuming outstanding auto-commit results during session closure were silently swallowed.

Back port of https://github.com/neo4j/neo4j-python-driver/pull/800

Exceptions raised during some cleanup steps in session.close could have left the session in an unclean state. In particular, the session wouldn't be marked as closed. Moreover, errors encountered when consuming outstanding auto-commit results during session closure were silently swallowed.