Hi guys, We are using mongo-connector to feed data from mongodb to elasticsearch. Everything went well until today we have a big insert / update to mongodb and mongo-connector started falling behind and taking hours to catch-up. Here is the mongo-connector config:

"mainAddress": "10.a.b.c:27017",
    "oplogFile": "/var/log/mongo-connector/oplog.timestamp",
    "noDump": false,
    "batchSize": -1,
    "verbosity": 2,
    "continueOnError": true,

 "docManagers": [
        {
            "docManager": "elastic2_doc_manager",
            "targetURL": "10.x.y.z:9200",
            "bulkSize": 1000,
            "__uniqueKey": "_id",
            "args": {
                "clientOptions": {"timeout": 60}
                 }
        }

Elasticsearch is setup as cluster of 3 servers. I'm looking for anyway to make mongo-connector tailing and update data faster.

Thanks everyone.

Hi,

Im trying to use the mongo-connector to push the contents on my Mongodb (v2.6.1) into SOLR (4.8). I've configured my schema and performed a few test runs using small collections and it works perfectly. However, whenever I try to use it with large (80M items) it crashes out after a few seconds.

This is the command I'm using:

mongo-connector -n scans-io.certificates -m localhost:27017 -t http://localhost:8983/solr/certificates -d mongo_connector/doc_managers/solr_doc_manager.py --no-dump -v

And here is what happens:

2014-05-14 20:12:48,451 - DEBUG - "POST /solr/certificates/update/?commit=false HTTP/1.1" 200 None
2014-05-14 20:12:48,452 - INFO - Finished 'http://localhost:8983/solr/certificates/update/?commit=false' (post) with body 'u'<add><do' in 0.003 seconds.
2014-05-14 20:12:48,452 - DEBUG - OplogThread: Doc is processed.
2014-05-14 20:12:48,452 - DEBUG - OplogThread: updating checkpoint afterprocessing new oplog entries
2014-05-14 20:12:48,453 - DEBUG - OplogThread: oplog checkpoint updated to Timestamp(1400094752, 438)
2014-05-14 20:12:48,453 - DEBUG - OplogThread: updating checkpoint after an Exception, cursor closing, or join() on thisthread.
2014-05-14 20:12:48,453 - DEBUG - OplogThread: oplog checkpoint updated to Timestamp(1400094752, 438)
2014-05-14 20:12:48,453 - DEBUG - OplogThread: Sleeping.  This batch I removed 0  documents and I upserted 2843 documents.
2014-05-14 20:12:50,454 - DEBUG - OplogThread: Getting cursor
2014-05-14 20:12:50,454 - DEBUG - OplogThread: Initializing the oplog cursor.
2014-05-14 20:12:50,454 - DEBUG - OplogThread: reading last checkpoint as Timestamp(1400094752, 438) 
2014-05-14 20:12:50,454 - DEBUG - OplogThread: Getting the oplog cursor and moving it to the proper place in the oplog.
2014-05-14 20:12:50,454 - DEBUG - OplogThread: Getting the oplog cursor in the while true loop for get_oplog_cursor
2014-05-14 20:12:50,455 - DEBUG - OplogThread: Cursor created, getting a count.
2014-05-14 20:12:50,898 - DEBUG - OplogThread: Count is 313375
2014-05-14 20:12:51,135 - DEBUG - OplogThread: Got the cursor, go go go!
2014-05-14 20:12:51,135 - ERROR - OplogThread: Last entry no longer in oplog cannot recover! Collection(Database(MongoClient('localhost', 27017), u'local'), u'oplog.rs')
2014-05-14 20:12:51,858 - ERROR - MongoConnector: OplogThread <OplogThread(Thread-2, stopped 4387258368)> unexpectedly stopped! Shutting down
2014-05-14 20:12:51,858 - INFO - MongoConnector: Stopping all OplogThreads
2014-05-14 20:12:51,858 - DEBUG - OplogThread: exiting due to join call.

I'm having an issue where mongo-connector is sending entries from the oplog that include $addToSet and $pull attributes. These documents are destroying my mapping (I have custom date formats declared), and eventually lead to errors and mongo-connector crashing.

Before I upgraded my cluster to v2.2.0 and mongo-connector to 2.3, I did not have these issues and mongo-connector was working like a champ. I'm not sure if these kinds of oplog entries were handled before, but it'd be great to get a rundown on that and if this issue is caused by the mongo-connector 2.3/elastic2_doc_manager upgrade or an issue with mongo.

I'm currently running:

• mongo-connector 2.3
• elastic2-doc-manager 0.1.0
• mongodb 2.2.7
• elasticsearch 2.2.0

Example of a doc containing $addToSet

{
    "ip_data": {
        "3": {
            "web_published": false
        }
    },
    "$addToSet": {
        "ip_data": {
            "web_published": true,
            "seen_first": "2016-02-18T15:43:56.114000",
            "active_last_seen": "2016-02-18T15:43:56.114000",
            "seen_last": "2016-02-18T15:43:56.114000",
            "version": "IPv4",
            "_id": "[redacted]"
        }
    }
}

Example of a doc containing $pull

{
    "$pull": {
        "ip_data": {
            "_id": "[redacted]"
        }
    }
}

I'm trying to connect from a remote MongoDB instance to a local Elasticsearch instance. The MongoDB instance is a replica set behind a proxy, and I'm connecting with SSL. I've tried tweaking things with varying degrees of success, the closest I've gotten is when I hit this error:

ServerSelectionTimeoutError: <ip_address_1>:27017: timed out,<ip_address_2>:27017: timed out,<ip_address_3>:27017: timed out

The 3 IP addresses it lists are correct for the replica set members, which indicates to me that it's actually connecting/authenticating properly, but some issue occurs afterward.

The command I'm using looks like this: mongo-connector -c mongo-connector-config.json -m mongodb://<ip_address>:<port>/?ssl=true -t 192.168.99.100:9200 -d elastic_doc_manager --ssl-ca-certs my-cert.pem --ssl-certificate-policy optional

The only things I'm specifying in my config are noDump: true and a few namespaces.

I have 2 replicaset in 2 different environnment (dev&prod) with no issue. They run mongo-connector 2.0.3 with pyMongo 2.8. Prod is running with python 2.6, dev with 2.7.

I tried recently to set up a 3thd env with dev data. I ran into fatal issue. This occured with pyMongo 2.0.3, 2.10 and pyMongo 2.8 and 2.8.1.

When starting mongo-connector service, after a constant number of 6000 doc inserted into ElasticSearch. The 6001nth is raising:

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/util.py", line 85, in wrapped
    func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/oplog_manager.py", line 185, in run
    for n, entry in enumerate(cursor):
  File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 1090, in next
    if len(self.__data) or self._refresh():
  File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 1039, in _refresh
    limit, self.__id))
  File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 958, in __send_message
    self.__compile_re)
  File "/usr/local/lib/python2.7/dist-packages/pymongo/helpers.py", line 121, in _unpack_response
    compile_re)
  File "/usr/local/lib/python2.7/dist-packages/bson/__init__.py", line 537, in decode_all
    tz_aware, uuid_subtype, compile_re))
  File "/usr/local/lib/python2.7/dist-packages/bson/__init__.py", line 331, in _elements_to_dict
    data, position, as_class, tz_aware, uuid_subtype, compile_re)
  File "/usr/local/lib/python2.7/dist-packages/bson/__init__.py", line 320, in _element_to_dict
    data, position, as_class, tz_aware, uuid_subtype, compile_re)
  File "/usr/local/lib/python2.7/dist-packages/bson/__init__.py", line 159, in _get_object
    encoded, as_class, tz_aware, uuid_subtype, compile_re)
  File "/usr/local/lib/python2.7/dist-packages/bson/__init__.py", line 331, in _elements_to_dict
    data, position, as_class, tz_aware, uuid_subtype, compile_re)
  File "/usr/local/lib/python2.7/dist-packages/bson/__init__.py", line 320, in _element_to_dict
    data, position, as_class, tz_aware, uuid_subtype, compile_re)
  File "/usr/local/lib/python2.7/dist-packages/bson/__init__.py", line 232, in _get_date
    dt = EPOCH_AWARE + datetime.timedelta(seconds=seconds)
InvalidBSON: date value out of range

If I set timeZoneAware to false, this is exactly the same, expect EPOCH_AWAREis replaced by EPOCH_NAIVE.

So you think the 6001nth record I imported is bad ? Nope. The first time I launched mongo-connector, I have already imported 180,000+ records, but the second time I juste droped the collection and re-imported all of them so that mongo-connector started to insert doc in ES at the time I start import. Same issue. Setting noDump=true had no effect. The proof the 6001nth doc is not bad ? By spliting import file into 2 parts, (1: 6000 records, 2: remaining), I got 6000 record then 100 imported before first error. 6000 and 100 where constants. By splitingn one more time (6000+6000+remaining), I was hable to import 6000+283 docs. Strange. So pyMongo was not detectin a bad BSON, there is a kind of race condition underneath.

continueOnError and batchSize have no effect, each time I restart mongo-connector without removing oplog-timestamp, even with a dropped collection, it will try to send to ES the same docs again and again. I expected batchSize=1 to force mongo-connector to have an up to date timestamp, but this is not working. the last few records before the fatal error are always "repeated".

The bad part is that I'm not able to launch mongo-connector only to refresh timestamp and exiting, is is crashing very fast and do not update anything. I'm still looking for a workaround : my goal start from an empty collection, start mongo-connector that will remain quiet, then starting to insert doc at a slow rate to see if it matters.

Please upgrade pyMongo dep if it solve this solution (pyMongo 3 is supposed to have a better date support), or forward the issue to pyMongo if the latest has the same issue. Anyway, you should fix mongo-connector so that oplog-timestamp is updated to avoid doc manager insertion replay.

Hi! Think it's needed to ajust the bulk insertion size. When the elasticsearch is left behind by too much, the bulk insertion fails. Here is the trackback:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/oplog_manager.py", line 490, in do_dump
    upsert_all(dm)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/oplog_manager.py", line 474, in upsert_all
    dm.bulk_upsert(docs_to_dump(namespace), mapped_ns, long_ts)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/util.py", line 38, in wrapped
    reraise(new_type, exc_value, exc_tb)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/util.py", line 32, in wrapped
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/doc_managers/elastic_doc_manager.py", line 189, in bulk_upsert
    for ok, resp in responses:
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/helpers/__init__.py", line 118, in streaming_bulk
    raise e
ConnectionFailed: ConnectionError(('Connection aborted.', error(111, 'Connection refused'))) caused by: ProtocolError(('Connection aborted.', error(111, 'Connection refused')))

Thanks

On updating a document in Mongo Db I get the following error in solr adn document is not updated.I do have the _id field as unique key

108554676 [http-bio-8080-exec-13] INFO org.apache.solr.update.processor.LogUpdateProcessor – [bookfalcons] webapp=/solr path=/update/ params={commit=true} {} 0 0 108554677 [http-bio-8080-exec-13] ERROR org.apache.solr.core.SolrCore – org.apache.solr.common.SolrException: Document is missing mandatory uniqueKey field: _id at org.apache.solr.update.AddUpdateCommand.getIndexedId(AddUpdateCommand.java:92) at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:716) at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:556) at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100) at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:247) at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:174) at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1916) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:780) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:427) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:217) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:724)

How to use mongo connector to indexing fields of embedded documents for solr? For example the field StreetName of:

{
  "_id" : ObjectId("52fa3674395d4602b8be4a1b"),
  "AddressDirectory" : {
    "Owner" : "Mayank",
    "Age" : "24",
    "Company" : "BIPL",
    "Address" : {
      "HouseNo" : "4",
      "StreetName" : "Rohini",
      "City" : "Delhi"
    }
  }
} 

The manual says the connector flattens the nested documents but in solr I cant see the flatted field StreetName.

I am new here for mongo connector with solr+mongo. Can you please help me here I am running command after setting up my replica set of mongo C:\Python27\Scripts>mongo-connector -m ppispcw28:27017 -n admin.servicenow -t http://estpcsw173:8983/solr/#/ -d solr_doc_manager

Output: No handlers could be found for logger "mongo_connector.util" Traceback (most recent call last): File "C:\Python27\lib\runpy.py", line 162, in run_module_as_main "main", fname, loader, pkg_name) File "C:\Python27\lib\runpy.py", line 72, in run_code exec code in run_globals File "C:\Python27\Scripts\mongo-connector.exe__main.py", line 9, in File "C:\Python27\lib\site-packages\mongo_connector\util.py", line 85, in wrapped func(args, _kwargs) File "C:\Python27\lib\site-packages\mongo_connector\connector.py", line 998, in main conf.parse_args() File "C:\Python27\lib\site-packages\mongo_connector\config.py", line 114, in parse_args option, dict((k, values.get(k)) for k in option.cli_names)) File "C:\Python27\lib\site-packages\mongo_connector\connector.py", line 789, in apply_doc_managers dm_instances.append(module.DocManager(target_url, *kwargs)) File "C:\Python27\lib\site-packages\mongo_connector\doc_managers\solr_doc_manager.py", line 76, in init self._build_fields() File "C:\Python27\lib\site-packages\mongo_connector\util.py", line 32, in wrapped return f(args, *_kwargs) File "C:\Python27\lib\site-packages\mongo_connector\doc_managers\solr_doc_manager.py", line 95, in _build_fields result = decoder.decode(declared_fields) File "C:\Python27\lib\json\decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Python27\lib\json\decoder.py", line 384, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded My primary suspect is solr config (as I am new for this too) Please help me here

2014-08-01 14:37:54,640 - ERROR - Call to <function at 0x1fb9d70> failed too many times in retry_until_ok Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 551, in *bootstrap_inner self.run() File "build/bdist.linux-x86_64/egg/mongo_connector/oplog_manager.py", line 141, in run cursor, cursor_len = self.init_cursor() File "build/bdist.linux-x86_64/egg/mongo_connector/oplog_manager.py", line 570, in init_cursor first_oplog_entry = retry_until_ok(lambda: cursor[0]) File "build/bdist.linux-x86_64/egg/mongo_connector/util.py", line 53, in retry_until_ok return func(_args, _kwargs) File "build/bdist.linux-x86_64/egg/mongo_connector/oplog_manager.py", line 570, in first_oplog_entry = retry_until_ok(lambda: cursor[0]) File "build/bdist.linux-x86_64/egg/pymongo/cursor.py", line 597, in __getitem raise IndexError("no such item for Cursor instance") IndexError: no such item for Cursor instance

Once this error pops up it looks like the connector is no longer processing any commands. I'll update a field in the database and I do not see the changes in elasticsearch.

I am able to successfully integrate between MONGODB & SOLR, using MONGO-CONNECTOR. And whenever, I update or add any thing, in the sample collection i have created, it copies only two or three fields in a documents, and rest of the fields data are not copied into solr. This is some thing I am not able to do it.

This is my collection and its document details. Name of collection: testdb

document inserted as follows:

db.testdb.insert( { ... _id: "101", ... name: "test", ... description: "descr", ... mydesc: "mydescr", ... nmdsc: "nmdsc1", ... coords: "coords1" ... })

And the data sync between solr and mongo logs says successful: 2014-01-17 19:35:38,462 - INFO - Finished 'http://:/solr/update/?commit=true' (post) with body '' in 0.210 seconds.

But when I execute a query to see the document data it says only these fields data:

{ "responseHeader": { "status": 0, "QTime": 0, "params": { "q": ":", "wt": "json" } }, "response": { "numFound": 1, "start": 0, "docs": [ { "id": "101", "description": "descr", "name": "test", "version": 1457486601392226300 } ] } }

Clearly i can see that following fields & respective data are not copied into solr:

... mydesc: "mydescr", ... nmdsc: "nmdsc1", ... coords: "coords1"

Following is my schema.xml:

For some reason the schema.xml is not getting copied, it can be found in the following link, which I had raised question in stackoverflow a week ago:

http://stackoverflow.com/questions/21188334/not-all-fields-are-copied-from-mongodb-to-solr-after-integration-using-mongo-con

> mongo-connector -m localhost:27017 -t localhost:9200 -d elastic2_doc_manager Traceback (most recent call last): File "c:\programdata\anaconda3\lib\site-packages\mongo_connector\connector.py", line 1098, in import_dm_by_path module = import(package, fromlist=(package,)) ModuleNotFoundError: No module named 'mongo_connector.doc_managers.elastic2_doc_manager'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "c:\programdata\anaconda3\lib\runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "c:\programdata\anaconda3\lib\runpy.py", line 87, in run_code exec(code, run_globals) File "C:\ProgramData\Anaconda3\Scripts\mongo-connector.exe_main.py", line 7, in File "c:\programdata\anaconda3\lib\site-packages\mongo_connector\util.py", line 107, in wrapped func(*args, **kwargs) File "c:\programdata\anaconda3\lib\site-packages\mongo_connector\connector.py", line 1409, in main conf.parse_args() File "c:\programdata\anaconda3\lib\site-packages\mongo_connector\config.py", line 124, in parse_args option.apply_function( File "c:\programdata\anaconda3\lib\site-packages\mongo_connector\connector.py", line 1122, in apply_doc_managers DocManager = import_dm_by_name(dm["docManager"]) File "c:\programdata\anaconda3\lib\site-packages\mongo_connector\connector.py", line 1089, in import_dm_by_name return import_dm_by_path(full_name) File "c:\programdata\anaconda3\lib\site-packages\mongo_connector\connector.py", line 1104, in import_dm_by_path raise errors.InvalidConfiguration( mongo_connector.errors.InvalidConfiguration: Could not import mongo_connector.doc_managers.elastic2_doc_manager. It could be that this doc manager has been moved out of this project and is maintained elsewhere. Make sure that you have the doc manager installed alongside mongo-connector. Check the README for a list of available doc managers. ImportError: No module named 'mongo_connector.doc_managers.elastic2_doc_manager'

> pip install 'mongo-connector[elastic5]' WARNING: Ignoring invalid distribution -acktrader (c:\programdata\anaconda3\lib\site-packages) ERROR: Invalid requirement: "'mongo-connector[elastic5]'" WARNING: Ignoring invalid distribution -acktrader (c:\programdata\anaconda3\lib\site-packages) WARNING: Ignoring invalid distribution -acktrader (c:\programdata\anaconda3\lib\site-packages) WARNING: Ignoring invalid distribution -acktrader (c:\programdata\anaconda3\lib\site-packages)

My versions MongoDB: 5.0 ElasticSearch: 8.4.0 Python: 3.8.11 OS: Windows 10

Hey,

firstup please tell me what additional information is needed to debug this error.

I ran the following command: mongo-connector --unique-key=id -n news-articles.articles -m localhost:27017 -t http://localhost:8983/solr/mongo_solr_collection -d solr_doc_manager

(Yes i am calling the collection with -t because the core does not work (I get an Error 404). )

The connector starts and logs the following:

2022-02-15 18:23:34,584 [ALWAYS] mongo_connector.connector:50 - Python version: 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110]
2022-02-15 18:23:34,586 [ALWAYS] mongo_connector.connector:50 - Platform: Linux-5.10.0-11-amd64-x86_64-with-glibc2.31
2022-02-15 18:23:34,587 [ALWAYS] mongo_connector.connector:50 - pymongo version: 4.0.1
2022-02-15 18:23:34,587 [WARNING] mongo_connector.connector:170 - MongoConnector: Can't find /srv/news_crawler/oplog.timestamp, attempting to create an empty progress log
2022-02-15 18:23:34,597 [ALWAYS] mongo_connector.connector:50 - Source MongoDB version: 5.0.6
2022-02-15 18:23:34,597 [ALWAYS] mongo_connector.connector:50 - Target DocManager: mongo_connector.doc_managers.solr_doc_manager version: 0.1.0
2022-02-15 18:25:33,715 [ERROR] mongo_connector.util:97 - Call to Collection(Database(MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True, replicaset='rs0'), 'news-articles'), 'collection_names') failed too many times in retry_until_ok
Traceback (most recent call last):
  File "/srv/news_crawler/venv/lib/python3.9/site-packages/mongo_connector/util.py", line 79, in retry_until_ok
    return func(*args, **kwargs)
  File "/srv/news_crawler/venv/lib/python3.9/site-packages/pymongo/collection.py", line 2579, in __call__
    raise TypeError("'Collection' object is not callable. If you "
TypeError: 'Collection' object is not callable. If you meant to call the 'collection_names' method on a 'Database' object it is failing because no such method exists.

I don't know what to do with this. MongoDB is up and running:

Current Mongosh Log ID:	620be3f843ef12a095c537e4
Connecting to:		mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+1.1.9
Using MongoDB:		5.0.6
Using Mongosh:		1.1.9
For mongosh info see: https://docs.mongodb.com/mongodb-shell/
   The server generated these startup warnings when booting:
   2022-02-15T17:19:31.349+01:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
   2022-02-15T17:19:32.383+01:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
   2022-02-15T17:19:32.384+01:00: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never'
Warning: Found ~/.mongorc.js, but not ~/.mongoshrc.js. ~/.mongorc.js will not be loaded.
  You may want to copy or rename ~/.mongorc.js to ~/.mongoshrc.js.
rs0 [direct: primary] test>

and the designated collection has several thousand items in it:

rs0 [direct: primary] news-articles> db.articles.find().count()
147203

Is rs0 setup wrong?

And solr is also up and running:

$ curl http://localhost:8983/solr/admin/cores?action=STATUS
{
  "responseHeader":{
    "status":0,
    "QTime":0},
  "initFailures":{},
  "status":{
    "mongo_solr_collection":{
      "name":"mongo_solr_collection",
      "instanceDir":"/srv/solr-8.11.1/server/solr/mongo_solr_collection",
      "dataDir":"/srv/solr-8.11.1/server/solr/mongo_solr_collection/data/",
      "config":"solrconfig.xml",
      "schema":"managed-schema",
      "startTime":"2022-02-15T15:09:55.153Z",
      "uptime":9084767,
      "index":{
        "numDocs":0,
        "maxDoc":0,
        "deletedDocs":0,
        "indexHeapUsageBytes":0,
        "version":2,
        "segmentCount":0,
        "current":true,
        "hasDeletions":false,
        "directory":"org.apache.lucene.store.NRTCachingDirectory:NRTCachingDirectory(MMapDirectory@/srv/solr-8.11.1/server/solr/mongo_solr_collection/data/index [email protected]; maxCacheMB=48.0 maxMergeSizeMB=4.0)",
        "segmentsFile":"segments_1",
        "segmentsFileSizeInBytes":69,
        "userData":{},
        "sizeInBytes":69,
        "size":"69 bytes"}}}}

I really don't know what I am doing wrong. Would be great if someone could point me in the right direction. 🙏

Problem

We have various mongo servers that we sync with a central server using mongo-connector with the mongo_doc_manager. They all sync into the same collection on the central server. When a new system is initialized, it has nothing in the DB and mongo-connector is running. When the first entry is made into the local DB, there is an oplog entry to create the collection. When mongo-connector tries to replay that command on the target DB, pymongo throws an exception because the collection already exists on the central server.

Note: this problem does not present itself if mongo-connector is started (without an existing oplog.timestamp) after the first entry in the local collection has already been made.

Solution

This problem is easily avoided by not trying to explicitly create the collection, effectively ignoring the create entry in the oplog. This does not cause any problems because mongo creates collections automatically whenever a document is inserted into a collection that does not yet exist. The only reason to explicitly create a collection is if special options are specified as per the documentation:

Normally collection creation is automatic. This method should only be used to specify options on creation. CollectionInvalid will be raised if the collection already exists.

Since mongo_doc_manager does not specify any options in the create_collection() call, that call should not be made.

I have Solr 8.9.0 installed on MacOS Catalina, and I’m trying to integrate it with MongoDB 5.0 Community Edition. I have installed the mongo-connector using pip3 install mongo-connector.

When I try to run a replica set using

mongod --replSet myDevReplSet

I get the following:

{"t":{"$date":"2021-10-12T21:30:25.414+01:00"},"s":"I", "c":"NETWORK", "id":4915701, "ctx":"-","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":13},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":13},"outgoing":{"minWireVersion":0,"maxWireVersion":13},"isInternalClient":true}}} {"t":{"$date":"2021-10-12T21:30:25.416+01:00"},"s":"I", "c":"CONTROL", "id":23285, "ctx":"-","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"} {"t":{"$date":"2021-10-12T21:30:25.420+01:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"} {"t":{"$date":"2021-10-12T21:30:25.420+01:00"},"s":"I", "c":"NETWORK", "id":4648602, "ctx":"main","msg":"Implicit TCP FastOpen in use."} {"t":{"$date":"2021-10-12T21:30:25.421+01:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"} {"t":{"$date":"2021-10-12T21:30:25.421+01:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"} {"t":{"$date":"2021-10-12T21:30:25.421+01:00"},"s":"I", "c":"REPL", "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationDonorService","ns":"config.tenantMigrationDonors"}} {"t":{"$date":"2021-10-12T21:30:25.421+01:00"},"s":"I", "c":"REPL", "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationRecipientService","ns":"config.tenantMigrationRecipients"}} {"t":{"$date":"2021-10-12T21:30:25.421+01:00"},"s":"I", "c":"CONTROL", "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":7591,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"TTL003164"}} {"t":{"$date":"2021-10-12T21:30:25.422+01:00"},"s":"I", "c":"CONTROL", "id":23403, "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"5.0.3","gitVersion":"657fea5a61a74d7a79df7aff8e4bcf0bc742b748","modules":[],"allocator":"system","environment":{"distarch":"x86_64","target_arch":"x86_64"}}}} {"t":{"$date":"2021-10-12T21:30:25.422+01:00"},"s":"I", "c":"CONTROL", "id":51765, "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Mac OS X","version":"19.6.0"}}} {"t":{"$date":"2021-10-12T21:30:25.422+01:00"},"s":"I", "c":"CONTROL", "id":21951, "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"replication":{"replSet":"myDevReplSet"}}}} {"t":{"$date":"2021-10-12T21:30:25.423+01:00"},"s":"E", "c":"CONTROL", "id":20568, "ctx":"initandlisten","msg":"Error setting up listener","attr":{"error":{"code":9001,"codeName":"SocketException","errmsg":"Address already in use"}}} {"t":{"$date":"2021-10-12T21:30:25.423+01:00"},"s":"I", "c":"REPL", "id":4784900, "ctx":"initandlisten","msg":"Stepping down the ReplicationCoordinator for shutdown","attr":{"waitTimeMillis":15000}} {"t":{"$date":"2021-10-12T21:30:25.423+01:00"},"s":"I", "c":"COMMAND", "id":4784901, "ctx":"initandlisten","msg":"Shutting down the MirrorMaestro"} {"t":{"$date":"2021-10-12T21:30:25.423+01:00"},"s":"I", "c":"SHARDING", "id":4784902, "ctx":"initandlisten","msg":"Shutting down the WaitForMajorityService"} {"t":{"$date":"2021-10-12T21:30:25.423+01:00"},"s":"I", "c":"NETWORK", "id":4784905, "ctx":"initandlisten","msg":"Shutting down the global connection pool"} {"t":{"$date":"2021-10-12T21:30:25.423+01:00"},"s":"I", "c":"REPL", "id":4784907, "ctx":"initandlisten","msg":"Shutting down the replica set node executor"} {"t":{"$date":"2021-10-12T21:30:25.423+01:00"},"s":"I", "c":"NETWORK", "id":4784918, "ctx":"initandlisten","msg":"Shutting down the ReplicaSetMonitor"} {"t":{"$date":"2021-10-12T21:30:25.423+01:00"},"s":"I", "c":"SHARDING", "id":4784921, "ctx":"initandlisten","msg":"Shutting down the MigrationUtilExecutor"} {"t":{"$date":"2021-10-12T21:30:25.423+01:00"},"s":"I", "c":"ASIO", "id":22582, "ctx":"MigrationUtil-TaskExecutor","msg":"Killing all outstanding egress activity."} {"t":{"$date":"2021-10-12T21:30:25.423+01:00"},"s":"I", "c":"COMMAND", "id":4784923, "ctx":"initandlisten","msg":"Shutting down the ServiceEntryPoint"} {"t":{"$date":"2021-10-12T21:30:25.423+01:00"},"s":"I", "c":"CONTROL", "id":4784925, "ctx":"initandlisten","msg":"Shutting down free monitoring"} {"t":{"$date":"2021-10-12T21:30:25.423+01:00"},"s":"I", "c":"CONTROL", "id":4784927, "ctx":"initandlisten","msg":"Shutting down the HealthLog"} {"t":{"$date":"2021-10-12T21:30:25.423+01:00"},"s":"I", "c":"CONTROL", "id":4784928, "ctx":"initandlisten","msg":"Shutting down the TTL monitor"} {"t":{"$date":"2021-10-12T21:30:25.423+01:00"},"s":"I", "c":"CONTROL", "id":4784929, "ctx":"initandlisten","msg":"Acquiring the global lock for shutdown"} {"t":{"$date":"2021-10-12T21:30:25.423+01:00"},"s":"I", "c":"-", "id":4784931, "ctx":"initandlisten","msg":"Dropping the scope cache for shutdown"} {"t":{"$date":"2021-10-12T21:30:25.423+01:00"},"s":"I", "c":"FTDC", "id":4784926, "ctx":"initandlisten","msg":"Shutting down full-time data capture"} {"t":{"$date":"2021-10-12T21:30:25.423+01:00"},"s":"I", "c":"CONTROL", "id":20565, "ctx":"initandlisten","msg":"Now exiting"} {"t":{"$date":"2021-10-12T21:30:25.423+01:00"},"s":"I", "c":"CONTROL", "id":23138, "ctx":"initandlisten","msg":"Shutting down","attr":{"exitCode":48}}

If I then login to MongoDB and run rs.initiate(), I get:

MongoServerError: This node was not started with the replSet option

Can anyone point me in the right direction?

Cheers!

I'm using the default mongo_connector command:

mongo-connector -m :
-t <replication endpoint URL, e.g. http://localhost:8983/solr>
-d <name of doc manager, e.g., solr_doc_manager>

I get the error on both glitch.com and on my local machine:

"-t: command not found"