• [x] Tested with the latest Haystack release
• [ ] Tested with the current Haystack master branch

Expected behavior

Haystack responds as expected (with an appropriate JSON response) and is able to be queried using a Solr backend

Actual behavior

500 Internal Server Error: KeyError: 'django_ct'

Steps to reproduce the behavior

1. Create Django project and add a model
2. You can try using the build_solr_schema command, but that produces an output incompatible (deprecated field types) with both Solr 6.4.1 and Solr 5.5.4 (so I basically have been trying to use this for the schema. I have tried using the schema output from both cases mentioned in both Solr 6.4.1 and Solr 5.5.4 but have not had any luck.
3. I have ensured the django_id and django_ct are defined as fields but that does not seem to be the issue here.

I suppose my main question to the community is whether or not anyone has gotten Haystack working with Solr 5.5.4 or Solr 6.4.1? I don't feel we should have to downgrade Solr any further due to lack of new features and for security reasons. Currently in our Django application we are using PySolr directly to query Solr and return JSON. Thanks for your time!

Configuration

• Operating system version: Ubuntu 16.04.2 LTS
• Search engine version: Solr 6.4.1 and Solr 5.5.4
• Python version: 3.5.2
• Django version: 1.10.5
• Haystack version: 2.6.0

elasticsearch 5.0 is in alpha already and should be added on future release of haystack. Maybe skipping ealsticsearch 2.0 support would be an option. elasticsearch-py already supports elasticsearch 5.0

https://www.elastic.co/guide/en/elasticsearch/reference/master/breaking-changes-5.0.html

Continuing and finishing PR #1148 after @acdha comment https://github.com/django-haystack/django-haystack/pull/1148#issuecomment-172620864

Test coverage seems good as well as Travis builds. Rebase from master is freshly done

Thanks for considering merging it.

This change is

I'm in the process of upgrading a project to Django 1.7. Rebuilding or updating the haystack index fails in combination with django-grappelli's dashboard. Please note, that in latest grappelli, the dashboard module/app doesn't have a models.py (which is allowed in Django 1.7 as far as i understand).

$ ./manage.py rebuild_index

WARNING: This will irreparably remove EVERYTHING from your search index in connection 'default'.
Your choices after this are to restore from backups or rebuild via the `rebuild_index` command.
Are you sure you wish to continue? [y/N] y
Removing all documents from your index because you said so.
All documents removed.
ERROR:root:Error updating grappelli.dashboard using default 
Traceback (most recent call last):
File "/path/to/env/local/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 189, in handle_label
    self.update_backend(label, using)
File "/path/to/env/local/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 203, in update_backend
    for model in get_models(label):
File "/path/to/env/local/lib/python2.7/site-packages/haystack/utils/app_loading.py", line 41, in get_models
    return [apps.get_app_config(app_label).get_model(model_name)]
File "/path/to/env/local/lib/python2.7/site-packages/django/apps/config.py", line 166, in get_model
    "App '%s' doesn't have a '%s' model." % (self.label, model_name))
LookupError: App 'grappelli' doesn't have a 'dashboard' model.
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
File "/path/to/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
File "/path/to/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
File "/path/to/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
File "/path/to/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
File "/path/to/env/local/lib/python2.7/site-packages/haystack/management/commands/rebuild_index.py", line 16, in handle
    call_command('update_index', **options)
File "/path/to/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 115, in call_command
    return klass.execute(*args, **defaults)
File "/path/to/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
File "/path/to/env/local/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 184, in handle
    return super(Command, self).handle(*items, **options)
File "/path/to/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 503, in handle
    label_output = self.handle_label(label, **options)
File "/path/to/env/local/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 189, in handle_label
    self.update_backend(label, using)
File "/path/to/env/local/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 203, in update_backend
    for model in get_models(label):
File "/path/to/env/local/lib/python2.7/site-packages/haystack/utils/app_loading.py", line 41, in get_models
    return [apps.get_app_config(app_label).get_model(model_name)]
File "/path/to/env/local/lib/python2.7/site-packages/django/apps/config.py", line 166, in get_model
    "App '%s' doesn't have a '%s' model." % (self.label, model_name))
LookupError: App 'grappelli' doesn't have a 'dashboard' model.

Django: 1.7.1 django-haystack: 2.3.1 django-grappelli: 2.6.2

INSTALLED_APPS = (
    'grappelli.dashboard',
    'grappelli',
    # ...
    'haystack',
)

The management command works when 'grappelli.dashboard' is removed from INSTALLED_APPS.

Based on #1460 - will be properly rebased once it's accepted.

So far only two features seem to be failing, according to the test suite:

• Faceted search (ES seems to throw an error about fielddata)
• Autocomplete

Any help to solve any of those would be appreciated.

This addresses the need for django-haystack's django style class based views. This is based on the discussion from the following PR:

• https://github.com/toastdriven/django-haystack/pull/826

This work was originally done by @bennylope with only minor changes from his original file:

• https://github.com/bennylope/elasticstack/blob/master/elasticstack/views.py

This PR addresses the following issues:

• Fixes #348
• Fixes #350
• Fixes #873
• Resolves #826
• Resolves #1131
• Fixes many tests that were previously broken prior to this PR

Hi, this PR adds Django 2.0 compatibility. As the PR https://github.com/django-haystack/django-haystack/pull/1557 did not see any progress recently I implemented everything needed for Django 2.0 compatibility here.

In particular I rewrote the rebuild_index command to strip certain arguments that are not used in the clear_index and update_index commands respectively. This is implemented in a relatively adhoc way, however looking at the amount of changes made to rebuild_index in the past I figured this might be the easiest way to implement this change.

Additionally I added Django 2.0 to the tests and excluded the combination Django2.0 + Python2.7 from the test suite so that the tests still pass. Maybe one should also consider adding python 3.6 to the test suite, however I did not do that yet.

Also note the change I made to test_haystack/solr_tests/test_solr_backend.py. This was necessary due to the recent update in pysolr. That update broke the django-haystack tests, as pysolr.__version__ is now a string instead of a tuple. The fix I implemented should be backwards compatible.

Please let me know if there is anything this MR needs to get merged, cheers, Martin

This is the Major upgrade addressing #1502 and started in #1488 to handle Solr6 (specifically 6.4 and 6.5). It adds new command parameters to build_solr_schema. The two new options allow you to specify a directory to put the config files in, and to automatically reload the core. This involved putting a new HAYSTACK_CONNECTIONS variable in place 'ADMIN_URL'.

One major change was that the template got renamed to schema.xml in the template directory instead of solr,xml

Along the way spelling suggestions format parsing was updated and handled (though the legacy style of returning only one suggestion was maintained -- we now have the plumbing to supply multiple). Upgrades were made to the tests to test spelling suggestion without collation to improve code coverage.

Upgrades were also made to handle django 11 by fixing the context passing to be native dicts in the main code base and the tests.

PEP8 changes were made, though not exhaustively.

Updated the Travis config to only use Trusty images as the others didn't have access to Oracle8 which was also required for newer Solr. Expanded test matrix to handle Django 1.11

TBD: Upgrade haystack to handle multiple spelling suggestions for Solr Something about ExtendedCollation from @acdha's other comments Expand matrix to handle multiple Solr versions (don't know what a reasonable anchor version is) probably many more:)

Gives haystack support for Elasticsearch 2.x

• Adds a new backend haystack.backends.elasticsearch2_backend.Elasticsearch2SearchEngine
• Reuses the code and tests for ES 1.x
• Changes facets (aggregations on ES 2.x)
• Changes delete_by_query (scroll API and bulk actions)
• "more like this" (More Like This Query)
• Modifies the dependecy on elasticsearch library version

Issue: https://github.com/django-haystack/django-haystack/issues/1247

Solr 5.0.0 is easy to deploy with cloud Cores with automatic managed-schema and no more schem.xml. Solr 5.0.0 can edit a core's configuration via with the Schema REST API. This is what this pull request does. Along with an updated documentation.

Refers to issue #1147 and try to resolve it...

Remove django.conf.urls.patterns, since they already have been removed from the doc and will be removed in Django 2.0

This removes in Django 1.9 the warning:

RemovedInDjango110Warning: django.conf.urls.patterns()
is deprecated and will be removed in Django 1.10. Update your urlpatterns to be a list of
django.conf.urls.url() instances instead.
  name='redactor_upload_file'),

Also Updates the command update_index to match new arguments parser in Django

• [ X] Tested with the current Haystack master branch

Expected behaviour

index is built

Actual behaviour

index is not built due to incompatibility with open search

Steps to reproduce the behaviour

try use AWS open search (which is elastic compatible with haystacks)

Configuration

• Operating system version: docker python
• Search engine version: AWS opensearch
• Python version: 3.10.1
• Django version: 4.1.3
• Haystack version: 3.2.1

Im trying to use AWS open search with Django Haystacks, which is fully compatible (Amazon have suggested). However when I try launch "./manage.py rebuild_index" when pointing a connection at the platform I am met with the error

    Traceback (most recent call last):
      File "/data/app/myapp/./manage.py", line 22, in <module>
        execute_from_command_line(sys.argv)
      File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
        utility.execute()
      File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 440, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
      File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 402, in run_from_argv
        self.execute(*args, **cmd_options)
      File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 448, in execute
        output = self.handle(*args, **options)
      File "/usr/local/lib/python3.10/site-packages/haystack/management/commands/rebuild_index.py", line 64, in handle
        call_command("clear_index", **clear_options)
      File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 198, in call_command
        return command.execute(*args, **defaults)
      File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 448, in execute
        output = self.handle(*args, **options)
      File "/usr/local/lib/python3.10/site-packages/haystack/management/commands/clear_index.py", line 64, in handle
        backend.clear(commit=self.commit)
      File "/usr/local/lib/python3.10/site-packages/haystack/backends/elasticsearch7_backend.py", line 117, in clear
        self.conn.indices.delete(index=self.index_name, ignore=404)
      File "/usr/local/lib/python3.10/site-packages/elasticsearch/client/utils.py", line 347, in _wrapped
        return func(*args, params=params, headers=headers, **kwargs)
      File "/usr/local/lib/python3.10/site-packages/elasticsearch/client/indices.py", line 334, in delete
        return self.transport.perform_request(
      File "/usr/local/lib/python3.10/site-packages/elasticsearch/transport.py", line 421, in perform_request
        _ProductChecker.raise_error(self._verified_elasticsearch)
      File "/usr/local/lib/python3.10/site-packages/elasticsearch/transport.py", line 638, in raise_error
        raise UnsupportedProductError(message)
    elasticsearch.exceptions.UnsupportedProductError: The client noticed that the server is not a supported distribution of Elasticsearch

when we've dug into this by looking in "site-packages/elasticsearch/transport.py" and the differences between the amazon headers and an elastic search header the only differences between the two is that amazon openserach "build_flavour" in it. Build flavour is used in the check_product function from transport.py

    @classmethod
        def check_product(cls, headers, response):
            # type: (dict[str, str], dict[str, str]) -> int
            """Verifies that the server we're talking to is Elasticsearch.
            Does this by checking HTTP headers and the deserialized
            response to the 'info' API. Returns one of the states above.
            """
        try:
            version = response.get("version", {})
            version_number = tuple(
                int(x) if x is not None else 999
                for x in re.search(
                    r"^([0-9]+)\.([0-9]+)(?:\.([0-9]+))?", version["number"]
                ).groups()
            )
        except (KeyError, TypeError, ValueError, AttributeError):
            # No valid 'version.number' field, effectively 0.0.0
            version = {}
            version_number = (0, 0, 0)

        # Check all of the fields and headers for missing/valid values.
        try:
            bad_tagline = response.get("tagline", None) != "You Know, for Search"
            bad_build_flavor = version.get("build_flavor", None) != "default"
            bad_product_header = (
                headers.get("x-elastic-product", None) != "Elasticsearch"
            )
        except (AttributeError, TypeError):
            bad_tagline = True
            bad_build_flavor = True
            bad_product_header = True

        # 7.0-7.13 and there's a bad 'tagline' or unsupported 'build_flavor'
        if (7, 0, 0) <= version_number < (7, 14, 0):
            if bad_tagline:
                return cls.UNSUPPORTED_PRODUCT
            elif bad_build_flavor:
                return cls.UNSUPPORTED_DISTRIBUTION

        elif (
            # No version or version less than 6.x
            version_number < (6, 0, 0)
            # 6.x and there's a bad 'tagline'
            or ((6, 0, 0) <= version_number < (7, 0, 0) and bad_tagline)
            # 7.14+ and there's a bad 'X-Elastic-Product' HTTP header
            or ((7, 14, 0) <= version_number and bad_product_header)
        ):
            return cls.UNSUPPORTED_PRODUCT

        return True

I commented out the build flavour checks in this function, and then the index built successfully! but ive noticed the file "elasticsearch/transport.py" and the code "def check_product" doesn't exist in this repo. I don't know where this is pulled from or how I can create a build that will comment out these lines

Thanks

Hey, thanks for contributing to Haystack. Please review the contributor guidelines and confirm that the tests pass with at least one search engine.

Once your pull request has been submitted, the full test suite will be executed on https://github.com/django-haystack/django-haystack/actions/workflows/test.yml. Pull requests with passing tests are far more likely to be reviewed and merged.

We are considering refactoring our enterprise search engine using NLP + Elasticsearch. Researching this topic - reading articles and watching some videos, I encountered Haystack - the solution provided by DeepsetAI. But since our backend is written in Django, I eventually came across this repository.

So I was wondering, what is the relation between django-haystack and deepset-ai-haystack? Are they both based on the same underlying technology (like ES and Solr are based on Lucene)? Is one the fork of another?

I'm just trying to gather as much information as possible before committing to one or the other solution.

• [x] Tested with the latest Haystack release
• [x] Tested with the current Haystack master branch

Expected behaviour

Given I have a proxy model class:

class CustomOrder(Order):
    class Meta:
        proxy = True

And an index for the parent model:

class OrderIndex(BaseIndex):
    def get_model(self):
        return Order

I would expect that instances of the proxy model class would be included in the index.

Actual behaviour

Instances of CustomOrder are ignored when indexing.

Is there a way to have the CustomOrder class included in the OrderIndex so that both the CustomOrder and Order class go to the one index?

• [X] Tested with the latest Haystack release
• [X] Tested with the current Haystack master branch

Expected behaviour

When we do a range facet query with open bound option (which is for now not part of Haystack but will be available once #978 is merged), Solr may return after, before, start and end fields as well as "counts" in the result (see Solr doc). These fields should be used in the transformed result. e.g. for a raw range result of:

{'price': {'counts': ['0.0', 0, '30.0', 0, '60.0', 0, '90.0', 0, '120.0', 0, '150.0', 3, '180.0', 1], 'gap': 30.0, 'before': 0, 'after': 5, 'between': 4, 'start': 0.0, 'end': 210.0}}

we would expects haystack to return the following range result:

{'price': [('0.0', 0), ('30.0', 0), ('60.0', 0), ('90.0', 0), ('120.0', 0), ('150.0', 3), ('180.0', 1), (210.0, 5)]}

I don't see any way to represent the [* to 0] range (result given in before field) in current haystack representation, any idea?

Actual behaviour

"after" and "end" raw result fields are ignored

e.g. for a raw range result of:

{'price': {'counts': ['0.0', 0, '30.0', 0, '60.0', 0, '90.0', 0, '120.0', 0, '150.0', 3, '180.0', 1], 'gap': 30.0, 'before': 0, 'after': 5, 'between': 4, 'start': 0.0, 'end': 210.0}}

we get the following:

{'price': [('0.0', 0), ('30.0', 0), ('60.0', 0), ('90.0', 0), ('120.0', 0), ('150.0', 3), ('180.0', 1)}

The fact that there is 5 results for the "210 and more" facet is lost and we even lose the fact that the last range is 180-210 and not 180+

Steps to reproduce the behaviour

1. Implement range faceting
2. create range query using start, gap, end and other=after options
3. Look at the result

Configuration

• Operating system version: Debian GNU/Linux 10 (buster)
• Search engine version: Apache Solr™ version 8.8.2
• Python version: Python 3.10.2
• Django version: 3.2.15 / 4.0
• Haystack version: 3.2.1