Description: Added a visual bar display for the following columns in performance analysis table and semantic similar utterance table

Performance analysis table columns:

1. Outcome columns
2. custom metric columns
3. ECE Semantic similar utterance column:
4. similarity

Checklist:

You should check all boxes before the PR is ready. If a box does not apply, check it to acknowledge it.

• [x] PRE-COMMIT. You ran pre-commit on all commits, or else, you ran pre-commit run --all-files at the end.
• [x] FRONTEND TYPES. Regenerate the front-ent types if you played with types and routes. Run cd webapp && yarn types while the back-end is running.
• [x] USER CHANGES. The changes are added to CHANGELOG.md and the documentation, if they impact our users.
• [x] DEV CHANGES.
  • Update the documentation if this PR changes how to develop/launch on the app.
  • Update the README files and our wiki for any big design decisions, if relevant.
  • Add unit tests, docstrings, typing and comments for complex sections.

Resolve #231

Description:

Better align with the language we are proposing in the EMNLP paper.

Checklist:

You should check all boxes before the PR is ready. If a box does not apply, check it to acknowledge it.

• [x] ISSUE NUMBER. You linked the issue number (Ex: Resolve #XXX).
• [x] PRE-COMMIT. You ran pre-commit on all commits, or else, you ran pre-commit run --all-files at the end.
• [x] USER CHANGES. The changes are added to CHANGELOG.md and the documentation, if they impact our users.
• [x] DEV CHANGES.
  • Update the documentation if this PR changes how to develop/launch on the app.
  • Update the README files and our wiki for any big design decisions, if relevant.
  • Add unit tests, docstrings, typing and comments for complex sections.

This is following the versioning of our documentation.

URLs without the version (for example https://servicenow.github.io/azimuth/getting-started that is linked in the README) still exist and seem to point to an older version (not main). I can imagine a couple of options:

1. Those URLs could redirect to /main/... ?
2. Those URLs could be the "main" ones? Replacing /main/...
3. Those URLs could die? And we make sure we convert all the links to them.

Resolve #

Description:

1. Added "desc" icon to Utterance Count to enable desc by default.

2. Added the tooltip for custom metric columns and ECE.

3. When there is only one pipeline configured...

The "Compare Pipelines" button on the dashboard is instead called "View details" and is still linking to the standalone page. The "Compare Baseline with" select menu is hidden.

The "Compare Baseline with" select menu has a default value of the first available pipeline.

Checklist:

You should check all boxes before the PR is ready. If a box does not apply, check it to acknowledge it.

• [x] ISSUE NUMBER. You linked the issue number (Ex: Resolve #XXX).
• [x] PRE-COMMIT. You ran pre-commit on all commits, or else, you ran pre-commit run --all-files at the end.
• [x] USER CHANGES. The changes are added to CHANGELOG.md and the documentation, if they impact our users.
• [x] DEV CHANGES.
  • Update the documentation if this PR changes how to develop/launch on the app.
  • Update the README files and our wiki for any big design decisions, if relevant.
  • Add unit tests, docstrings, typing and comments for complex sections.

Resolve #257

Description:

Redirect to page not found not working on root route

Usually everyone tends to type something on the root route ("/") to navigate to the next page. With the existing approach this feels a like a bad ux experience. And also later when the app needs to have different routes which are not related to data, they can go to their route like /settings/my_profile , /users/:userID. This will be useful when the app grows.

Checklist:

You should check all boxes before the PR is ready. If a box does not apply, check it to acknowledge it.

• [x] ISSUE NUMBER. You linked the issue number (Ex: Resolve #XXX).
• [x] PRE-COMMIT. You ran pre-commit on all commits, or else, you ran pre-commit run --all-files at the end.
• [ ] USER CHANGES. The changes are added to CHANGELOG.md and the documentation, if they impact our users.
• [ ] DEV CHANGES.
  • Update the documentation if this PR changes how to develop/launch on the app.
  • Update the README files and our wiki for any big design decisions, if relevant.
  • Add unit tests, docstrings, typing and comments for complex sections.

Description: Add description on each section of azimuth UI

Checklist:

You should check all boxes before the PR is ready. If a box does not apply, check it to acknowledge it.

• [x] PRE-COMMIT. You ran pre-commit on all commits, or else, you ran pre-commit run --all-files at the end.
• [x] FRONTEND TYPES. Regenerate the front-ent types if you played with types and routes. Run cd webapp && yarn types while the back-end is running.
• [x] USER CHANGES. The changes are added to CHANGELOG.md and the documentation, if they impact our users.
• [x] DEV CHANGES.
  • Update the documentation if this PR changes how to develop/launch on the app.
  • Update the README files and our wiki for any big design decisions, if relevant.
  • Add unit tests, docstrings, typing and comments for complex sections.

Is your feature request related to a problem? Please describe. pip install gdown doesn't work perhaps due to various dependency mismatches

Collecting gdown
  Using cached gdown-4.4.0.tar.gz (14 kB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/zia.hasan/anaconda3/bin/python /Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/yl/8d2pf1t54_3gknvrf_qs8_cc0000gp/T/pip-build-env-2rnpl_90/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel
       cwd: None
  Complete output (44 lines):
  Traceback (most recent call last):
    File "/Users/zia.hasan/anaconda3/lib/python3.8/runpy.py", line 194, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/Users/zia.hasan/anaconda3/lib/python3.8/runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/__main__.py", line 26, in <module>
      sys.exit(_main())
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_internal/cli/main.py", line 73, in main
      command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_internal/commands/__init__.py", line 104, in create_command
      module = importlib.import_module(module_path)
    File "/Users/zia.hasan/anaconda3/lib/python3.8/importlib/__init__.py", line 127, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
    File "<frozen importlib._bootstrap>", line 991, in _find_and_load
    File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 783, in exec_module
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 24, in <module>
      from pip._internal.cli.req_command import RequirementCommand, with_cleanup
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 16, in <module>
      from pip._internal.index.package_finder import PackageFinder
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 21, in <module>
      from pip._internal.index.collector import parse_links
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_internal/index/collector.py", line 14, in <module>
      from pip._vendor import html5lib, requests
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_vendor/requests/__init__.py", line 114, in <module>
      from . import utils
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_vendor/requests/utils.py", line 25, in <module>
      from . import certs
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_vendor/requests/certs.py", line 15, in <module>
      from pip._vendor.certifi import where
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_vendor/certifi/__init__.py", line 1, in <module>
      from .core import contents, where
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip/_vendor/certifi/core.py", line 12, in <module>
      from importlib.resources import read_text
    File "/Users/zia.hasan/anaconda3/lib/python3.8/importlib/resources.py", line 11, in <module>
      from typing import Iterable, Iterator, Optional, Set, Union   # noqa: F401
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/typing.py", line 1359, in <module>
      class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
    File "/Users/zia.hasan/anaconda3/lib/python3.8/site-packages/typing.py", line 1007, in __new__
      self._abc_registry = extra._abc_registry
  AttributeError: type object 'Callable' has no attribute '_abc_registry'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/zia.hasan/anaconda3/bin/python /Users/zia.hasan/anaconda3/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/yl/8d2pf1t54_3gknvrf_qs8_cc0000gp/T/pip-build-env-2rnpl_90/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel Check the logs for full command output.

Describe the solution you'd like Perhaps update the tutorial/documentation to better handle this. poetry dev-dependency?

Description:

Checklist:

You should check all boxes before the PR is ready. If a box does not apply, check it to acknowledge it.

• [x] PRE-COMMIT. You ran pre-commit on all commits, or else, you ran pre-commit run --all-files at the end.
• [x] FRONTEND TYPES. Regenerate the front-ent types if you played with types and routes. Run cd webapp && yarn types while the back-end is running.
• [x] USER CHANGES. The changes are added to CHANGELOG.md and the documentation, if they impact our users.
• [x] DEV CHANGES.
  • Update the documentation if this PR changes how to develop/launch on the app.
  • Update the README files and our wiki for any big design decisions, if relevant.
  • Add unit tests, docstrings, typing and comments for complex sections.

Resolve #295

Description:

It adds a scrollbar in the performance table, but I think it is tolerable.

Checklist:

You should check all boxes before the PR is ready. If a box does not apply, check it to acknowledge it.

• [x] ISSUE NUMBER. You linked the issue number (Ex: Resolve #XXX).
• [x] PRE-COMMIT. You ran pre-commit on all commits, or else, you ran pre-commit run --all-files at the end.
• [x] USER CHANGES. The changes are added to CHANGELOG.md and the documentation, if they impact our users.
• [x] DEV CHANGES.
  • Update the documentation if this PR changes how to develop/launch on the app.
  • Update the README files and our wiki for any big design decisions, if relevant.
  • Add unit tests, docstrings, typing and comments for complex sections.

Resolve #140

Description:

Reorder confusion matrix by default with reverse cuthill mckee, so that classes with confusion are closer to one another in the matrix.

Still TODO:

• [x] A toggle needs to be added in the UI to preserve the original class order if that is preferred.
• [x] The documentation also needs to be updated, but I'll wait for the UI to take the screenshot.

Checklist:

You should check all boxes before the PR is ready. If a box does not apply, check it to acknowledge it.

• [x] ISSUE NUMBER. You linked the issue number (Ex: Resolve #XXX).
• [x] PRE-COMMIT. You ran pre-commit on all commits, or else, you ran pre-commit run --all-files at the end.
• [x] FRONTEND TYPES. Regenerate the front-ent types if you played with types and routes. Run cd webapp && yarn types while the back-end is running.
• [x] USER CHANGES. The changes are added to CHANGELOG.md and the documentation, if they impact our users.
• [x] DEV CHANGES.
  • Update the documentation if this PR changes how to develop/launch on the app.
  • Update the README files and our wiki for any big design decisions, if relevant.
  • Add unit tests, docstrings, typing and comments for complex sections.

Resolve #265

New news

No longer a draft PR; should run acceptably on French. Language-switching from config, updated behavioral tests, and multilingual FAISS encoder now included. Also more consistent reference to language-specific defaults in docs.

Old news (for reference):

Draft PR because:

• Goal was to get it working on a branch / not meant to be merged.
• More work required or preferred:
  • Language-switching from config (not hard-coded French, as it is now) = required
  • Changes for behavioral tests
  • Changes for FAISS encoder

Description:

New news

This code makes Azimuth work on French data/pipelines, and the language can be selected in the config.

Main changes:

• Part of Speech tagging: French defaults for spaCy model and dependency tags (to detect subjects and objects)
• Behavioral tests: French neutral tokens; fixed removal of space before punctuation (also submitted nlpaug issue)
• Config: Dynamic selection of language-specific defaults in the config

Out of scope things that could improve French Azimuth:

• Multilingual FAISS encoder (English-trained option works acceptably but not optimally)
• Multilingual tokenizer (Only used for comparing average token count per utterance between train and eval, so should have minimal impact on any conclusions)

Old news (for reference)

Addressed one aspect of Azimuth running for French models: spaCy part-of-speech tagging. This branch loads a French model (instead of English) and the missing_xyz smart tag POS/dependency tag lists have been updated to include French tags.

Several issues came up:

1. Single quote issue: Dataset included text with a single quote instead of an apostrophe, which broke POS-tagging (in particular, j' was not tagged as a subject) and search from the control panel. I added a new function to replace single quotes with apostrophes and used it both for searching and filtering. It worked reasonably well; out of 370 instances of j' in the eval set, 326 were tagged missing_verb before the fix, and 22 afterward (most of which started with "j'aimerais").
2. So then there was a problem with "j'" in "j'aimerais" being tagged as a subject (sometimes). This was solved with the md spaCy model (instead of sm).
3. SpaCy sometimes failed to identify verbs when they were the first word (common, especially with the imperative tense, e.g. "aidez-moi...") and/or capitalized (i.e. even in the middle of the sentence, which is not typical). Changing to the md model was a substantial improvement and fixed the majority of the issue. The exceptions seem to all be cases when the verb is capitalized (but not all capitalized verbs have problems; most don't). The capitalization issue seems to be fixed with the lg model...but it's like 500 MB...so I'm not sure if that's worth it (and haven't done it yet).

Checklist:

You should check all boxes before the PR is ready. If a box does not apply, check it to acknowledge it.

• [x] ISSUE NUMBER. You linked the issue number (Ex: Resolve #XXX).
• [x] PRE-COMMIT. You ran pre-commit on all commits, or else, you ran pre-commit run --all-files at the end.
• [x] FRONTEND TYPES. Regenerate the front-ent types if you played with types and routes. Run cd webapp && yarn types while the back-end is running.
• [x] USER CHANGES. The changes are added to CHANGELOG.md and the documentation, if they impact our users.
• [x] DEV CHANGES.
  • Update the documentation if this PR changes how to develop/launch on the app.
  • Update the README files and our wiki for any big design decisions, if relevant.
  • Add unit tests, docstrings, typing and comments for complex sections.

Resolve #

Description:

Checklist:

You should check all boxes before the PR is ready. If a box does not apply, check it to acknowledge it.

• [x] ISSUE NUMBER. You linked the issue number (Ex: Resolve #XXX).
• [x] PRE-COMMIT. You ran pre-commit on all commits, or else, you ran pre-commit run --all-files at the end.
• [x] USER CHANGES. The changes are added to CHANGELOG.md and the documentation, if they impact our users.
• [x] DEV CHANGES.
  • Update the documentation if this PR changes how to develop/launch on the app.
  • Update the README files and our wiki for any big design decisions, if relevant.
  • Add unit tests, docstrings, typing and comments for complex sections.

Resolve #

Description:

Checklist:

You should check all boxes before the PR is ready. If a box does not apply, check it to acknowledge it.

• [x] ISSUE NUMBER. You linked the issue number (Ex: Resolve #XXX).
• [x] PRE-COMMIT. You ran pre-commit on all commits, or else, you ran pre-commit run --all-files at the end.
• [x] USER CHANGES. The changes are added to CHANGELOG.md and the documentation, if they impact our users.
• [x] DEV CHANGES.
  • Update the documentation if this PR changes how to develop/launch on the app.
  • Update the README files and our wiki for any big design decisions, if relevant.
  • Add unit tests, docstrings, typing and comments for complex sections.

Resolve #344

Description:

The tests should pass after rebasing on #350.

Checklist:

You should check all boxes before the PR is ready. If a box does not apply, check it to acknowledge it.

• [x] ISSUE NUMBER. You linked the issue number (Ex: Resolve #XXX).
• [x] PRE-COMMIT. You ran pre-commit on all commits, or else, you ran pre-commit run --all-files at the end.
• [x] USER CHANGES. The changes are added to CHANGELOG.md and the documentation, if they impact our users.
• [x] DEV CHANGES.
  • Update the documentation if this PR changes how to develop/launch on the app.
  • Update the README files and our wiki for any big design decisions, if relevant.
  • Add unit tests, docstrings, typing and comments for complex sections.

Description:

• I fixed a few issues with config scopes and expirable modules. Modules don't need AzimuthConfig as a scope if they are affected by smart tags, as long as they are ExpirableModules. Since FilterableModules are already ExpirableModules, they were all ok except MetricsbyFilter.
• I also separated the metrics from the ModelContractConfig, to avoid recomputing all predictions if we only change the metrics.

Checklist:

You should check all boxes before the PR is ready. If a box does not apply, check it to acknowledge it.

• [x] ISSUE NUMBER. You linked the issue number (Ex: Resolve #XXX).
• [x] PRE-COMMIT. You ran pre-commit on all commits, or else, you ran pre-commit run --all-files at the end.
• [x] USER CHANGES. The changes are added to CHANGELOG.md and the documentation, if they impact our users.
• [x] DEV CHANGES.
  • Update the documentation if this PR changes how to develop/launch on the app.
  • Update the README files and our wiki for any big design decisions, if relevant.
  • Add unit tests, docstrings, typing and comments for complex sections.

Resolve #348

Description:

Exclude desired fields from cache hash by setting an extra parameter exclude_from_cache.

Checklist:

You should check all boxes before the PR is ready. If a box does not apply, check it to acknowledge it.

• [x] ISSUE NUMBER. You linked the issue number (Ex: Resolve #XXX).
• [x] PRE-COMMIT. You ran pre-commit on all commits, or else, you ran pre-commit run --all-files at the end.
• [x] USER CHANGES. The changes are added to CHANGELOG.md and the documentation, if they impact our users.
• [x] DEV CHANGES.
  • Update the documentation if this PR changes how to develop/launch on the app.
  • Update the README files and our wiki for any big design decisions, if relevant.
  • Add unit tests, docstrings, typing and comments for complex sections.

Resolve #345

Description:

• I took the opportunity to do some refactoring so we can skip the validation tasks and start-up tasks in some cases. For now, this is only useful in the update_config route, in the except condition.
• This refactoring triggered some new mypy warnings that I have fixed.

Checklist:

You should check all boxes before the PR is ready. If a box does not apply, check it to acknowledge it.

• [x] ISSUE NUMBER. You linked the issue number (Ex: Resolve #XXX).
• [x] PRE-COMMIT. You ran pre-commit on all commits, or else, you ran pre-commit run --all-files at the end.
• [x] USER CHANGES. The changes are added to CHANGELOG.md and the documentation, if they impact our users.
• [x] DEV CHANGES.
  • Update the documentation if this PR changes how to develop/launch on the app.
  • Update the README files and our wiki for any big design decisions, if relevant.
  • Add unit tests, docstrings, typing and comments for complex sections.