Hi, I'm trying to configure SmartSim for an Intel+Nvidia Volta node on our local cluster. I was able to get the conda environment setup and successfully executed 'pip install smartsim'. However, when I tried the next step 'smart --device gpu', I get this error:

(SmartSim-cime) [[email protected] SmartSim]$ smart --device gpu

Backends Requested

PyTorch: True
TensorFlow: True
ONNX: False

Running SmartSim build process... Traceback (most recent call last): File "/home/rta/.conda/envs/SmartSim-cime/bin/smart", line 424, in cli() File "/home/rta/.conda/envs/SmartSim-cime/bin/smart", line 421, in cli builder.run_build(args.device, pt, tf, onnx) File "/home/rta/.conda/envs/SmartSim-cime/bin/smart", line 89, in run_build self.install_torch(device=device) File "/home/rta/.conda/envs/SmartSim-cime/bin/smart", line 138, in install_torch if not self.check_installed("torch", self.torch_version): File "/home/rta/.conda/envs/SmartSim-cime/bin/smart", line 121, in check_installed installed_major, installed_minor, _ = installed_version.split(".") ValueError: too many values to unpack (expected 3)

This PR adds running the coverage tests to the GitHub Actions CI. In addition to running the coverage tests, the resulting coverage report is uploaded to codecov, and a PR bot will report any coverage changes that a given PR causes. We also now have a codecov badge for our README.md.

This PR fixes #162. There are four cases that this PR accounts for which have been confirmed to work on local machine and Horizon.

1. User attempts to run smart build when RAI_PATH is set and backends are installed
   • Inform user that backends are already built, tell them where they are built, and notify the user that there is no reason to build.
2. User attempts to run smart build when RAI_PATH is set, but backends are not installed
   • Inform user that before running smart build, they should unset RAI_PATH
3. User attempts to run smart build when RAI_PATH is not set and backends are installed
   • Inform user that they need to run smart clean before running smart build
4. User attempts to run smart build when RAI_PATH is not set and backends are not installed
   • Allow smart build to continue

Description

When in batch mode SmartSim creates the batch script then executed on target machine. Each generated srun execution in the script overwrites PATH, LD_LIBRARY_PATH and PYTHONPATH using export parameter of srun. This can be inconvenient if added preamble (using add_premable function) wishes to modify one of those since the result will never propagate to the target machine.

Justification

Some use-cases require sourcing scripts before using specialised hardware. Those scripts often tend to modify above environmental variables. Generally user should be able to change those in preamble.

Implementation Strategy

PATH, LD_LIBRARY_PATH and PYTHONPATH should not be changed by using export parameter of srun but rather in one of two following ways:

1. Adding export option to generated SBATCH script : #SBATCH --export and removing export from srun parameters of generated script. Srun will propagate those and the user will have a chance to add additional paths.
2. Simply modifying those variables as a part of sbatch script by adding export PATH=$PATH:smartsim_paths.... to the script after preamble.

This should not change the way SmartSim works while giving the user more flexibility on preamble scripts.

Description

Add a tutorial for using SmartSim on Slurm based systems.

Justification

Currently, we only have two tutorials and neither cover Slurm functionality

Implementation Strategy

The following items should be covered in a SmartSim-Slurm tutorial

• [x] Launching jobs in a previously obtained interactive slurm allocation (using SrunSettings)
• [x] Launching batch jobs with SbatchSettings in ensembles
• [x] Getting and releasing allocations through the slurm interface
• [x] Launching on allocations obtained through the smartsim.slurm interface
• [x] Creating and launching the SlurmOrchestrator on interactive allocations and as a batch

The tutorial should be implemented as a Jupyter Notebook and use nbsphinx to host as a part of the documentation.

This is a fix for the codecov badge in the README displaying as 'unknown' rather than displaying the repository's code coverage percentage. This occurs because the coverage report is never uploaded on the develop branch. Instead, the coverage report is ran on the compare branch of a PR. This is a problem because the codecov badge is linked to the develop branch. Since no coverage report is currently being uploaded from the develop branch, the badge is not able to display the repository's code coverage percentage.

The solution introduced in this PR is to run the coverage tests & upload to codecov on pushes to the develop branch, in addition to the already existing event trigger for all PRs.

Fix build_docs github action that was previously broken. Some other minor improvements were added as part of this fix.

Changes include:

• build_docs action is triggered when we push to develop branch
  • Previously, it was on PR closed, which isn't necessarily a merge, i.e. a contributor could close a PR without merging and the action would still trigger
• Use GITHUB_TOKEN instead of personal authentication (PA) credentials to push
• Use "[email protected]" as the git author (user) to distinguish automated commits from human commits
• Check static files from doc branch into develop, so that we can modify them in a regular PR to develop, rather than have to edit the doc branch directly. I'd like for us to get to a point where doc branch pushes are fully automated, so we should never have to modify it directly.
  • Static files include: .nojekyll (both top-level and per-sphinx-build), CNAME, and top-level index.html

Created a parameter that is used for the port argument when instantiating an Orchestrator in the test suite.

Originally, the default port parameter was a hardcoded port number. However, if two local tests run on the same system, local Orchestrators will all be launched on the same port which will result in test failure.

I added the method get_test_port() to the WLMUtils class in conftest.py and replaced each original port argument with the new parameter get_test_port(). Now, local Orchestrators will all be launched on different ports rather than the same.

This PR adds the ability to use KeyDB in place of Redis as the database through a new --keydb flag for smart build. This flexibility enables users to maximize their throughput if they desire.

Adds new RunSettings methods to grow the list of interface functions that are available across all RunSettings objects where possible. Methods added for similar arguments that are shared by two or more of the existing RunSettings sub-classes.

Do let me know if I am missing any additional methods for arguments that should be included, or implementations for existing RunSettings.

After the last update of Summit, many internal mechanisms of SmartSim stopped working. Here is a list of the issues and what I did to mitigate them:

• ERF files used for mpmd now don't accept simple rank_count, but need rank id. Now the LSFOrchestrator will run each shard through the rank with the same id (rank 0 will run shard 0 and so on). It is the same as before, but we specify it.
• ERF files now don't accept more than one app on the same host. I suspect this is a bug, but this just means we cannot run more than one shard per host. This did not result in any change, but limits our features.
• Environment variables are read the wrong way. Specifying more than one env var resulted in wrong handling (everything was assigned to first var). We now store the formatted env vars as a list of strings which is then parsed correctly.
• Killing a jsrun process does not kill its spawned processes anymore. This caused most of the problems, as we were relying on it to stop applications. I turned JsrunSteps into managed ones. This means we use jslist to get the status of a jsrun call inside an allocation. It works, but if anything but SmartSim launches a jsrun command, the matching step id could be lost due to a race condition (ids are assigned incrementally, starting from 0, we mock the Slurm format of alloc_id.step_id internally, to distinguish them from batch jobs). Users will need to only use SmartSim within an allocation.

Added an explicit call to set the number of tasks for the PALS mpiexec settings. This now sets the number of tasks with "--np", whereas before it was being set with "--n", which was giving and error on Polaris.

Models can now connect to a co-located database using a Unix Domain Socket (UDS). In synthetic benchmarks, this can lead to significant improvements over using the loopback interface over TCP/IP. To ensure compatibility with previous scripts we now provide the following interfaces:

• colocate_db (original): Will throw a DeprecationWarning but otherwise just wraps colocate_db_tcp
• colocate_db_tcp: Listens for connections over the loopback
• colocate_db_uds: Listens for connections over UDS

Allows users to launch individual models through an experiment as a batch job with batch settings.

exp = Experiment(...)
rs = exp.craete_run_settings(...)
bs = exp.create_batch_settings(...)
model = exp.create_model("MyModel", rs, batch_settings=bs)
exp.start(model)  # launches the model as a batch job

If a model with batch settings is added to a higher order entity (e.g. an ensemble), the batch settings of the higher order entity will be used and the batch settings of the model will be ignored to avoid an "sbatch-in-sbatch" scenario or similar .

Description

Unix domain sockets allow for more direct communication for entities that exist on the same device. This feature was implemented for the SmartRedis client, but support to configure the orchestrator and clients via SmartSim still needs to be added.

Justification

Users of the co-located deployment of the orchestrator will benefit from increased performance. Initial results show that the latency is significantly reduced.

Implementation Strategy

• [ ] Modify the colocated orchestrator method of Model to include an option for UDS
• [ ] Ensure that the SSDB variable set during experiment launch points to the UDS address of the orchestrator

Description

The documentation currently is fairly prescriptive with regard to GCC and Python versions, stating that SmartSim has build issues with GCC>=10 and only supports up to Python. The former this has to do with error messages during compilation of RedisAI or one of its dependencies. However anecdotally @ashao has compiled with GCC 11 and 12 without apparent issue. This should be double checked and the documentation updated to specify which versions of GCC are likely to work out of the box. For Python, the team recalls reasons why 3.10 would not work, but this should be revisted and documented

Justification

Users who use one newer versions of GCC will be less likely to avoid using SmartSim due to apparent compatibility issues.

Implementation Strategy

• [ ] RedisAI 1.2.3, 1.2.5, 1.2.7: Test SmartSim builds with GPU on horizon which has GCC versions 10.3.0, 11.2.0, and 12.2.0 and on Python 3.10, 3.11
• [ ] Evaluate whether there is a quick workaround for errors with GCC 10
• [ ] Update documentation

Currently users do not have access to logging capabilities in SmartRedis. This means that users do not have a detailed understanding of client activity in order to debug complex workflows. Completion of this epic will enable logging in all four clients with varying levels of verbosity.