summary

• maintenance release, no new features
• bug fix for pypyr.steps.filereplace to honor flat format :ff directive
• python 3.11 compatibility confirmed
  • on python 3.11 runtimes, the tomli dependency won't install separately anymore since it's part of stdlib now

what's changed

• pypyr.steps.filereplace get_formatted X2 by @yaythomas in https://github.com/pypyr/pypyr/pull/304
• new codecov uploader by @yaythomas in https://github.com/pypyr/pypyr/pull/305
• py 3.11 & conditional tomli dep by @yaythomas in https://github.com/pypyr/pypyr/pull/306
• Bump version: 5.7.0 → 5.7.1 by @yaythomas in https://github.com/pypyr/pypyr/pull/307

Full Changelog: https://github.com/pypyr/pypyr/compare/v5.7.0...v5.7.1

summary

• New switch step for IF-ELSE style branching in your pipelines!
• New argskwargs parser to combine plain args and key-value pairs (key=value) from the cli
• keyvaluepairs and dict parsers now support having = in the value, so you can have key=one+one=two parse to {'key': 'one+one=two'}
  • these parsers now also support passing args with no =, in which case arg1 becomes {'arg1': ''}

what's changed

• switch by @yaythomas in https://github.com/pypyr/pypyr/pull/299
• typing on dsl.cof by @yaythomas in https://github.com/pypyr/pypyr/pull/300
• gh action checkout v3 by @yaythomas in https://github.com/pypyr/pypyr/pull/301
• argskwargs parser & partitioning on = for kvp by @yaythomas in https://github.com/pypyr/pypyr/pull/302

Full Changelog: https://github.com/pypyr/pypyr/compare/v5.6.0...v5.7.0

summary

• you can now use a custom error message when pypyr.steps.assert raises an exception.
• new pypyr.steps.venv step to create venvs in parallel from yaml or toml config.
• new venv-create pipeline so you can provision venvs concurrently from declarative config without having to write your own pipeline or script.
• pypyr is now built & packaged by the excellent PEP517 compliant flit.

what's changed

• flake8 new version line length by @yaythomas in https://github.com/pypyr/pypyr/pull/286
• from setup.py to flit by @yaythomas in https://github.com/pypyr/pypyr/pull/290
• remove redundant dict on parser context init by @yaythomas in https://github.com/pypyr/pypyr/pull/292
• venv creator & log to stdout by @yaythomas in https://github.com/pypyr/pypyr/pull/296
• ops/build uses venv for publish verify by @yaythomas in https://github.com/pypyr/pypyr/pull/297

Full Changelog: https://github.com/pypyr/pypyr/compare/v5.5.0...v5.6.0

summary

• cmdOut for pypyr.steps.cmd & pypyr.steps.shell gets dotted attribute access.
  • instead of cmdOut[returncode] access value like this: cmdOut.returncode
  • backwards compatible - the old-style dict-like accessors will still work.
  • closes #272
• Introduce pypyr.steps.cmds & pypyr.steps.shells to run programs or shell statements asynchronously as parallel (concurrent) sub-processes.
  • cmdOut for these works as with the serial versions of the steps using the new dotted attribute access introduced in this release.
  • closes #273
• Set default encoding for all the cmd/shell steps using default_cmd_encoding in config. You can use the $env variable PYPYR_CMD_ENCODING to initialize this value, in addition to the usual config files.
  • closes #275

what's changed

• async cmds + shells by @yaythomas in https://github.com/pypyr/pypyr/pull/274
• release 5.5 & default_cmd_encoding for serial by @yaythomas in https://github.com/pypyr/pypyr/pull/276

Full Changelog: https://github.com/pypyr/pypyr/compare/v5.4.0...v5.5.0

Summary

• pypyr.steps.cmd & pypyr.steps.shell now also takes a list input to run multiple commands/shell statements in the same step!

- name: pypyr.steps.cmd
  in:
    cmd:
      - echo 1
      - echo 2

- name: pypyr.steps.cmd
  in:
    cmd:
      run:
        - echo 3
        - echo 4
      save: False
      cwd: mydir/subdir

• Both the cmd and shell steps expanded to allow:
  • decode output in different encodings
  • save output as raw bytes or as encoded text.
  • File output for stdout/stderr.
    • File create mode of append or overwrite.
  • stderr can redirect to /dev/stdout
  • Both stdout & stderr can redirect to /dev/null

#  when save: True
- name: pypyr.steps.cmd
  comment: when save is True
  in:
    cmd:
      run: curl https://myurl.blah/diblah
      save: True
      cwd: .
      bytes: False
      encoding: utf-8

#  when save: False (this is default)
- name: pypyr.steps.cmd
  comment: when save is False (the default when `save` not set)
  in:
    cmd:
      run: curl --cert certfile --key keyfile https://myurl.blah/diblah
      cwd: ..
      stdout: ./path/out.txt
      stderr: ./path/err.txt
      append: False

What's Changed

• cmd & shell list input by @yaythomas in https://github.com/pypyr/pypyr/pull/271

Full Changelog: https://github.com/pypyr/pypyr/compare/v5.3.0...v5.4.0

Summary

Create shortcuts to your pypyr run commands & their input args. This is handy for creating short & sweet aliases for longer pipeline run commands.

So if you have a pipeline you normally run like this:

$ pypyr arb/my-pipeline arg1=1234 arg2="/path/to long/annoying path to type/x" arg3="arb'hello"

You can create a shortcut alias for this like so:

shortcuts:
  my-shortcut:
    pipeline: arb/my-pipeline
    args:
      arg1: 1234
      arg2: /path/to long/annoying path to type/x
      arg3: "arb'hello"

And now for the same pipeline + inputs you can just run:

$ pypyr my-shortcut

You can create your shortcut definition in pyproject.toml or in the pypyr yaml config file.

You can make your shortcuts project specific, or for the entire user profile, or globally for your entire system.

What's Changed

• Shortcuts by @yaythomas in https://github.com/pypyr/pypyr/pull/268
• Bump version: 5.2.0 → 5.3.0 by @yaythomas in https://github.com/pypyr/pypyr/pull/269

Full Changelog: https://github.com/pypyr/pypyr/compare/v5.2.0...v5.3.0

Summary

• pypyr is now configurable with yaml or pyproject.toml!
• Explicitly over-ride encoding on any filesystem steps

What's Changed

• friendly err on malformed pipeline. by @yaythomas in https://github.com/pypyr/pypyr/pull/250
• config & encoding by @yaythomas in https://github.com/pypyr/pypyr/pull/255
• file encoding from config by @yaythomas in https://github.com/pypyr/pypyr/pull/259
• windows tests & config platform indicators by @yaythomas in https://github.com/pypyr/pypyr/pull/261
• typings for config, platform by @yaythomas in https://github.com/pypyr/pypyr/pull/262
• FakePath PathLike & parent also fake by @yaythomas in https://github.com/pypyr/pypyr/pull/264
• pypyr.steps.python & version bump 5.2.0 by @yaythomas in https://github.com/pypyr/pypyr/pull/266

Full Changelog: https://github.com/pypyr/pypyr/compare/v5.1.0...v5.2.0

Finally, toml, has come, to pypyr! 🎆

• pypyr.parser.tomlfile to initialise pipeline context with a toml file
• pypyr.steps.fetchtoml to read toml file into context as an object with typing
• pypyr.steps.filewritetoml to write specified context to output toml file - lets you create toml files on-the-fly programmatically.
• pypyr.steps.fileformattoml to read input tomls specified by path, glob or list of paths, replace {formatting expressions} and write to output file(s).

Working with files:

• pypyr.steps.fileread to read files in text or binary mode into context.
• pypyr.steps.filewrite to write or append to file in text or binary modes.

What's Changed

• pypyr.steps.fileread by @yaythomas in https://github.com/pypyr/pypyr/pull/244
• pypyr.steps.filewrite by @yaythomas in https://github.com/pypyr/pypyr/pull/245
• toml by @yaythomas in https://github.com/pypyr/pypyr/pull/246

Full Changelog: https://github.com/pypyr/pypyr/compare/v5.0.0...v5.1.0

Implement adr2 relative pipelines + api changes.

In brief, this release lets pipelines reference custom modules & child pipelines relative to the pipeline itself, rather than the current directory. This lets you create portable, re-usable & composable pipeline libraries.

Breaking Changes

This is a major version increment because it comes with BREAKING CHANGES:

1. API: pipelinerunner.run() replaces both pipelinerunner.main() and pipelinerunner.main_with_context()
2. API: def get_pipeline_definition(pipeline_name, working_directory) signature for custom pype loaders changes to def get_pipeline_definition(pipeline_name, parent)
3. CLI: the —dir flag now only sets the directory for ad hoc custom Python modules, it does NOT also set the directory for pipelines anymore
4. Final removal of deprecated get_formatted_iterable, get_formatted_string #195 & pypyr.steps.contextset #184. Where previously these would just give deprecation warnings, they are now completely removed.
5. pypyr.pypeloaders.fileloader renamed pypyr.loaders.file

Non-Breaking Changes

• You can now access the current pipeline’s metadata & loader information from within a pipeline with context.current_pipeline
• Improve handling of absolute paths in file loader only to search path once, rather than unnecessarily go through the same relative path lookup sequence with the same path.
• Typing support added for the pypyr API entrypoint.

Detailed Technical Breakdown:

• Introduce new classes to model pipeline payload, rather than just using the bare dict-like yaml directly.
  • PipelineInfo - pipeline metadata set by loader. This maintains a pipeline’s parent/path info so that child pipelines can load relative to the parent.
  • PipelineDefinition - this wraps the pipeline payload and its metadata (PipelineInfo) to allow pypyr to cache it all with one reference
• Add new Pipeline class for the run-time properties of a single run.
  • The Pipeline references the shared cached PipelineDefinition.
  • Move run + load_and_run logic from pypyr.pipelinerunner to the new Pipeline class. This massively streamlines the pipeline invocation process, since run/load_and_run can just operate on the shared Pipeline state rather than sling a bunch of args between different functions as before.
• Add a call-stack of running Pipeline instances on Context. i.e Parent -> child1 -> child2 where the root pipeline calls child pipelines via pype
  • Add current_pipeline attribute to Context, controlled with a context manager to scope itself to an individual pipeline run’s lifespan.
    • This means that steps can access current pipeline’s properties.
    • This allows pypyr.steps.pype to find the current (i.e parent) pipeline’s metadata such as path, to load child pipeline relative to the calling pipeline’s location.
    • When a child pipeline completes, the calling pipeline (i.e the previous Pipeline in the call-stack) becomes the current pipeline
• Amend pypyr.steps.pype to instantiate Pipeline object to load_and_run() child.
  • pype now deals the context.current_pipeline.pipeline_definition.info metadata to work out whether to cascade parent path down to child, so child can load relative to the parent.
  • Notably, the parent loader now cascades to the child, so pipeline authors don’t need explicitly to set the same custom loader repeatedly for each child.
  • Given the context manager controlling current pipeline scope in Pipeline.load_and_run_pipeline remove the clumsy side-shuffle for pipeline_name, working_dir to swap out these values as child pipe runs and swap these back when it completes/errors.
• Remove global PipelineCache. Replace with distinct pipeline cache per loader.
  • This resolves a long standing limitation where pypyr assumed unique pipeline names across all loaders.
  • The per-loader pipeline cache stores PipelineDefinition objects.
  • Introduce Loader class, which wraps loader & its pipeline cache. Loader is what the LoaderCache caches.
  • Thus LoaderCache -> Loader -> _pipelineCache -> PipelineDefinition
• File loader has a private file cache keyed on absolute path of file
  • This is to prevent >1 load+parse where the same underlying pipeline.yaml file has different names in the loader’s pipeline cache
    • e.g (name=‘dir/mypipe’, parent=None) and (name=‘mypipe’, parent=‘dir’) both resolve to dir/mypipe.yaml
  • Caching a reference to the PipelineDefinition object, so not duplicating memory
• Improve handling of absolute paths in file loader only to search path X1, rather than unecessarilly go through the same relative path lookup sequence with the same path.
• File loader get_pipeline_definition now returns a PipelineDefinition with PipelineFileInfo to store file-system specific metadata for the loader pipeline
• Remove working_dir global. The py_dir input on run() now refers ONLY to module paths, NOT pipeline locations.
  • The CLI —dir flag, or py_dir input on run() basically adds the specified directory to sys.path.
• Add current pipeline’s parent directory to sys.path on load. This allows child pipelines to resolve custom modules relative to itself.
• pypyr.dsl.Step does not need StepsRunner anymore, because it can get it from the context.current_pipeline instead.
• Recode (some) integration tests to take advantage of list pypyr.steps.append step and checking that for output on return context rather than intercepting logger.NOTIFY.
• Rename master branch to main in CI/CD GitHub actions
• Add typing annotations to the public run() function and the Pipeline class public accessors. The idea is NOT to type pypyr exhaustively, just to provide annotations for the sensible/likely entrypoint to enhance API user experience. Include py.typed in pypyr package.
• Remedy packaging snafu where tests.common was deploying alongside pypyr because exclude condition in find_packages didn't include wildcard for sub packages.

What's Changed

• Relative pipelines & API run() replaces main/main_with_context by @yaythomas in https://github.com/pypyr/pypyr/pull/243

Full Changelog: https://github.com/pypyr/pypyr/compare/v4.6.0...v5.0.0

What's Changed

• Fix failing ops/build (failing on linting issues and locale dependent failure) by @vlcinsky in https://github.com/pypyr/pypyr/pull/226
• Evaluate step skip only if the run check returned true by @Reskov in https://github.com/pypyr/pypyr/pull/234
• pypyr.steps.append & pypyr.steps.add by @yaythomas in https://github.com/pypyr/pypyr/pull/235
• shorter alias set for contextsetf. py 3.10. by @yaythomas in https://github.com/pypyr/pypyr/pull/238

New Contributors

• @vlcinsky made their first contribution in https://github.com/pypyr/pypyr/pull/226

Full Changelog: https://github.com/pypyr/pypyr/compare/v4.5.0...v4.6.0

• add retry backoff strategies. #216.
• prevent duplicates in sys.path (#218) on main* api entry points. thread-safe existence check on sys.path list.
• prevent redundant multiple addition of notify log-level on main* api entry. Move initialization code to package __init__ (#219).

• Context parsers that create an entry in context now initialise to empty rather than None. This means you can directly use something like {argList}, {argDict} and argString (initialising respectively to [], {}, '') directly for things like foreach loops without having to worry about None checks. Your existing truthy checks for these values will work as before. closes #214.

• New step pypyr.steps.pyimport to import references to the py-string namespace.
  • This includes an underlying api signature change by removal of pypyr.utils.expressions.eval_string(), but this is sufficiently far down the call-chain that it shouldn’t affect any normal pipeline operator or api consumer.
  • See #110 for details.
• pypyr.steps.contextclearall wipes pyimport imported references in addition to the key/values inside context.
• Simplify pypyr.steps.py syntax by allowing a new py (rather than pycode) input. This allows pipeline authors to use context key names directly as variables, rather than have to specify them as keys in context (my_var vs context[‘my_var’]).
  • see #204 for details on simplifying the pypyr.steps.py syntax.
  • the old pycode will keep on working in the same way, so no need to worry about backwards compatibility for your existing pipelines.
• Allow substitutions on Retry max. Resolves #207. Excellent bug find & fix by @Reskov, much thanks 🙌 🙌 🙌 as ever for a superb contribution! 🔥 🔥 🔥
• foreach can now use any iterable, including generators. Closes #209

• Streamline main entrypoint API. close #201.
  • main() allows consumer to set pype loader, rather than having to drop further down into api to load_and_run_pipeline()
  • new main_with_context() to input dict to initialise context, and bypass context_parser entirely. Also returns the Context object after pipeline run completes.
  • make all non-essential args optional to allow minimal calls to main entrypoint without having to add optional=None style inputs.
  • This is fully backwards compatible.
• pypyr.steps.pype
  • defaults useParentContext to False is pipeArgs specified.
  • pipeArgs shlex-es input string
  • set pipeline_name on child pipeline rather than use parent pipeline name
• working_dir uses Path object rather than string
• pypyr.steps.echo remove redundant string check.

• python 3.9 compatibility officially confirmed. All good. Woo!
• New Flat ff and Recursive rf formatting specifiers. #195
  • Maintain backwards compatibility for all current formatting functionality.
  • The get_formatted_string, get_formatted_iterable & get_processed_string methods on Context() are now all deprecated. Use get_formatted_value instead. All of the deprecated functions will keep on working as before, but will print a WARN to the output.
  • get_formatted and get_formatted_value remain the official best ways of getting formatted values.
• New !jsonify yaml special tag directive. This converts context object to a json string. #197.
• New pypyr.steps.jsonparse step. Parse and deserialise a json string into Context. #199
• Bare pypyr.steps.assert syntax. #196
• Add new utils.asserts for extra flexibility with nested key validation.
• PyStrings optimised. Not so that you'd notice, but still, it'll be a smidge faster.
• Friendlier error messages when yaml/json context parsers reject documents that do NOT have a mapping at root.

• Better description output - add (skipping) to output if the step is not running because run is False or skip is True. Ref #158.
• First release published from shiny new GitHub Action CI/CD!
• Add License to wheel published to pypi

• BREAKING CHANGE: final deprecation of in args persisting after step execution. For full discussion please see #177. Preview functionality where setting environment variable $PYPYR_IN_CLEAN = 1 is now the default. You can remove the $PYPYR_IN_CLEAN variable from this release onwards. Henceforth pypyr removes in args from context after step completes. If you want to persist values in context, use pypyr.steps.contextsetf, pypyr.steps.contextcopy or pypyr.steps.default.
• BREAKING CHANGE: pypyr.steps.assert raises an AssertionError rather than a ContextError. Ref #188.
• The cli now defaults to a simplified log format that is way less noisy. If you still want the full, old style logs, run pypyr with --log 25 set explicitly. pypyr my pipe --log 25. Ref #187
  • description decorator output also simplified to echo only your actual custom text.
• Failure Groups now support Stop instructions. This allows you to exit the failure handler reporting success, so pypyr can report success if you consider the error condition handled. Ref #175.
• pypyr.steps.contextset renamed pypyr.steps.contextcopy. The old name will keep on working, but with a deprecation warning. Ref #184.
• internal: improvements to integration testing error reporting.

• Line + Column Number count from 1 not 0. fix #151
• update README to point at new website pypyr.io.
• update CONTRIBUTING to point at new website.
• update documentation and pypi setup with new repo name.

• Allow non-string keys on context.get_formatted_iterable and context merge (including the step pypyr.steps.contextmerge). Resolves #179. Much thanks to @Reskov for a detailed & helpful bug report!

ATTENTION: Please start preparing all your pipelines to be ready for the next major release where in context arguments will be removed from context on step completion. This shouldn't be too much of an issue. For any steps where you need to set enduring context, use contextsetf or contextset.

• Allow using a call control-of-flow instruction nested inside while/for/retry loops. Previously a nested call instruction would over-write the parent call's configuration and thus create confusion on the next loop round the parent. Much thanks 🙏 to @Reskov for identifying this tricky little recursion issue & assistance in resolution. Close #176
• in context parameters only available for the duration of the step they decorate. Previously in items would be added to context and stay in context after the step finishes. This is a potentially breaking change for existing pipelines, but worth it, full discussion of reasons here #177.
  • For this release this change in functionality is not the default, but enabled by setting the following environment variable PYPYR_IN_CLEAN = 1. You can set it selectively on a single pypyr run using standard shell syntax like: $ PYPYR_IN_CLEAN=1 pypyr mypipeline arg1 arg2
  • README updates to document #177 resolution as the default behaviour - in context parameters cleaned from context on step completion to prepare for next major release with the breaking change..

pypyr.log.logger.set_root_logger(log_level, log_path) call moved from pipelinerunner.main() to cli.main()

this has the side-effect that pipelinerunner.main() signature has changed, with log_level and log_path removed. API consumers should update.

Reason being API consumers should set their own log handlers, since handler configuration should be the prerogative of the calling application, not the invoked library.

Re #172, #173

• Retry Decorator correctly handles errors raised from .call or .jump steps when checking error type with retryOn and stopOn. Close #170 - thanks to @Reskov for reporting!
• add tarball to PyPI deploy artefacts.

• When importing a step that does exist (mystep.blah), a failing import mymodule.blah inside mystep.blah would give a misleading error and hide the actual source of the problem. Fixes #166. Eternal thanks to @irancati for raising the issue!

With a million thank yous 🙏 to @Reskov for these much needed fixes/enhancements! 🎆

• Resolves issue #149. pypyr.steps.debug now supports outputting any type, including PyString & SicString
• Resolve pypyr.steps.contextmerge issue to support merging any type, including PyString & SicString.

• BREAKING CHANGE: Remove deprecated step input context for (ref #118)
  • assert
  • env
  • fetchjson
  • fetchyaml
  • fileformat
  • fileformatjson
  • fileformatyaml
  • filereplace
  • tar
• BREAKING CHANGE: cli context input now uses standard cli spacing rather than needing to str close everything. Closes #94
  • So where you had:
  • pypyr mypipeline "key1=value1,key2=value2”
  • Now instead:
  • pypyr mypipeline key1=value1 key2=value2
• BREAKING CHANGE: Big API changes to pipelinerunner, stepsrunner and Step
• BREAKING CHANGE: pypyr.parser.commas replaced by pypyr.parser.keys
• Caching pipelines, context parsers, loaders, step run functions. Basically cache anything that loads from disk. Closes #126
• Custom step groups. Closes #126
• Jump, Call, and Stop ref #128 #127 #123
• The ./pipelines default directory location no longer necessary. To run a pipeline ./dir/mypipe.yaml, just go pypyr dir/mypipe
• README+help updates for all of the above

This one is all @Reskov 🎉 eternal gratitude!

• new NOTIFY logging level added. this level is much less verbose than even INFO, and only displays explicits echos that you specify in your pipeline, and also the description from the step decorator. This should make it much easier to see what your pipeline is doing without the extra detail that can make things confusing and harder to follow. #132
• under-the-hood api improvement for more performant logging #152

With eternal thanks to @Reskov 🎉 for next level super useful improved error handling! 🙇

• #139 - pypyr now saves all run-time errors to context under runErrors. This means subsequent steps can use error information from previous steps.
• #135 - error information now include the pipeline source yaml line+col numbers

• pypyr.steps.glob lets you get a list of paths from a glob, and combine different globs into a combined output result list. Issue #145
• with much thanks to @Reskov 🎉 , resolves issue #143 - merge step now supports special tag directives

• new Retry decorator allows steps to retry automatically when step encounters an error. ref #130
• fix run_step AttributeError might have caught or hidden AttributeErrors in the step code itself, rather than just when the step module didn't contain a run_step function. ref #129
• while loop checks stop condition only at end of each iteration. previously if a stop condition evaluated True even before the loop started the loop wouldn't run at all. now the loop will run once. If you don't want the step to run, use the skip: True decorator instead.
• it's now easier to write infinite While loops with an explicit stop: False condition

• add string interpolation to pypyr.steps.pype input.