Ralph is a command-line tool to fetch, extract, convert and push your tracking logs from various storage backends to your LRS or any other compatible storage or database backend.

Overview

Ralph

Ralph is a command-line tool to fetch, extract, convert and push your tracking logs (aka learning events) from various storage backends to your LRS or any other compatible storage or database backend.

Quick start guide

Ralph is distributed as a standard python package; it can be installed via pip or any other python package manager (e.g Poetry, Pipenv, etc.):

$ pip install ralph-malph

Once installed, the ralph command should be available in your PATH. Try to invoke the program usage thanks to the --help flag:

$ ralph --help

You should see a list of available commands and global flags for ralph. Note that each command has its own usage that can be invoked via:

$ ralph COMMAND --help

You should substitute COMMAND by the target command, e.g. list, to see its usage.

Documentation

We try our best to maintain an up-to-date reference documentation for this project. If you intend to install, test or contribute to ralph, we invite you to read this documentation and give us feedback if some parts are unclear or your use case is not (or poorly) covered.

Contributing

This project is intended to be community-driven, so please, do not hesitate to get in touch if you have any question related to our implementation or design decisions.

We try to raise our code quality standards and expect contributors to follow the recommandations from our handbook.

License

This work is released under the MIT License (see LICENSE).

Comments
  • ✨(models) add edx navigational event pydantic models

    ✨(models) add edx navigational event pydantic models

    Purpose

    With the validate and select command newly implemented, it is now possible to implement edx tracking logs events pydantic models to validate incoming events in ralph.

    Proposal

    The first events implemented are the easiest (navigation).

    • [x] event models
    • [x] event fixtures
    • [x] event tests
    needs review feature 
    opened by quitterie-lcs 9
  • Add WebSocket to backends documentation

    Add WebSocket to backends documentation

    Purpose

    Add WebSocket to Backends documentation

    Proposal

    WebSocket

    The webSocket backend is read-only and can be used to get real-time events.

    If you use OVH's Logs Data Platform (LDP), you can retrieve a WebSocket URI to test your data stream by following instructions from the official documentation.

    Backend parameters

    • ws_uri: the WebSocket uri (e.g. wss://example.com/websocket)
    improvement WIP 
    opened by p-bizouard 6
  • ✨(converters) add edx server event xapi converter

    ✨(converters) add edx server event xapi converter

    Purpose

    The edx server events are the most simple and common events in the edx tracking logs. They are logged at almost all requests made to the edx lms. We want to convert them into xAPI format.

    Proposal

    ServerEventSchema validates the edx server event.

    ServerEventToXapi uses the ServerEventSchema to validate the edx server event and then converts it to xAPI format.

    XapiConverterSelector deserializes the edx events and tries to match a known converter for the event conversion.

    feature 
    opened by SergioSim 6
  • Add helm chart

    Add helm chart

    Purpose

    Add Helm chart for Ralph

    Description

    Created the chart under src/helm I tried to migrate the logic from the .j2 tray files. The version label was replaced as the helm convention for app.kubernetes.io/version The checksum from the secret name was deleted. The annotation checksum/config: was used instead on the secret, the checksum is automatically generated from vault.yaml it help to update the secret object every time a value is updated. Note that the .Chart.AppVersion is better used to track the version deployed and it should reflect the image.tag There is only one vars files values.yaml the vault is under specific file vault.yaml used only for generating the secrets.

    Questions

    • Why do you have two service with the same config? the only difference is the label static
    • What is deployment_stamp ?
    • What is the purpose of prefix ?
    • How would you like to manage env.?

    ToDo

    • [ ] Add README.md

    You can use helm template . under ./src/helm/ralph to check the output of the chart, it will show you the final manifest that will be deployed.

    feature 
    opened by waammar 5
  • Add MongoDB backend

    Add MongoDB backend

    Purpose

    MongoDB is a classical database for LRSs such as LearningLocker or Apereo Learning Analytics Initiative. Supporting this backend seems a must-have for Ralph.

    Proposal

    • [x] add LearningLocker parser
    • [x] add MongoDB backend
    • [x] ~add learning locker parser~ edit: postponed to a new PR
    needs review feature 
    opened by jmaupetit 5
  • ✨(converter) add edx to xapi video converter

    ✨(converter) add edx to xapi video converter

    Purpose

    edX video events can be converted to xAPI for most of them. The interaction events can still not be converted.

    Proposal

    • [x] load_video to initialized
    • [x] play_video to played
    • [x] pause_video to paused
    • [x] stop_video to terminated
    • [x] seek_video to seeked
    improvement needs review FUN 
    opened by quitterie-lcs 4
  • ✨(models) add xAPI page_viewed and page_terminated models

    ✨(models) add xAPI page_viewed and page_terminated models

    Purpose

    These pydantic models can be used for xAPI format validation.

    Proposal

    The page_viewed pydantic model correspond to the edX simple server event. The page_terminated pydantic model correspond to the edX page_close browser event.

    • [X] page_viewed pydantic model
    • [X] page_terminated pydantic model
    • [x] tests
    feature WIP 
    opened by SergioSim 4
  • ✨(models) add edx server event model

    ✨(models) add edx server event model

    Purpose

    The edx server events are the most simple and common events in the edx tracking logs. They are logged at almost all requests made to the edx lms. We want to validate them.

    Proposal

    • [x] ServerEventModel
    • [x] ServerEventFactory
    • [x] Tests
    feature WIP 
    opened by SergioSim 4
  • Add test fixtures

    Add test fixtures

    Purpose

    • A event test fixture factory could help to test the function of the GELFParser

    Proposal

    • [ ] Implementation of the event test fixture factory
    feature WIP 
    opened by SergioSim 4
  • Add

    Add "event" test fixture factory

    Purpose

    • Document the different events that are logged as "tracking logs" in the LMS to understand how, were and when things get logged
    • A event test fixture factory could help to test the function of the GELFParser

    Proposal

    • [ ] Implementation of the event test fixture factory
    • [ ] Events documentation
    feature WIP 
    opened by SergioSim 4
  • ✨(cli) add the new

    ✨(cli) add the new "auth" command

    Purpose

    Generating credentials to authenticate to our LRS should be smooth and painless for our users.

    Proposal

    • [x] add a new auth command to ease new user credentials generation
    • [x] add pydantic models for user credentials checking and credentials file validation
    • [x] track authentication issues to the LRS server
    needs review feature 
    opened by jmaupetit 3
  • `make bootstrap` is failing due to missing build dependences

    `make bootstrap` is failing due to missing build dependences

    Bug Report

    Hi all, I'm excited to get started developing with Ralph. Getting set up I ran into an issue, however. On the master branch make bootstrap errors out pip installing psutil due to gcc not being available.

    Expected behavior/code make bootstrap builds the docker images cleanly.

    Actual Behavior An error occurs:

    #0 22.22       creating build/temp.linux-aarch64-cpython-39/psutil
    #0 22.22       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=594 -DPy_LIMITED_API=0x03060000 -DPSUTIL_LINUX=1 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -I/usr/local/include/python3.9 -c psutil/_psutil_common.c -o build/temp.linux-aarch64-cpython-39/psutil/_psutil_common.o
    #0 22.22       C compiler or Python headers are not installed on this system. Try to run:
    #0 22.22       sudo apt-get install gcc python3-dev
    #0 22.22       error: command 'gcc' failed: No such file or directory
    

    Steps to Reproduce

    1. Sync the Ralph master branch
    2. run make bootstrap
    3. The above error occurs

    Environment

    • Ralph version: master branch
    • Platform: Docker Desktop on Mac OS 13.0.1

    Possible Solution I was able to get a successful build by adding the required development tools, see the PR here: https://github.com/bmtcril/ralph/pull/1

    bug 
    opened by bmtcril 1
  • 📝(project) add helm chart tutorial

    📝(project) add helm chart tutorial

    Purpose

    Ralph can be deployed to a Kubernetes cluster using its official Helm Chart. We need a step-by-steph guide to use it.

    Proposal

    This tutorial guides you to the deployment of a complete learning analytics stack to Kubernetes using Helm.

    FUN WIP 
    opened by jmaupetit 1
  • Implement xAPI forum models in `ralph`

    Implement xAPI forum models in `ralph`

    Purpose

    xAPI forum used in ashley models have to be described in ralph for validation purpose and usage of ralph as a library

    Proposal

    • [ ] Define pydantic models associated to each forum learning statement templates
    • [ ] Define selector for each model
    • [ ] Write model and selector tests
    feature 
    opened by quitterie-lcs 0
  • Integrate mypy for Python type checking

    Integrate mypy for Python type checking

    Purpose

    It has been decided that we use typing in all ralph project. We have to integrate a static type checker in our linting toolbox.

    Proposal

    • Integrate mypy in the project (local tooling for development + CI)
    improvement 
    opened by quitterie-lcs 0
  • 🏗️(backends) unify data backends fs

    🏗️(backends) unify data backends fs

    Purpose

    Storage and Database backends have similar interfaces and usage.

    Proposal

    Unify the interfaces of database and storage backends into a single DataBackend interface.

    • [x] add Data and LRS Backend interfaces
    • [x] add FileSystemDataBackend
    needs review feature 
    opened by SergioSim 1
  • Ora2 events

    Ora2 events

    Purpose

    ORA2 events describe the activity for assessments pair reviewing. Edx defined models and their selectors are described with Pydantic.

    Proposal

    Define Pydantic model for:

    • openassessmentblock.create_submission,
    • openassessmentblock.get_peer_submission,
    • openassessmentblock.get_submission_for_staff_grading,
    • openassessmentblock.peer_assess,
    • openassessmentblock.save_submission,
    • openassessmentblock.self_assess,
    • openassessmentblock.staff_assess,
    • openassessmentblock.student_training_assess_example,
    • openassessment.submit_feedback_on_assessments,
    • openassessment.upload_file,
    needs review feature 
    opened by quitterie-lcs 0
Releases(v3.1.0)
  • v3.1.0(Nov 17, 2022)

    Added

    • EdX to xAPI converters for video events

    Changed

    • Improve Ralph's library integration by unpinning dependencies (and prefer ranges)
    • Upgrade fastapi to 0.87.0
    Source code(tar.gz)
    Source code(zip)
  • v3.0.0(Oct 19, 2022)

    Added

    • Implement edx video browser events pydantic models
    • Create a post endpoint for statements implementing the LRS spec
    • Implement support for the MongoDB database backend
    • Implement support for custom queries when using database backends get method (used in the fetch command)
    • Add dotenv configuration file support and python-dotenv dependency
    • Add host and port options for the runserver cli command
    • Add support for database selection when running the Ralph LRS server
    • Implement support for xAPI statement forwarding
    • Add database backends status checking
    • Add health LRS router
    • Tray: add LRS server support

    Changed

    • Migrate to python-legacy handler for mkdocstrings package
    • Upgrade click to 8.1.3
    • Upgrade elasticsearch to 8.3.3
    • Upgrade fastapi to 0.79.1
    • Upgrade ovh to 1.0.0
    • Upgrade pydantic to 1.9.2
    • Upgrade pymongo to 4.2.0
    • Upgrade python-keystoneclient to 5.0.0
    • Upgrade python-swiftclient to 4.0.1
    • Upgrade requests to 2.28.1
    • Upgrade sentry_sdk to 1.9.5
    • Upgrade uvicorn to 0.18.2
    • Upgrade websockets to 10.3
    • Make backends yield results instead of writing to standard streams (BC)
    • Use pydantic settings management instead of global variables in defaults.py
    • Rename backend and parser parameter environment variables (BC)
    • Make project dependencies management more modular for library usage

    Removed

    • Remove YAML configuration file support and pyyaml dependency (BC)

    Fixed

    • Tray: do not create a cronjobs list when no cronjob has been defined
    • Restore history mixin logger
    Source code(tar.gz)
    Source code(zip)
  • v2.1.0(Apr 25, 2022)

    Added

    • Implement edx problem interaction events pydantic models
    • Implement edx textbook interaction events pydantic models
    • ws websocket stream backend (compatible with the fetch command)
    • bundle jq, curl and wget in the fundocker/ralph Docker image
    • Tray: enable ralph app deployment command configuration
    • Add a runserver command with basic auth and a whoami route
    • Create a get endpoint for statements implementing the LRS spec

    Changed

    • Upgrade uvicorn to 0.17.4
    • Upgrade elasticsearch to 7.17.0
    • Upgrade sentry_sdk to 1.5.5
    • Upgrade fastapi to 0.73.0
    • Upgrade pyparsing to 3.0.7
    • Upgrade pydantic to 1.9.0
    • Upgrade python-keystoneclient to 4.4.0
    • Upgrade python-swiftclient to 3.13.0
    • Upgrade pyyaml to 6.0
    • Upgrade requests to 2.27.1
    • Upgrade websockets to 10.1
    Source code(tar.gz)
    Source code(zip)
  • v2.0.1(Jul 15, 2021)

  • v2.0.0(Jul 12, 2021)

    Added

    • xAPI video interacted pydantic models
    • xAPI video terminated pydantic models
    • xAPI video completed pydantic models
    • xAPI video seeked pydantic models
    • xAPI video initialized pydantic models
    • xAPI video paused pydantic models
    • convert command to transform edx events to xAPI format
    • EdX to xAPI converters for page viewed andpage_close events
    • Implement core event format converter
    • xAPI video played pydantic models
    • xAPI page viewed and page terminated pydantic models
    • Implement edx navigational events pydantic models
    • Implement edx enrollment events pydantic models
    • Install security updates in project Docker images
    • Model selector to retrieve associated pydantic model of a given event
    • validate command to lint edx events using pydantic models
    • Support all available bulk operation types for the elasticsearch backend (create, index, update, delete) using the --es-op-type option

    Changed

    • Upgrade elasticsearch to 7.13.2
    • Upgrade python-swiftclient to 3.12.0
    • Upgrade click to 8.0.1
    • Upgrade click-option-group to 0.5.3
    • Upgrade pydantic to 1.8.2
    • Upgrade sentry_sdk to 1.1.0
    • Rename edx models
    • Migrate model tests from factories to hypothesis strategies
    • Tray: switch from openshift to k8s (BC)
    • Tray: remove useless deployment probes

    Fixed

    • Tray: remove version immutable field in DC selector
    Source code(tar.gz)
    Source code(zip)
  • v1.2.0(Feb 26, 2021)

    Added

    • EdX server event pydantic model and factory
    • EdX page_close browser event pydantic model and factory
    • Tray: allow to specify a self-generated elasticsearch cluster CA certificate

    Fixed

    • Tray: add missing Swift variables in the secret
    • Tray: fix pods anti-affinity selector

    Removed

    • pandas is no longer required
    Source code(tar.gz)
    Source code(zip)
  • v1.1.0(Feb 4, 2021)

    Added

    • Support for Swift storage backend
    • Use the push command --ignore-errors option to ignore ES bulk import errors
    • The elasticsearch backend now accepts passing all supported client options

    Changed

    • Remove click_log package dependency
    • Upgrade pyyaml to 5.4.1
    • Upgrade pandas to 1.2.1
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Jan 13, 2021)

    Added

    • Implement base CLI commands (list, extract, fetch & push) for supported backends
    • Support for ElasticSearch database backend
    • Support for LDP storage backend
    • Support for FS storage backend
    • Parse (gzipped) tracking logs in GELF format
    • Support for application's configuration file
    • Add optional sentry integration
    • Distribute Arnold's tray to deploy Ralph in a k8s cluster as cronjobs
    Source code(tar.gz)
    Source code(zip)
Owner
France Université Numérique
Collaboratively Building Innovative Digital Learning Services
France Université Numérique
cmdpxl: a totally practical command-line image editor

cmdpxl: a totally practical command-line image editor

Jieruei Chang 476 Jan 07, 2023
NudeNet wrapper made to provide a simple cli interface to the library

Nudenet Wrapper. Small warpper script for NudeNet Made to provide a small and easy to use cli interface with the library. You can indicate a single im

1 Oct 20, 2021
Oil is a new Unix shell. It's our upgrade path from bash to a better language and runtime

Oil is a new Unix shell. It's our upgrade path from bash to a better language and runtime. It's also for Python and JavaScript users who avoid shell!

2.4k Jan 08, 2023
A helper program to play wordle

A helper program to play wordle

1 Jan 22, 2022
EODAG is a command line tool and a plugin-oriented Python framework for searching, aggregating results and downloading remote sensed images while offering a unified API for data access regardless of the data provider

EODAG (Earth Observation Data Access Gateway) is a command line tool and a plugin-oriented Python framework for searching, aggregating results and downloading remote sensed images while offering a un

CS GROUP 205 Jan 03, 2023
Shortcut-Maker - It is a tool that can be set to run any tool with a single command

Shortcut-Maker It is a tool that can be set to run any tool with a single command Coded by Dave Smith(Owner of Sl Cyber Warriors) Command list 👇 pkg

Dave Smith 10 Sep 14, 2022
Command-line program for organizing and managing ebook collections

Command-line program for organizing and managing ebook collections. It is a Python port from the original shell scripts ebook-tools

Raul 14 Nov 12, 2022
Freaky fast fuzzy Denite/CtrlP matcher for vim/neovim

Freaky fast fuzzy Denite/CtrlP matcher for vim/neovim This is a matcher plugin for denite.nvim and CtrlP.

Raghu 113 Sep 29, 2022
AlienFX is a CLI and GUI utility to control the lighting effects of your Alienware computer.

AlienFX is a Linux utility to control the lighting effects of your Alienware computer. At present there is a CLI version (alienfx) and a gtk GUI versi

Stephen Harris 218 Dec 26, 2022
Wordle-cli - Command-line clone of Josh Wardle's WORDLE

Command-line clone of Josh Wardle's WORDLE, inspired by Paul Battley's Ruby vers

Klipspringer 32 Jan 03, 2023
Double Pendulum visualised with fetching system information in Python.

Show off your terminal, in style. A nice relaxing double pendulum simulation using ASCII, able to simulate multiple pendulums at once, and provide tra

Nekurone 62 Dec 14, 2022
spade is the next-generation networking command line tool.

spade is the next-generation networking command line tool. Say goodbye to the likes of dig, ping and traceroute with more accessible, more informative and prettier output.

Vivaan Verma 5 Jan 28, 2022
Command line client for Audience Insights

Dynamics 365 Audience Insights CLI The AuI CLI is a command line tool for Dynamics 365 Audience Insights. It is based on the customerinsights Python l

Microsoft 8 Jan 09, 2023
This is a Command Line program to interact with your NFTs, Cryptocurrencies etc

This is a Command Line program to interact with your NFTs, Cryptocurrencies etc. via the ThirdWeb Platform. This is just a fun little project that I made to be able to connect to blockchains and Web3

Arpan Pandey 5 Oct 02, 2022
A Python module and command line utility for working with web archive data using the WACZ format specification

py-wacz The py-wacz repository contains a Python module and command line utility for working with web archive data using the WACZ format specification

Webrecorder 14 Oct 24, 2022
Use case: quick JSON processing/restructuring with jq without terminal

alfred-jq Alfred workflow to conveniently process JQ on clipboard based on a jq query Also available at: packal/jq Use case: quick JSON processing/res

T on Meta Mode 5 Sep 30, 2022
inklayers is a command line program that exports layers from an SVG file.

inklayers is a command line program that exports layers from an SVG file. It can be used to create slide shows by editing a single SVG file.

11 Mar 29, 2022
A super simple wallet application for the NANO cryptocurrency that runs in the terminal

Nano Terminal Wallet A super simple wallet application for the NANO cryptocurrency that runs in the terminal Written in 2021 by NinjaSnail1080 (Discor

9 Jul 22, 2022
A simple note taker CLI program written in python

note-taker A simple note taker program written in python This allows you to snip your todo's, notes, and your tasks easily without extra charges Requi

marcusz 4 Nov 02, 2021
pypyr task-runner cli & api for automation pipelines.

pypyr task-runner cli & api for automation pipelines. Automate anything by combining commands, different scripts in different languages & applications into one pipeline process.

pypyr 471 Dec 15, 2022