A system for managing CI data for Mozilla projects

Overview

Treeherder

What's Deployed Build Status Node dependencies Status Node devDependencies Status Documentation Status Code style: black

Description

Treeherder is a reporting dashboard for Mozilla checkins. It allows users to see the results of automatic builds and their respective tests. The Treeherder service manages the etl layer for data ingestion, web services, and the data model behind Treeherder.

Instances

Treeherder exists on two instances: staging for pre-deployment validation, and production for actual use.

Installation

The steps to run Treeherder are provided here.

The steps to run only the UI are provided here.

Links

Visit our project tracking Wiki here.

For other setup and configuration, visit our readthedocs page here.

File any bugs you may encounter here.

Contributing

Everyone is welcome to contribute!

If a bug is not assigned to someone, you can request the bug be assigned to you. You should ask the component owner with your request ("Request information" in Bugzilla and mention in Github).

If you do not receive a response within 2-3 days, you can follow up in the #treeherder matrix channel.

After addressing the issue, make sure every test passes before sending a pull request.

We also recommend setting an upstream remote that points to the Mozilla's Github repo, in addition to origin that points to your fork. You should then frequently use git rebase upstream rather than merging from your fork to keep your branch current. There are less conflicts this way and the git history is cleaner.

Sending a Pull Request

We receive contributions from both Bugzilla and Github. We have some specifications to keep track of them:

  1. If your bug comes from Bugzilla

    After addressing the issue, please send a pull request to this repository, with the Bugzilla's number ID in the title, so that our bot attaches your patch to the corresponding Bugzilla bug.

    "Bug xxxxxx - [title of the bug or brief explanation]"

    For example: "Bug 123456 - Fix scrolling behavior in Perfherder"

  2. If your bug comes from Github

    In the description of the pull request, please mention the issue number. That can be done by typing #[issue's number].

    For example: "This pull request fixes #5135".

    Github automatically links both issue and pull request to one another.

Comments
  • Bug 1395356 - Use auth0 instead of login.taskcluster.net for SSO

    Bug 1395356 - Use auth0 instead of login.taskcluster.net for SSO

    Rough summary of the changes

    Front end

    The auth callback is written in React and lives under the /login.html endpoint. It communicates with Treeherder using the localStorage.

    Credential expiration

    The Django user session expiration is set to expire when the client access token or the id token expires (whichever one expires first). These values are controlled by the IAM team. Presently, the access token expires after 1 day and the id token expires after a week. That being said, the session will therefore expire after 1 day. If you want this value change, we simply need to send a request to the IAM team.

    Credential renewal

    Renewals are set to happen every 15 minutes or so. The renewal is skewed slightly so that different open tabs don't renew at the same time. Once renewal happens, both tokens are renewed and the Django session is updated.

    Migration

    If the userSession localStorage key is not set, then the user will be logged out including logging out from the Django session. In other words, all users will be automatically logged out when the merge to production happens.

    opened by helfi92 48
  • Bug 1315808 - New Log Viewer integration

    Bug 1315808 - New Log Viewer integration

    Main Features

    • Be able to highlight different error lines
    • Get rid of "double hashing" when linking to a specific line number
    • Smooth scrolling
    • Highlight single and multiple lines
    • Follow log (option to scroll to the bottom of the log)

    This change is Reviewable

    opened by helfi92 38
  • Bug 1117583 - Make a custom form for filing intermittent bugs

    Bug 1117583 - Make a custom form for filing intermittent bugs

    Apparently I can't reopen a closed pull request if I've force-pushed to it in the time it was closed. Fun.

    Anyway... This is very close to being ready for review. Still need to fix up a few things, but I'm hoping to get that done by the end of the week.

    Review on Reviewable

    opened by KWierso 36
  • Bug 1108764 - Allow easy sharing of links to a specific log line in the log viewer

    Bug 1108764 - Allow easy sharing of links to a specific log line in the log viewer

    This is the first draft. I still have to change some stuffs(lot of stuffs), but I'd like to have some feedback, to see if I'm doing everything right, or if something can be changed.

    Review on Reviewable

    opened by gbrmachado 33
  • Bug 1539232 - Switch Perfherder to react-router

    Bug 1539232 - Switch Perfherder to react-router

    This is the final react conversion pr - woohoo!

    In addition to setting up the routes, I've made these other changes:

    • Using top-level of app as a cache for projects, frameworks, alerts data and compare data
    • Cleanup files and move constants to dedicated perfherder file
    • Removed angular-related libraries and bump down the neutrino entry and asset limits

    @ionutgoldan it'd be worth you giving all the views a quick test. I unfortunately had a lot of squashing of commits to do during the rebase and I want to make sure the new retrigger action is working as expected. I'd also like to ensure that the fetched data that's stored from the alerts view (when navigating back from an individual alert), and the fetched data that's stored from the compare view (when navigating back from the comparesubtest view) works as expected and doesn't cause any issues. I've launched it to prototype.

    FYI @karlht and @davehunt

    opened by sarah-clements 26
  • 1600761 - Visual/Color improvements to Treeherder apps

    1600761 - Visual/Color improvements to Treeherder apps

    Description of issue

    Some color combinations does not pass the WCAG contrast recommendation.

    Also, for color blindness distinction, texts with red and green should be accompanied by an icon.

    Strategy

    Using Firefox Accessibility Inspector's beta "Check for issues", I've checked for "Contrast" issues. That was done for each view and a different commit was made. Then, it was a manual color tweaking of each issue.

    Some colors became harder to distinguish because of the contrast rate. That was seem mostly in blue and green colors. Maybe propose a different color hue?

    Testing

    Those were mostly visual and using Accessibility and Inspector devtools tabs.

    opened by yogmel 25
  • Bug 1286686 - Initial move of Treeherder-Tests into /selenium folder

    Bug 1286686 - Initial move of Treeherder-Tests into /selenium folder

    Moving the contents of the Treeherder-Tests repo into the /selenium folder. Tests will not run automatically, as Travis will need to be set up. This is just to get the content into the primary repo going forward so that the Treeherder-Tests repo can be decommissioned.


    This change is Reviewable

    opened by rbillings 25
  • Bump django from 3.1.13 to 3.2.8 in /requirements

    Bump django from 3.1.13 to 3.2.8 in /requirements

    Bumps django from 3.1.13 to 3.2.8.

    Commits
    • 4540e97 [3.2.x] Bumped version for 3.2.8 release.
    • 65367b0 [3.2.x] Added release date for 3.2.7.
    • 51e4dbf [3.2.x] Refs #27694 -- Doc'd lookups that can be chained with HStoreField key...
    • 031ffc5 [3.2.x] Corrected field and model check messages in docs.
    • 7607fe9 [3.2.x] Removed obsolete GEOS 3.5 requirement note.
    • 6760f4f [3.2.x] Fixed #33083 -- Fixed selecting all items in the admin changelist whe...
    • e235c78 [3.2.x] Fixed broken links and redirects in docs.
    • 51e76c9 [3.2.x] Used :rfc: role in docs/topics/conditional-view-processing.txt.
    • d4a587a [3.2.x] Fixed #33077 -- Fixed links to related models for admin's readonly fi...
    • 561a1c0 [3.2.x] Fixed typo in docs/intro/reusable-apps.txt.
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies python 
    opened by dependabot[bot] 22
  • [WIP] Fetch the CI changelog and display it in the perfherder timeline

    [WIP] Fetch the CI changelog and display it in the perfherder timeline

    This is a just a WIP demo at the moment, but I'm filing a PR for more visibility/discussion. This patch adds some red lines to the Perfherder graph to show when changes to our CI machines have occurred so that sheriffs have an easier time telling whether a change in perf numbers is due to a code change or a CI config change. Hovering over/clicking on the red lines will show a popup describing the change with links to the commit.

    Note: this currently shows every change from the CI changelog for testing, so there are a lot more red lines than we'd normally see. In a real version of this, we'd probably restrict it to only show actual deployments.

    javascript 
    opened by jimporter 22
  • Bug 1600759 - Perfherder's Graph Alt. View for Screen Readers

    Bug 1600759 - Perfherder's Graph Alt. View for Screen Readers

    Description of issue

    Perfherder's Graph View contains graphs and elements which are not screen reader accessible.

    Tasks:

    • [X] Get Graph data into table;
    • [X] Highlight Revision;
    • [x] Add Tooltip information;
    • [ ] When highlight, show and focus highlighted row;
    • [ ] Toogle column's visibility;
    • [x] Add tests.

    Next PR:

    • (Needs investigation) Show trends and outliers;

    Proposed Solution

    Perfherder's Graph View, with the Table showing. On the left, there are legend cards, with all tests information. Turned on tests have distinct colors, such as blue and red and different icons. On the right, there is first a row of buttons: "Last day", "Add test data" and "Graphs View". Below that, there is the table, with navigation on top. The first column refers to the revision test date. The other columns correspond to each test data. There 7 columns in total (One is the date column and the other six are tests). Some rows have a yellow background. Below the table, there are two input fields and a "Highlight alerts" button.

    • A button next to "Add test data" was added, to toggle between Graph and Table view.

    • If there is any highlighted revision, its table row will have a yellow background and an aria-label of "highlighted revision".

    • Users can show and hide test via Checkbox in Legend Card: The same table as the first screenshot, but the first two Legend cards are turned off, that is, they are grey. This reflects in the table, which now has 5 columns instead of 7.

    Testing

    • Compared some data points in the graph to the ones shown in table cells.
    • Turned highlighted revisions on and off and check if it propagated both in the graph and table.
    • Still was not able to test with SR. I will edit here once it's done.
    opened by yogmel 21
  • Bug 1574628 - Improve graph colors and add symbols

    Bug 1574628 - Improve graph colors and add symbols

    Description of issue

    The Graph view in Perfherder is not very usable for color blind users, mainly when there are multiple types of data plotted. To improve it, it was suggested to choose a friendlier color palette and also use auxiliary symbols to help distinguish between data.

    • [x] Change symbols in graph to different ones
      • [X] When graph is first created
      • [x] When more graphs are added
    • [x] Insert symbol in LegendCard
    • [X] Propose better color palette

    Proposed Solution

    • In order to change the graph symbols, the VictoryScatter prop symbol in VictoryChart's library was used. The symbol name is passed from GraphView to GraphContainer, based on the stored value from constants.js

    • In LegendCard, the name of the symbols have been retrieved, but in order to insert the icon, some decisions must be made - the icons are <svg> and live in another file? Are they React Components? Maybe we could use the same svg Victory Chart uses.

    • As for the color palette, some research and experiments were made. 3 of the 6 colors were changed.

    Testing

    A visual test was made, with help of NoCoffee Add-On. It was tested with all available Color deficiency options.

    Screenshot of proposed color scheme for Perfherder's graph. There are legend cards on the left with 6 different border colors. On the right, there is the graph, with different symbols scattered

    opened by yogmel 20
  • Add GhA task to run the probe-scraper workflow

    Add GhA task to run the probe-scraper workflow

    This ensures:

    • On pull-requests probe-scraper can verify that Glean metrics are declared correctly.
    • On pushes to the default branch probe-scraper is informed to generate metrics data in the pipeline, to fill the Glean Dictionary and populate the table schemas.

    This is part of Step 4 of adding Glean to your product. See also https://github.com/mozilla/probe-scraper/pull/538 for enabling it in probe-scraper to begin with. This requires enabling GitHub actions on this repository. Someone with write/admin access probably needs to do that.

    opened by badboy 2
  • Bug 1785833 - ingest bug creation date

    Bug 1785833 - ingest bug creation date

    Ingest the bug creation timestamp to count new intermittent bugs without depending on timestamp of first classification. Setting as bugs as duplicates breaks this assumption.

    opened by Archaeopteryx 1
Releases(2020-09-17)
Owner
Mozilla
This technology could fall into the right hands.
Mozilla
Dockerized service to backup all running database containers

Docker Database Backup Dockerized service to automatically backup all of your database containers. Docker Image Tags: docker.io/jandi/database-backup

Jan Dittrich 16 Dec 31, 2022
📦 Powerful Package manager which updates plugins & server software for minecraft servers

pluGET A powerful package manager which updates Plugins and Server Software for minecraft servers. Screenshots check all to check installed plugins fo

106 Dec 16, 2022
sysctl/sysfs settings on a fly for Kubernetes Cluster. No restarts are required for clusters and nodes.

SysBindings Daemon Little toolkit for control the sysctl/sysfs bindings on Kubernetes Cluster on the fly and without unnecessary restarts of cluster o

Wallarm 19 May 06, 2022
Micro Data Lake based on Docker Compose

Micro Data Lake based on Docker Compose This is the implementation of a Minimum Data Lake

Abel Coronado 15 Jan 07, 2023
Deploying a production-ready Django project using Nginx and Gunicorn

django-nginx-gunicorn This project is for deploying a production-ready Django project using Nginx and Gunicorn. Running a local server of Django is no

Arash Sayareh 8 Jul 03, 2022
Define and run multi-container applications with Docker

Docker Compose Docker Compose is a tool for running multi-container applications on Docker defined using the Compose file format. A Compose file is us

Docker 28.2k Jan 08, 2023
Simple ssh overlay for easy, remote server management written in Python GTK with paramiko

Simple "ssh" overlay for easy, remote server management written in Python GTK with paramiko

kłapouch 3 May 01, 2022
CTF infrastructure deployment automation tool.

CTF infrastructure deployment automation tool. Focus on the challenges. Mirrored from

Fake News 1 Apr 12, 2022
Learning and experimenting with Kubernetes

Kubernetes Experiments This repository contains code that I'm using to learn and experiment with Kubernetes. 1. Environment setup minikube kubectl doc

Richard To 10 Dec 02, 2022
Ajenti Core and stock plugins

Ajenti is a Linux & BSD modular server admin panel. Ajenti 2 provides a new interface and a better architecture, developed with Python3 and AngularJS.

Ajenti Project 7k Jan 03, 2023
Cross-platform lib for process and system monitoring in Python

Home Install Documentation Download Forum Blog Funding What's new Summary psutil (process and system utilities) is a cross-platform library for retrie

Giampaolo Rodola 9k Jan 02, 2023
Self-hosted, easily-deployable monitoring and alerts service - like a lightweight PagerDuty

Cabot Maintainers wanted Cabot is stable and used by hundreds of companies and individuals in production, but it is not actively maintained. We would

Arachnys 5.4k Dec 23, 2022
Travis CI testing a Dockerfile based on Palantir's remix of Apache Cassandra, testing IaC, and testing integration health of Debian

Testing Palantir's remix of Apache Cassandra with Snyk & Travis CI This repository is to show Travis CI testing a Dockerfile based on Palantir's remix

Montana Mendy 1 Dec 20, 2021
HB Case Study

HB Case Study Envoy Proxy It is a modern Layer7(App) and Layer3(TCP) proxy Incredibly modernized version of reverse proxies like NGINX, HAProxy It is

Ilker Ispir 1 Oct 22, 2021
Hatch plugin for Docker containers

hatch-containers CI/CD Package Meta This provides a plugin for Hatch that allows

Ofek Lev 11 Dec 30, 2022
This is a tool to develop, build and test PHP extensions in Docker containers.

Develop, Build and Test PHP Extensions This is a tool to develop, build and test PHP extensions in Docker containers. Installation Clone this reposito

Suora GmbH 10 Oct 22, 2022
CDK Template of Table Definition AWS Lambda for RDB

CDK Template of Table Definition AWS Lambda for RDB Overview This sample deploys Amazon Aurora of PostgreSQL or MySQL with AWS Lambda that can define

AWS Samples 5 May 16, 2022
Tiny Git is a simplified version of Git with only the basic functionalities to gain better understanding of git internals.

Tiny Git is a simplified version of Git with only the basic functionalities to gain better understanding of git internals. Implemented Functi

Ahmed Ayman 2 Oct 15, 2021
SSH tunnels to remote server.

Author: Pahaz Repo: https://github.com/pahaz/sshtunnel/ Inspired by https://github.com/jmagnusson/bgtunnel, which doesn't work on Windows. See also: h

Pavel White 1k Dec 28, 2022
DC/OS - The Datacenter Operating System

DC/OS - The Datacenter Operating System The easiest way to run microservices, big data, and containers in production. What is DC/OS? Like traditional

DC/OS 2.3k Jan 06, 2023