Closes: https://github.com/pygfx/pygfx/issues/345 #145

This PR contributes a sphinx gallery that converts examples into a nice RST file including output renders. It also reorganizes the example folder into introductory, feature_demo, and validation examples to track them better and to display examples by category in the gallery.

Note that this is a large PR. Most of the changes are fairly light-weight, though, so a lot of the touching lines are things like adding a title in the docstring or renaming an example. The critical pieces are in conf.py (the sphinx config) and in utils/gallery_scraper.py.

This PR adds the framework for running examples on CI, just as in https://github.com/pygfx/wgpu-py/pull/238

Please help me identify which examples we want to run in the test suite!

This PR is the combination and refactoring of #309 and #319. It preliminarily realizes relatively complete lighting, shadows, and PBR system.

Main features:

• Lights & Shadows (and some helpers)

  • AmbientLight
  • DirectionalLight & DirectionalLightShadow
  • PointLight & PointShadow
  • SpotLight & SpotLightShadow

• New Phong Lighting model and Physically-Based Lighting model

• IBL support

• Two materials that are affected by light, MeshPhongMaterial, MeshStandardMaterial

  • The shader logic of MeshPhongMaterial has been greatly modified, so some old examples using MeshPhongMaterial may have some compatibility problems.
  • MeshStandardMaterial is a standard physically based material, using Metallic-Roughness workflow (Disney Principled)

I think for me, the process of realizing these functions is also the process of exploring some "wgpu" best practices. There are still some issues that need to be discussed, some of which have been written in the code comments. Maybe I need to sort it out later, and we can discuss some specific issues in detail.

Try to add a simple lighting system.

Now there are only PointLightand DirectionalLight, and only MeshPhongMaterialand MeshFlatMaterial are affected by the light. When using these two materials, if no light is added to the scene, a default DirectionalLight perpendicular to the screen used, Otherwise, use the light added to the scene.

Maybe some implements are not good enough, but it's a beginning.

Run the light_viewer.py example to see the specific effect.

Howdie, and thanks for this very cool looking library... just getting my toes wet.

forgive the basic question, and apologies if this is discussed elsewhere, but I'm seeing washed out colors in the astronaut image when I follow the examples:

# using either:..

image = gfx.Image(
    gfx.Geometry(grid=tex),
    gfx.ImageBasicMaterial(clim=(0, 255)),
)

image = gfx.Mesh(
    gfx.plane_geometry(512, 512),
    gfx.MeshBasicMaterial(map=tex.get_view(filter="linear")),
)

compared to vispy:

I'm on a mac (12.6), python 3.10, PyQt5 (though I also see it with jupyter-rfb in notebook).

curious if this would a lower OS-level driver-type issue? or something I can tackle within the pygfx code?

Trying my hand at a first contribution. I'm a bit confused by how buffers are bound and accessed.

Right now, this doesn't work (sizes are not read at all, it seems).

Text rendering

An overview of our options. Latest update 16-11-2021.

Goals

I think our goal should be:

• Aim for high quality text rendering (though not necessarily perfect).
• Make sure that there is a default font that looks good and is complete and always available.
• Provide support (one way or another) for all languages, including CJK.
• Preferably support user-provided fonts, so graphs in papers can be publication-worthy quality with a uniform style.
• Preferably an approach where we can start simple and improve on with further steps.

Methods to render text

Leveraging a GUI toolkit

We could leverage the text support of e.g. Qt to create text overlays. We have an example for this in pygfx.

Bitmaps

Historically, fonts are rendered using glyph bitmaps that match the screen resolution. All glyphs in use are put into an atlas: a large image/texture that has all glyphs packed together.

A downside is that the bitmaps should match the screen pixels, so the bitmaps are font-size specific, and rendering glyphs at an angle is not possible without degrading quality.

SDF

In 2007 Chris Green from Valve published an approach that uses scalar distance fields (SDF) to render glyphs. This approach makes it possible to create a special kind if bitmap (which encodes distances to the edge instead of a mask), that can then be used to render that glyph at any scale and rotation. This was a big step in games, but also for generic visualization.

There are variations on the way to calculate an SDF. The current industy standard seems to be anti-aliased euclidian distance transform.

MSDF

Around 2015, Viktor Chlumsky published his Master thesis, in which he proposes a method for a Multi-channel Signed Distance Field (MSDF). The key contribution is that directional information is encoded too, making is possible to produce sharp corners. It produces better results than normal SDF, at a smaller scale. The fragment shader is changed only slightly, causing just a minor performance penalty.

Glyphy

The Glyphy C library converts bezier curves to arcs and then calculates the signed distance to those arcs in the shader, producing better results than normal SDF. It's not well documented, but is by the author of Harfbuzz ...

Vector rendering

It's also possible to render the Bezier curves from the glyphs directly on the GPU. See e.g. http://wdobbie.com/. This approach seems to require some preprocessing to divide the bezier curves into cells. If this can be changed to create an atlas (i.e. each cell is one glyph) then this may be a feasible approach for a dynamic context like ours. Also see Slug, which looks pretty neat, but is not open source and covered by a patent.

Also see this fairly simple approach: https://medium.com/@evanwallace/easy-scalable-text-rendering-on-the-gpu-c3f4d782c5ac It uses a prepass to calculate the winding number for the rough shape, then composes the final shape and applies aa in a fragment shader. A disadvantage (for us) is that it needs multiple passes, making it harder to fit into our renderer, I think.

How do other libs do text rendering?

How visvis does text

https://github.com/almarklein/visvis/tree/master/text

Visvis produces glyphs on the fly using FreeType. Freetype is available by default on Linux, on Windows it must be installed. If FreeType is not available, visvis falls back on system that uses pre-rendered fonts. Visvis also ships with FreeSans, so there is always a good sans font available. Further, it can make use of other fonts installed on the system.

The rendered glyps are on a fixed resolution and put in an atlas texture. When rendered, the glyph is sampled using a smoothing kernel, giving reasonable results at multiple scales.

Further, visvis includes a mini-SDL to create bold, italic, and math symbols using escaping with backslash.

How vispy does text

https://github.com/vispy/vispy/tree/main/vispy/util/fonts and https://github.com/vispy/vispy/tree/main/vispy/visuals/text

Vispy produces SDF glyphs on the fly. It first gets a glyph bitmap using FreeType (via Rougier's FreeType wrapper) on Windows and Linux, and Quartz on MacOS. The bitmap is then converted to an SDF using either the GPU or Cython. It can use most fonts installed on the system. The code to list and load fonts is platform specific.

The SDF glyphs are packed into an atlas texture, and rendered using regular SDF rendering.

How Matplotlib does text

Unless I'm mistaken, they use the GUI backend to render text.

Discussion

SDF rendering already provides pretty solid results and is widely used. MSDF seems like a very interesting extension that may make it possible to create better results with smaller SDF sizes. Whether this is true depends also on the complexity of the glyphs - it has been reported that it does not provide much advantage for e.g. Chinese glyphs. We'd need tests to visually judge what glyph sizes are needed in both approaches.

In terms of dependencies, to create SDF we can get away with just FreeType to create the bitmaps, plus steal some code from vispy to generate the SDF. For MSDF there is that one project, but its not that actively maintained, so at this point it may be a liability. It may be worth investigating if its possible to generate an MSDF from a bitmap.

Vector rendering seems to be popular lately. It produces better results, but is more complex and less performant than SDFs. Further you may need a bitmap fallback for high scales. There is also the question of how to generate the vector-data from the font files.

If you want to do render text perfectly, you should take into account kerning, ligatures, combinatory glyphs, left-to-right languages, etc. We can do kerning, but the rest is way more complex. Tools that handle these correctly are e.g. Slug and Harfbuzz. The latter might be feasible to use - there are multiple Python wrappers.

More details obtaining a glyph

I imagine that somewhere in our code we have a method create_glyph(font, char) that returns an SDF/MSDF/bezier-curve-set and some metadata. We can place this info in an atlas texture together with other used glyphs. Now, how does this function work?

The below more or less assumes SDF / MSDF, but similar questions apply to the Bezier-based rendering.

Using prerendered glyph maps

We could produce prerendered glyph atlas, so create_glyph() samples one glyph from that (offline) atlas and to put it in our GPU atlas. That way, the dependencies for rendering the glyphs are only needed by devs, and not the users. See e.g. https://github.com/Chlumsky/msdf-atlas-gen A disadvantages is that the font and available characters are limited by what atlasses we chose to ship. If we'd ship CJK fonts, that would grow the package size a lot, so maybe we'd need a pygfx-cjk-fonts or something.

Some measurements for rendering all chars from a font:

• OpenSans -> 1011 glyphs -> 1.5MB @ 32 0.5MB @ 16
• FreeSans -> 4372 glyphs -> 7MB @ 32 2.3MB @ 16
• NotoSans -> 2300 glyps -> 3.5MB @ 32 1.1MB @ 16
• NotoSansJP -> 16735 glyps -> ?? 15.1MB @ 16

This shows that with 16x16 MSDF glyphs, the atlas is about 3x the size of the .ttf file. Or 9x for 32x32. Not bad, but I don't now yet whether 16x16 is enough, especially for the more complex glyphs in CJK fonts.

Using a tool to render glyph on the fly

If we can produce the SDF glyph from the fontfile at the user, then support for all languages, and custom fonts is obtained. The cost is that we need a dependency for this.

One option is FreeType. There is Rougiers https://github.com/rougier/freetype-py which provides binaries for multiple platforms (but not MacOS M1 or Linux aarch64 yet). IIUC freetype can render a glyph bitmap at any scale, but to create an SDF we need to do some extra work ourselves. I am not sure if a MSDF is feasible, because you'd miss the vectors, or can we do edge detection on a high-rez glyph to find the orientation?

Another option is https://github.com/Chlumsky/msdfgen. We'd need to build and distribute the code for all platforms that we support. Since this code is not that actively maintained, this feels a bit tricky.

Typesetting

Typesetting is the process of positioning the quads that contain the glyphs. This includes kerning (some combination of glyhs have a custom distance). Also inclded are justifcation, alignment, text wrapping etc.

I think its not too hard to do this in pure Python, though existing solutions may make things easier. Examples are Slug and Harfbuzz, but these are not feasible dependencies. I don't know of any Python libraries for typesetting.

For the kerning we would need the kerning maps. FreeType can provide this info. I think that msdfgen also provides it in the CSV/JSON metadata.

Formatting

We should consider a formatting approach so users can create bold/italic text etc. Visvis uses escaping, but maybe an micro-HTML-subset would be nicer. Or something using braces. Or maybe its out of scope. This is definitely something that we can do later.

Fonts of interest

• OpenSans is a pretty nice and open font.
• FreeSans is a GNU project that aims to cover almost all Unicode, except CJK.
• Noto is a family of over 100 fonts that aims to have full Unicode coverage. It basically has a font for each script. The plain NotoSans has over 3k glyphs and with support for Latin, Cyrillic and Greek scripts it covers 855 languages.

Closes #271 and closes #20

Todo:

• General
  • [x] High level version of the API.
  • [x] Document the text rendering process.
  • [x] Be specific about what font properties affect what, and where the user can set them.
  • [x] Text can be a child of another object.
  • [x] Make it such that some steps can be overloaded for custom behavior.
  • [x] Make it such that the builtin steps can (hopefully) be replaced without affecting the API.
  • [x] Flesh out the API.
  • [x] Make things work for RTL fonts.
  • [x] Make basics work for vertical scripts.
  • [x] Fix bug that when text is replaced, parts of the old text seems to linger sometimes.
  • [x] Implement geometry.bounding_box() (or gfx.show() wont work).
• Discuss
  • [x] Place to put properties that affect positioning.
  • [x] Screen-space vs world-space, how to express in API.
  • [x] What builtin fonts to include (Noto sans!).
  • [x] What should happen with the text object's transform if it's set to render in screen space.
• Itemization
  • [x] Plain text
  • [x] Very basic markdown.
  • [x] Tests.
• Font selection
  • [x] Basic font selection (stub).
  • [x] Include builtin fonts, a selection of noto fonts.
  • [x] Decide what to do about font management. #379
  • [x] Clean up the font manager module.
  • [x] Include / delete builtin fonts.
  • [x] Tests.
• Shaping
  • [x] Stub shaping that just about works.
  • [x] Basic shaping with Freetype.
  • [x] Advanced shaping with Harfbuzz.
  • [x] Tests.
• Glyph generation and atlas
  • [x] Glyph generation with Freetype.
  • [x] Atlas that can pack variable sized glyphs.
  • [x] Dynamic atlas sizing (grow and shrink).
  • [x] Tests.
• Layout (minimal stuff)
  • [x] Put chars in a straight line.
  • [x] Text size.
  • [x] Alignment: TextGeometry.anchor
  • [x] Support custom layout.
  • [x] Rotate text at an angle, also in screen space.
  • [x] Tests.
• Rendering
  • [x] Render rectangles at the glyph positions.
  • [x] Render the texture content.
  • [x] Actual SDF rendering.
  • [x] Approximate weight.
  • [x] Slanting.
  • [x] Outlines.
  • [x] Picking.
  • [x] Validation examples.
• Examples
  • [x] Minimal text example.
  • [x] Show white on black, and black on white and make sure they feel equally thick.
  • [x] A waterfall of glyphs, to strain the atlas.
  • [x] Use new gfx.show()
• Final stuff
  • [x] Update new validation example screenshots.
  • [x] Docs.
  • [x] More tests?
  • [x] Fix todos & self-review.

Tasks for later are listed in #391.

Status: can render text, but each character is just a little rectangle. Note: it says "Hello world" :)

Closes #112, closes #116. Depends on https://github.com/pygfx/wgpu-py/pull/181

This touches on some of the core mechanics of the renderer and how it interacts with the canvas. This affects having multiple canvases, subplots, overlays, sub-scene rendering, and post-processing.

• [x] Support for using multiple canvases in the same application, and rendering world-objects in multiple, because the device is shared/global.
• [x] Support for overlays because you can renderer.render() multiple times and set a new clear param.
• [x] Support for subplots because of a new region param in renderer.render() which is in logical pixels and is translated to a viewport in physical pixels.
• [x] Remove post-processing system (in favor of a single submit-step).
• [x] Fix issues with submit related to texture size and format.
• [x] Allow multiple renderers per target?
• [x] Rename any leftover 'submits" to "flush to target" or something similar.
• [ ] ~Maybe provide hooks for submit step in wgpu-py.~
• [x] Maybe rethink offscreen rendering (if we allow WgpuRenderer(texture)).
• [x] Resolve any todo's that I added along the way.
• [x] Update docs in guide a bit. #119

How the renderer now works (high level POV)

The renderer's .render(scene, camera) function renders that scene to it's internal render target. This target is an implementation details (it may change e.g. with OIT). You can call .render() multiple times. The color and depth are automatically cleared before the first .render() call.

The final result is automatically "submitted" to the texture provided by the canvas. This submit-step includes any FSAA. One can also call .submit_to_texture() to submit the result to another texture instead.

Approach per use-case

For each of these at least one example is added.

Use multiple canvases

You can now use multiple canvases. Each canvas can have exactly one renderer associated with it. You can render one scene to multiple canvases.

Blending scenes

By calling .render() multiple times, the results should just blend, as if they were in one scene. For opaque objects this is easy. Eventually this should work with semitransparent objects too (if/when we get OIT working). Not sure what specific use-cases would be, but anyway, this is how it works :)

Subplots

By rendering multiple times, but to different region rects, you can implement subplots.

Overlays

By rendering multiple scenes on top of each other and calling .clear(color=False, depth=True) in between, the last scene is always on top.

A scene in a scene

The typical example of having a surface in your scene (a TV screen if you like) that depicts another scene can now be implemented by rendering the subscene via an offscreen canvas, then submitting the result into a texture that is rendered via a mesh in the outer scene.

Post-processing

Similar to the above we can make the outer scene just a single quad that draws the inner scene and do something to it. But the canvas and renderer can be re-used.

An alternative is to provide hooks for the submit step to allow for simple post-processing mechanics. Also e.g. fog. Maybe for later.

Before this PR there is a system for post-processing steps, basically because the submit-step has been generalized. I plan to remove that to simplify things.

(I don't think this PR has an issue tracking the problem. If it does, please reference)

The current example in the guide is broken; gfx.BoxGeometry changed to gfx.box_geometry and the example doesn't add light sources, so you will only see a black screen.

This PR updates the guide to match what we are currently doing in the cube example. It also rewrites it a bit to reflect the addition of light sources and adds some (static) visualizations.

I also found a typo in Contributing.md that I'm fixing here.

In a previous discussion (https://github.com/pygfx/pygfx/pull/371#discussion_r1014485572) I learned that you often don't need to interact with the canvas object directly when using pygfx. There are, of course, exceptions when this can be advantageous, but in general this is not the case. Because of this, we have introduced renderer.request_draw which wraps canvas.request_draw.

Taking this thought and thinking it to its conclusion I think it would make sense to also see if we can avoid all calls to canvas by default so that we only have to instantiate it when needed. In particular, I was wondering if we could change

from wgpu.gui.auto import WgpuCanvas, run
import pygfx as gfx

renderer = gfx.renderers.WgpuRenderer(WgpuCanvas())

to

import pygfx as gfx, run 

renderer = gfx.renderers.WgpuRenderer()

and make wgpu.gui.auto.WgpuCanvas the default canvas. We would, of course, keep the parameter and allow users to set it manually if needed. My angle here is to see if we can reduce boilerplate code.

On the same token, we could wrap run and avoid the import of wgpu for minimal examples. It's always nice to not have to import lower-level functionality when dealing with something on a more high level.

For example in this video the length of each of these lines (left to right) is 8410, and only a few points along the line have high alpha values. The artifact is that arrows appear at the edges of the where the colors fade out (could it be an interpolation thing?).

https://user-images.githubusercontent.com/9403332/210306005-19fa7d20-b237-416b-af0e-849ae57a6200.mp4

The lines are instantiated like this:

pygfx.LineMaterial

pygfx.Line(
  geometry=pygfx.Geometry(positions=<[8410, 3]>, colors=<[8410, 4]),
  material=material(thickness=20, vertex_colors=True)
)

I tried adjusting the camera near and far planes but it doesn't make a difference. GPU is an nvidia RTX 3090, OS is Debian 11, vulkaninfo seems fine.

Anyways the purpose is to use color intensity of a very long line that represents timepoints to indicate the magnitude of a quantitative variable, not sure if it's possible to do this with a simple Mesh instead if that's better than a Line?

At present, all rendering calculations in our shader are in world-space, but it is better to convert all objects to view-space before entering the shader.

That is to say, we don't need to pass in "model_matrix" and "view_matrix" respectively to the shader, just pass in their product "model_view_matrix".

This has many advantages:

1. When using view space for calculation, all camera positions are fixed at the coordinate origin, which is conducive to simplifying some algorithms.

2. When the scene is large, it is calculated in the world space. Some numbers may be large (objects that are far from the origin of the world coordinates), which will lead to calculation accuracy problems. In view space, all numbers are relatively small.

3. Each vertex in the shader needs to perform the operation of projection_matrix * view_matrix * model_matrix * local_pos. When using view space, we only need to perform the calculation of projection_matrix * model_view_matrix * local_pos, which saves one matrix multiplication. I think this point is also true for other calculations. Some "constant calculations" (The calculation results are the same for each vertex or fragment) should be performed on the CPU as much as possible (only calculate once), rather than on each vertex or fragment (such as the conversion of constant color from srgb-space to linear-space). This is a good way to improve rendering performance.

If we decide to make this change, it may have a large impact and may affect other features developed at the same time. Therefore, we'd better make this change when the version is relatively stable.

The plot prefix is a mechanism from Sphinx-gallery to select examples for inclusion in the gallery. We now also have # sphinx_gallery_pygfx_render = True, which we can set to False, right? And the prefix can (I assume) be set to an empty string?

cc @FirefoxMetzger

I am looking into the way shaders are written in pygfx. According to the docs, this is done by subclassing WorldObjectShader which has to define a class-level variable called type. Would it be useful to rename this variable to kind in order to avoid aliasing the built-in type function?

• Interactive selection (e.g. by rectangle or polygon)
  • Support retrieving picking info for a rectangular region in a single call, instead of just a single pixel
  • Disable objects from participating in picking