Subsections

- Blend programming environment
- Supplementary color selection
- Processed tracks arrangement
- User library, System library and Project directories
- Syntax of Blend Algebra / Blend Program functions
- The compilation script
- Environment variables influencing plugin configuration
- What comes in the plugins
- Caveats
- Hardware acceleration
- Debugging
- Portability

##
Blend Algebra and Blend Program

The Blend Algebra/Blend Program plugins provide the advanced feature of combining and modifying color pixels and transparencies of several tracks according to a mathematic algorithm written entirely by the user in the form of compact and simple piece of code. Such user defined algorithms (blend functions) are compiled, dynamically linked and immediately executed inside CINELERRA-GG on the fly without the need to restart the application, reload the project, reattach plugins, and so on.

The following description is based on the /cinelerra-5.1/doc/README.blendalg that accompanies the code by the author of both the code and the document. It is the authoriative documentation; the most detailed/original source and has additional commentary and programming information. If you encounter problems or confusion when using these plugins, always refer directly to that document here. If running CINELERRA-GG from an AppImage, testing has shown that it should work as well as if run from a build - just be aware of filename extensions that appear by default in the input field for the funtion name and change them accordingly.

The idea behind these plugins6 is that the user can program sufficiently short and simple blending algorithms, called Functions, by following detailed instructions with very little programming skills required. Blend Algebra and Blend Program come with a library of Functions (System Library) that mimic the 30 overlay effects found in the Patchbay or Overlay plugins. The user can modify these Functions as needed or create new ones. To look at these example functions, there is an Edit... button in the respective plugin dialog. You will need a text editor to do so, which by default is emacs if it is installed. If not, you can set your favorite editor via the corresponding environment variable in a terminal (export CIN_EDITOR='name editor') and then start CINELERRA-GG from that same terminal.

If you can not wait to get started, checkout these examples here. You will also find test projects to download to learn more about the plugins and to see the various ways they can be used to create your own functions.

Both plugins, Blend Algebra and Blend Program, are similar. They are described together along with their differences. The main difference between the two plugins is that Blend Algebra is like a function; it combines its arguments (tracks) and yields the result which is placed into the single track configured for output. However, Blend Program does not produce a function result, but instead directly modifies its arguments. If more than one track is to be modified, Blend Program is the only choice. If a single track is to be processed, Blend Program is the best choice and might run slightly faster. Any Blend Algebra function can be rewritten to become a Blend Program function. But if a function processes several tracks and modifies only one, it will run faster in Blend Algebra which has more controls in its plugin dialog.

How the plugin works is defined by the user. The word function is used to denote this for either of the two plugins throughout this description. These functions are for their plugins the same as the plugin shared objects in the main CINELERRA-GG binary. But these functions are simple enough that they can be easily written by users and manipulated in binary or in source form. The plugins make it possible to have a user library and a system library of functions. Thus, an experienced user can have a number of such functions to reuse in many projects.

The plugin is attached to a track as usual. Generally, it needs several tracks as arguments and it has to be attached to the additional tracks as a shared effect.

All the parameters described below are keyframable, including user functions themselves. The top area of the dialog menu deals with programmatic aspects. The middle area of the dialog is color specific. The bottom area of the dialog has controls for the argument track ordering, similar to that of the Overlay plugin.

Figure 10.26: Blend Algebra and Blend Program GUI

###
Blend programming environment

Function
  This textbox contains the filename of the user's function source text. The most convenient method to select the function is via the Attach... dialog, but you can just enter its filename directly. If the function pathname does not start with '/', it is interpreted relative to the main project directory. An empty function name means that no function will be executed and the tracks will remain unchanged. If the entered filename does not exist, again nothing will be done, but in this case the user can create a source file via the Edit... button. An error message will be shown when a directory is selected in the file selection dialog instead of a filename. Although you can use any filename, it is highly recommended to use distinct suffixes: .ba for Blend Algebra, .bp for Blend Program or you will have to keyin specific filenames in some cases.
Attach...
  This button opens the file selection dialog menu as usual for Cinelerra. By default it shows only files with the recommended suffixes (*.ba or *.bp), although this can be changed by specifying a filter. The user can select an existing file or enter a filename. You can exit the dialog with Cancel so that nothing is changed or with OK for changes to be in effect. In that case, the selected function will be set as current, and its name will appear in the Function: input field above. An error message will be shown if the user presses Edit... with an empty function name or when a directory is selected in file selection dialog instead of a filename. If the entered filename does not end with .ba or .bp, the corresponding suffix will be appended automatically. If you use a filename with another suffix or none at all, you have to change the name directly in the Function: input field after exiting the File Selection window.

Figure 10.27: File Selection window

  The file selection dialog menu has six additional buttons specific to the Blend Algebra/Blend Program plugins. The three buttons on top left change the current directory; the three buttons on bottom left do simple file operations.
  =>Project
    display the files in the project directory (where the main project .xml file resides).
  =>Userlib
    display the files in the user library.
  =>Syslib
    display the files in the system library.
  Copy to project
    copies the currently selected function source file into the project directory.
  Copy to userlib
    copies the currently selected function source file into the user library. There is no button to copy a function into the system library because the system library is meant for functions distributed with Cinelerra, which can get overwritten on reinstallation or upgrade. For details on "copying to/from" project, userlib, or syslib, see "User library, System library, and Cinelerra Project directories" in a subsequent section.

  Edit...
    This is used in the same way as the Edit... in the main plugin window as described next.
Edit...
  Open the currently selected function source file in the configured external editor. The editor that will be used is defined by the environment variable $CIN_EDITOR, with emacs being the default if not redefined. It must be a windowed editor, like GNU Emacs (default), or KDE's Kate, or Gedit, or ancient Xedit, not a bare console editor like vi.

      		export CIN_EDITOR=kate
      		export CIN_EDITOR=gedit
      		...

      		$ ~/cinelerra5/cinelerra-5.1/bin/./cin

  If using a console editor, it must be defined to run in a terminal emulator, like xterm, or KDE's Konsole, for example:

      		export CIN_EDITOR='konsole -e vi'
      		$ ~/cinelerra5/cinelerra-5.1/bin/./cin

  This is the same Edit... button as in the file selection window of the Attach... button.
Refresh
  The external editor is started in the background so after editing and saving a function, the user can leave the editor in that window. Alternatively the user can edit a function outside of CINELERRA-GG. In order not to check modification times of all the functions on each video frame, CINELERRA-GG has to know when a function can get changed. That is what pressing Refresh is for. It indicates that now is the time to check functions, and to recompile some of them. You can expect immediate recompilation of the function which is under the timeline cursor (if any), that is if the source became newer than the shared object. But any other functions not under the cursor will not be recompiled until first access.
Detach
  This button clears the function name in the Function: input field and if there was a function in effect, it will be cleared in the tracks.
Color Space:
  Here is defined in which color space the frame pixels are to be passed to the function. The obvious choice is the default auto so that you will be working in the color space which is required by the function itself. In addition, RGB, YUV, and HSV color spaces are supported as well as the choice of project. With this choice, you will be working in the native color space of the project (as defined in Settings → Format). If the required color space does not match the native one, pixel color components will be transformed to the required space as floating point numbers. After returning from the function, pixels of the result will be transformed back to the native color space of the project.
Parallelize processing
  This checkbox, ON by default, lets you execute user function code in a parallelized (multithreading) manner, provided that the function itself supports this. It is described further on as to how the function code is executed.

###
Supplementary color selection

There may be illegal floating point operations in the user's code, such as division by zero producing infinity, or square root from minus unity producing NaN (not-a-number). Even if only legal finite floating point numbers are computed, the results may go out of range. This dialog section controls what to do in such cases.

Clip color values
  If this checkbox is on, which is the default, first the function results will be clipped to range. Clipping takes place before back transformation to the project color space. The bounds are defined as follows:

      		RGB: [0.0 .. 1.0] for R, G, B
      		YUV: [0.0 .. 1.0] for Y, [-0.5 .. +0.5] for U, V
      		HSV: [0.0 .. 360.0] for H, [0.0 .. 1.0] for S, V
      		Transparency: [0.0 .. 1.0] for Alpha

  If the project color space is RGB(A)-8 Bit or YUV(A)-8 Bit, clipping occurs unconditionally because of intrinsically restricted dynamic range of 8-bit storage per channel. For RGB(A)-FLOAT clipping can be switched off. What occurs with unclipped pixels later, depends on further destinations in the internal video processing pipeline. Moreover, displaying drivers handle unclipped colors differently. For example, an RGB pixel with unclipped red component R=-1.0 and two other components G=B=0.0 is displayed as black (that is with R clipped to zero) by the OpenGL driver, but as red (that is taken with opposite sign, R=1.0) by the X11 driver. Explicit clipping in the plugin eliminates such driver dependency.
Chroma key or substitution color:
  Even after optional clipping, there can still remain some non-finite results, like NaN. These results are bad and should be eliminated as soon as possible. If any of color components or alpha channel is NaN, the complete pixel (all components) is replaced with the configured color and the configured opacity. Substitution color is passed as additional arguments to user functions and can be used for artistic purposes, such as chroma keying.
Select key color...
  This button opens color selection dialog to select the color which will be used as substitution for NaNs or infinities in the results.
Get from color picker
  Use this button to copy the color selected earlier in the color picker tool of the CINELERRA-GG Compositor window.
Substitution opacity:
  The slider sets the opacity (alpha-channel) value to be used together with substitution color. Substitution color and opacity are passed as additional arguments to user functions and can also be used for artistic purposes.

###
Processed tracks arrangement

Track order:
  If the plugin is attached to N tracks, its user functions get pixels of the tracks numbered from 0 to N-1. This defines which of the tracks is to be the first (i.e. number 0), either top or bottom, much like in the Overlay plugin.
Output track:
  This defines into which track (top or bottom) the result will be placed, similar to the Overlay plugin. This control applies to Blend Algebra only.
Hide input tracks, use output exclusively
  If this checkbox is ON (the default), frames of the tracks the plugin is attached to are cleared and then replaced by the function result. If the checkbox is OFF, only the output track frame is replaced by the result; all the other tracks are left unchanged. By comparison, the Overlay plugin has no such option and its input tracks are always cleared as in the on state. This control applies to the Blend Algebra plugin only.

All the parameters are keyframable, including user functions themselves. That is, a plugin can have several functions attached at the same time and switch between them while rendering. The color and alpha channels of the substitution (key) color are interpolated between keyframes; the other parameters are switched.

###
User library, System library and Project directories

The user library directory is that defined by the environment variable $CIN_USERLIB, or if not defined, $HOME/.bcast5lib as a fallback default. The functions for Blend Algebra reside there in the subdirectory dlfcn/ba and for Blend Program in dlfcn/bp (dlfcn stands for dynamic loaded function). Having user functions in such a library in a single place makes it convenient to reuse the same functions in different projects. The Copy to userlib button in the Attach... file selection dialog is handy to save such functions in the user library for future use.

The reason to define user library $HOME/.bcast5lib by default (and not $HOME/.bcast5) is the following. There is a common workaround: if a newer version of CINELERRA-GG does not start, to delete $HOME/.bcast5 completely. Doing so, all user functions would be deleted if they were saved there.

The system library is defined by the environment variable $CIN_DAT, the CINELERRA-GG installation path, and under the same subdirectories dlfcn/ba or dlfcn/bp. System library is meant for functions distributed with CINELERRA-GG.

When a complete project is to be exported, or fed to a render farm, it is often necessary to place all the resources under the same directory where the project .xml file resides. If some functions are used which reside elsewhere (for example, in a user library), they could not be found after exporting the project to another computer. The Copy to project button in the Attach... file selection dialog is used to copy such functions to the project directory in a convenient manner.

###
Syntax of Blend Algebra / Blend Program functions

Blend Algebra/Blend Program functions are written in C, and with usage of a set of special cpp macros hiding from the user all the Cinelerra internals. This approach makes it easy to write functions of moderate complexity with almost no background in C programming. All macros defined for user functions are written in capital letters. A typical Blend Algebra function consists of the following logical blocks.

    	<optional static variable definitions>

    	BLEND_ALGEBRA_INIT

    	<macros and/or C statements for the INIT (preparation) phase will be executed once before processing of each frame>

    	BLEND_ALGEBRA_PROC

    	<macros and C statements for the PROC (processing) phase will be executed repeatedly for each pixel in the frame>

    	BLEND_ALGEBRA_END

The macro BLEND_ALGEBRA_STOP terminates execution of the INIT or PROC phase immediately and returns to the calling plugin.

The structure of a Blend Program function is the same, only the word ALGEBRA in all macros must be changed to PROGRAM. The following macros can be used in the INIT phase:

    	COLORSPACE_RGB
    	COLORSPACE_YUV
    	COLORSPACE_HSV

Use any of these COLORSPACE statements to declare the working color space inside the user function. If there is no such declaration, the native color space of the project (RGB or YUV) will be used.

REQUIRE_TRACKS (<n of tracks>)
Declares the minimum required number of tracks the function uses. If the effect is attached to more tracks, it is fine. If there are less tracks, it is an error and the function is not executed. The absence of such declaration means that the function can process any number of tracks (one or more).

PARALLEL_SAFE
This declaration means that it is safe to execute the function in parallel. Without this statement the function will be executed sequentially, independent of the state of the parallelization checkbox in the plugin GUI.

The following special variables (macros) can be used for queries:

PARALLEL_REQUEST
1 == parallelization requested in the GUI
0 == parallelization switched off

TOTAL_TRACKS
The actual number of tracks passed to the function.

HAS_ALPHA
1 == the project color space has alpha channel (transparency)

WIDTH, HEIGHT
The dimensions of processed frames in numbers of pixels.

The following variables (macros) can be used in the PROC phase. In addition to the next variables, TOTAL_TRACKS, HAS_ALPHA, WIDTH, HEIGHT can be used as well.

Color (and alpha) components of pixel from i-th track. Track numbering is 0-based, as standartized for C arrays. So, the first track has number 0, the last one - number TOTAL_TRACKS-1. All the color components are floating point values of the C type 'float'. Here the letters R, G, B, H, S, V, etc. are for readability. If the user has declared COLORSPACE_RGB, and then written something like H(i), it does not mean that he should get the H-component of color autoconverted to the HSV color space. Actually R, Y, H are mutually identical to the 1-st color component, so are G, U, S to the 2-nd, and B, V to the 3-rd one.

    	R(i), G(i), B(i)
    	Y(i), U(i), V(i)
    	H(i), S(i), V(i)
    	A(i)

Corresponding color components of the key (substitution) color in the working color space of the function and the opacity slider.

    	KEY_R, KEY_G, KEY_B
    	KEY_Y, KEY_U, KEY_V
    	KEY_H, KEY_S, KEY_V
    	KEY_A

Color components for the result, for Blend Algebra only. Blend Programs return no result so do not have these definitions.

    	R_OUT, G_OUT, B_OUT
    	Y_OUT, U_OUT, V_OUT
    	H_OUT, S_OUT, V_OUT
    	A_OUT

Integer X and Y coordinates of the current pixel in its frame, in the ranges of [0 .. WIDTH] and [0 .. HEIGHT], respectively.

    	PIX_X, PIX_Y

These macros clip color components of the i-th track to the bounds appropriate in the respective color space.

    	CLIP_RGB(i)
    	CLIP_YUV(i)
    	CLIP_HSV(i)
    	CLIP_RGBA(i)
    	CLIP_YUVA(i)
    	CLIP_HSVA(i)
    	CLIP_A(i)

Clip all color components (including alpha) for all tracks.

    	CLIP_RGB_ALL
    	CLIP_YUV_ALL
    	CLIP_HSV_ALL

Like clipping track color components, but for the result of a Blend Algebra function (not relevant for Blend Programs).

    	CLIP_RGB_OUT
    	CLIP_YUV_OUT
    	CLIP_HSV_OUT
    	CLIP_RGBA_OUT
    	CLIP_YUVA_OUT
    	CLIP_HSVA_OUT
    	CLIP_A_OUT

Clipping is done according to the inherent bounds of the respective color space as follows:

    	RGB: [0.0 .. 1.0] for R, G, B
    	YUV: [0.0 .. 1.0] for Y, [-0.5 .. +0.5] for U, V
    	HSV: [0.0 .. 360.0] for H, [0.0 .. 1.0] for S, V
    	Transparency: [0.0 .. 1.0] for Alpha

H color channel of HSV is brought into range by repeated rotation around 360 degrees; all the others by simple clipping.

ABS(x), SQR(x), MAX(x,y), MIN(x,y) absolute value, square, max and min values for floating point arguments. Can be used in any phase of a function.

TO_RAD(x), TO_DEG(x) conversion from degrees to radians and vice versa.

Both macros clip the 'x' argument to bounds between y and z. CLIP returns value leaving x unchanged. CLAMP assigns that value to x.

    	CLIP(x,y,z)
    	CLAMP(x,y,z)

All the macros are defined in the text header $CIN_DAT/dlfcn/BlendAlgebraStart for Blend Algebra and the analogous header BlendProgramStart for Blend Program in the same dlfcn subdirectory of the Cinelerra installation directory. These headers are prepended to the user functions during compilation. The user with a basic C knowledge can easily understand what happens in these headers.

Of course, you can use any valid C statements inside user functions. As they are linked with -lm, the standard C math library and with glibc, almost any mathematical or C library function can be called. C-style comments can also be used, and are welcome.

###
The compilation script

Compilation of Blend Algebra and Blend Program functions is carried out by a Perl script BlendAlgebraCompile.pl and BlendProgramCompile.pl. Both scripts are distributed in the dlfcn subdirectory of $CIN_DAT, the Cinelerra installation directory. Like the famous ContextManual.pl script of Context Help, the system script at the first call is copied into $CIN_CONFIG ($HOME/.bcast5) where it can be edited by the user and adapted according to their needs.

While a regular compilation, the script works in that directory where the function source file resides. First, the BlendAlgebraStart/BlendProgramStart header is prepended to the source to produce an intermediate file with the .c suffix. This C source is compiled by the system C compiler (gcc by default) and linked with -lm to produce the shared object ready for dynamic loading (attaching to the plugin).

Usually the script is executed by the respective CINELERRA-GG plugin when a user function is to be compiled. But the script can also be run by the user explicitly. Execution can be controlled by options starting with the '-' character, the first argument not starting with '-' being the function name to compile. The following options are recognized:

-compile do not check modification time, compile unconditional
-cfile do not remove intermediate .c file
-opt add optimizing options to compiler command line
-debug add debugging options to compiler command line
-warn add warning options to compiler command line
-edit open function source in text editor
-verbose verbose execution
-noapi do not copy itself to $HOME/.bcast5
-h, -help, -? print short help and current configuration

###
Environment variables influencing plugin configuration

In the beginning of the script some variables can be redefined according to the user's needs: the C compiler executable, compiler options for optimization or debugging, and text editor executable. These environment variables are taken into account in various places in Blend Algebra/Blend Program plugins.

$CIN_CC: C compiler executable (default: gcc)
$CC: C compiler executable if $CIN_CC is not defined (default: gcc)
$CIN_EDITOR: text editor (default: emacs)
$CIN_DAT (set by Cinelerra binary): Cinelerra installation directory. The Blend Algebra / Blend Program system library with scripts, headers, and functions distributed with Cinelerra resides in the subdirectory $CIN_DAT/dlfcn.
$CIN_CONFIG (set by Cinelerra binary): user's config directory, usually
$HOME/.bcast5.
$CIN_USERLIB: user's library (default: $HOME/.bcast5lib). User's functions reside in the subdirectory $CIN_USERLIB/dlfcn.

###
What comes in the plugins

In the distribution 30 Blend Algebra functions are provided for all the 30 CINELERRA-GG overlay modes according to the formula defined in the CinelerraGG manual. In principle, they are not needed as the built in overlayer engine is OpenGL accelerated and runs faster. Nevertheless, these functions can be handy as starting examples for users who may like to modify the formula in some way. Furthermore, their behavior differs from that of the standard overlayer in several aspects:

- Overlayer produces different results in RGB and YUV color spaces; Blend Algebra yields identical results independently of color space of the project.
- In RGB(A) FLOAT color space, overlayer results can get out of range and then be rendered differently depending on the display driver used. Blend Algebra clipping, when switched on, can prevent this undesirable effect.
- Overlayer always clears its input frames. In Blend Algebra the user has a choice, either to clear them, or to leave them intact.

There are several example functions from totally different areas of application: a user programmable plugin is a really universal tool and can be handy in unexpected cases. A programmer can sometimes make use of this plugin when creating new effects. It has the big advantage that you can more quickly test different combinations without spending time to restart CINELERRA-GG, recompile it, reload project, and reattach plugins, etc. See the examples here.

There is the ydiff.ba function, actually representing the complete ydiff application running inside a plugin. In addition, there is a Transition example, and 2 more function examples showing use of the Blend Programs: the chromakey.bp and background.bp functions. They too serve as excellent starting points for further user experimentation. The usage of the provided functions is either self evident or briefly described in the comments of their source files.

NOTE: make install copies the functions into Cinelerra's bin directory in source (*.ba, *.bp) and binary (*.so) forms via cp -a, to preserve modification times of the files. Otherwise, if modification time of the function source accidentally becomes newer than that of its binary, the plugin will repeatedly try to recompile the function. If they are installed system-wide so that the user has no permission to modify them, such recompilations will fail.

###
Caveats

User's functions can have bugs. Most probable bugs in a user function can be illegal arithmetic such as division by zero or logarithm of a negative number (FPE, floating exception), and out of bounds indexation of arrays (usually leading to SEGV). Most modern processors do not generate FPE. Instead, they generate NaN as the result of an illegal arithmetic instruction. After calling user functions, the plugins test the results not to be NaN and replace them with the configured substitution color if necessary.

SEGV is more problematic. If the user has written in his function, let's say, REQUIRE _TRACKS(2), and then accessed R(1000000), most likely the Cinelerra binary will crash. Although theoretically it could be possible to trap SEGV via sigaction()/setjmp() /longjmp(), in Cinelerra it is problematic. In a multithreaded application, which Cinelerra is, only one signal handler for all its threads at the same time is allowed, and Cinelerra already has a SEGV handler which can not be overwritten. So the user is responsible for his indexation bugs, which can easily occur in off-by-one errors, like referencing elements such as R(TOTAL_TRACKS) (the last legal element being R(TOTAL_TRACKS-1)), or R(i) where i has been decremented to -1.

The user is allowed to call any C function from the -lm and glibc libraries. But some functions are not thread safe (which means PARALLEL_SAFE macro should not be used) and some others have undesired side effects. For example, should the user write inside his function something like exit(R(0));, Cinelerra would immediately halt. The user is left to imagine what would happen after the following expressione:

    	R_OUT = R (system ("killall -9 X"));

###
Hardware acceleration

Blend Algebra/Blend Program functions are not hardware accelerated which means that the function algorithm is not programmed as an OpenGL shader. However, functions can be parallelized using the load balancing engine of Cinelerra, and can be compiled with optimization (this is switched on by default). The default optimizing option (defined in BlendAlgebraCompile.pl and BlendProgramCompile.pl) is -O2. One can redefine it to -O3 or 0fast. It has to be noted that using 0fast or -ffast-math options can lead to ignoring some IEEE rules for floating point math, for example some intrinsic tests on infinities or NaN which can yield unpredictable results.

###
Debugging

Although user's function can be compiled with debugging option such as -g or -ggdb, debugging functions can be tricky. The programmer can set a breakpoint into the function only after its code has been loaded in memory by the plugin, and set another breakpoint into plugin only after that plugin itself has been attached to a track. Moreover, a function can get detached from memory under some conditions, and then its breakpoints will be lost.

Debugging printout inside the PROC phase, although possible, would seem too verbose: the PROC phase would be called for full-HD footage 2073600 times per frame! Here perhaps one could make use of PIX_X, PIX_Y coordinates at some selected place like in the example:

    	if (PIX_X == 320 && PIX_Y == 200)

###
Portability

The current implementation should be portable as long as you have a working C compiler such as gcc, the default to compile CINELERRA-GG, or clang which has been known to work; but other compilers may be compatible as well. Portability to other architectures that are Unix-like and ELF based should work, but not Windows.

NOTE: For developers who want to take advantage of creating a plugin to generate an effect in the same manner as Blend Algebra/Blend Program, please reference Unique Blend plugins workflow. This methodology makes it easier for the programmer to more quickly test different combinations without spending time to restart CINELERRA-GG, recompile it, reload your project, reattach plugins, and so on.

#### Footnotes

... plugins 6
  credit to Georgy