The stencil buffer is like the depth and color buffers, except stencil pixels don't represent colors or depths, but have application-specific meanings. The stencil buffer isn't directly visible like the color buffer, but the bits in the stencil planes form an unsigned integer that affects and is updated by drawing commands, through the stencil function and the stencil operations. The stencil function controls whether a fragment is discarded or not by the stencil test, and the stencil operation determines how the stencil planes are updated as a result of that test [43].
Stencil buffer actions are part of OpenGL's fragment operations. Stencil testing occurs immediately after the alpha test, and immediately before the depth test. If GL_STENCIL_TEST is enabled, and stencil planes are available, the application can control what happens under three different scenarios:
Whether a stencil operation for a given fragment passes or fails has nothing to do with the color or depth value of the fragment. The stencil operation is a comparison between the value in the stencil buffer for the fragment's destination pixel and the stencil reference value. A mask is bitwise AND-ed with the value in the stencil planes and with the reference value before the comparison is applied. The reference value, the comparison function, and the comparison mask are set by glStencilFunc(). The comparison functions available are listed in Table 4.
Stencil function and stencil test are often used interchangeably in these notes, but the ``stencil test'' specifically means the application of the stencil function in conjunction with the stencil mask.
| Comparison | Description of comparison test between reference and stencil value |
| GL_NEVER | always fails |
| GL_ALWAYS | always passes |
| GL_LESS | passes if reference value is less than stencil buffer |
| GL_LEQUAL | passes if reference value is less than or equal to stencil buffer |
| GL_EQUAL | passes if reference value is equal to stencil buffer |
| GL_GEQUAL | passes if reference value is greater than or equal to stencil buffer |
| GL_GREATER | passes if reference value is greater than stencil buffer |
| GL_NOTEQUAL | passes if reference value is not equal to stencil buffer |
If the stencil test fails, the fragment is discarded (the color and depth values for that pixel remain unchanged) and the stencil operation associated with the stencil test failing is applied to that stencil value. If the stencil test passes, then the depth test is applied. If the depth test passes (or if depth testing is disabled or if the visual does not have a depth buffer), the fragment continues on through the pixel pipeline, and the stencil operation corresponding to both stencil and depth passing is applied to the stencil value for that pixel. If the depth test fails, the stencil operation set for stencil passing but depth failing is applied to the pixel's stencil value.
Thus, the stencil test controls which fragments continue towards the framebuffer, and the stencil operation controls how the stencil buffer is updated by the results of both the stencil test and the depth test.
The stencil operations available are described in Table 5.
| Stencil Operation | Results of Operation on Stencil Values |
| GL_KEEP | stencil value unchanged |
| GL_ZERO | stencil value set to zero |
| GL_REPLACE | stencil value replaced by stencil reference value |
| GL_INCR | stencil value incremented |
| GL_DECR | stencil value decremented |
| GL_INVERT | stencil value bitwise inverted |
The glStencilOp() call sets the stencil operations for all three stencil test results: stencil fail, stencil pass/depth buffer fail, and stencil pass/depth buffer pass.
Writes to the stencil buffer can be disabled and enabled per bit by glStencilMask(). This allows an application to apply stencil tests without the results affecting the stencil values. Keep in mind, however, that the GL_INCR and GL_DECR operations operate on each stencil value as a whole, and may not operate as expected when the stencil mask is not all ones. Stencil writes can also be disabled by calling glStencilOpGL_KEEP, GL_KEEP, GL_KEEP(GL_KEEP, GL_KEEP, GL_KEEP).
There are three other important ways of controlling and accessing the stencil buffer. Every stencil value in the buffer can be set to a desired value by calling glClearStencil() and glClearGL_STENCIL_BUFFER_BIT(GL_STENCIL_BUFFER_BIT). The contents of the stencil buffer can be read into system memory using glReadPixels() with the format parameter set to GL_STENCIL_INDEX. The contents of the stencil buffer can also be set using glDrawPixels().
Different machines support different numbers of stencil bits per pixel. Use glGetIntegervGL_STENCIL_BITS, ...(GL_STENCIL_BITS, ...) to see how many bits are available. If multiple stencil bits are available, glStencilMask(a)nd the mask argument to glStencilFunc() can be used to divide up the stencil buffer into a number of different sections. This allows the application to store separate stencil values per pixel within the same stencil buffer.
The following sections describe how to use the stencil buffer in a number of useful multipass rendering techniques.