Maya and Mental Ray

Maya's interface to Mental Ray is -- for the most part -- straightforward and extensive. You construct your scene as you would for the usual Maya rendering, and the interface would take care of translating everything (including the shaders) to its MR equivalent. The rest of this document will describe the finer points of using this interface.

Deploying MR

Open the plug-in manager and load mayatomr.mll. Then open the render globals and switch the render using parameter to mental ray. Afterwards you can just render as usual.

To render a scene in command line window, type

mayarender_with_mr scene

where scene is the scene file name.

You can also save out the scene as a mi file and render using the ray command: simply run file > export all (options) and set the export file type to mentalRay. If you set file format to binary, data such as point co-ordinates will be written out in a compact binary format, resulting in smaller file size. The tabulator size next to the ASCII item refers to the number of spaces in each line indentation -- an unimportant parameter if you don't plan on reading the output MI. You can choose to write out the scene in every frame to a separate MI file. This could be useful if you plan to render the sequence in different frame blocks, but take note that per frame MI output is not incrementally changed, meaning the total file size will be larger than a unified multi-frame MI output. The includes startup file option specifies whether to $include [mayatomr installation path]/maya.rayrc.

MR render globals

Most of the parameters in there are pretty straightforward if you know the MI format, and you should leave the esoteric ones alone if you don't plan to use your own custom-written shaders. If you do plan to use such shaders, read on.

In the translation > performance section, the export derivatives parameters specify whether to calculate bump basis vectors for the surfaces. These vectors are stored in the state arrays bump_x_list and bump_y_list.

In the translation > customization section, turn off state shader, light linker and Maya options if you don't plan to use the mayabase shader library.

In the custom entities > custom globals section, you can set the renderer's version range (as two numbers), the (space-separated) shader libraries to link to and the (again space-separated) shader declaration files to include. These information will be written to the beginning of the MI file.

In the custom entities > custom scene section, you can create a special node called mentalrayText to specify what's to be written at either the beginning of the scene, the options block, the light definition region, the camera definition, or the object instantiation region. To create a mentalrayText node, just hit the mapping button next to one of the labels, type in a block of sensible text, and hit the apply button. With the custom text mode chooser, you can specify whether to fill the corresponding MI region with just your text block (replace), insert your text block at the beginning of the existing output (preface) or append the text block to the end of the existing output (append).

Surface approximation

To control the tessellation of a surface precisely, select the surface and open window > rendering editors > mental ray > approximation editor. By default, the plug-in will try to match Maya's native tessellation settings in the shape node. If you click the create button, a special surface approximation node will be attached to the selected surface, containing all the tessellation parameters that you've come to know and love from MR. You can use the same editor to edit or reassign surface approximation nodes.

Displacement allowance

As you may recall, displaced surfaces need a max displace parameter in their definitions. In Maya, you set this parameter in the concerned shape's attribute editor, in the displacement map > bounding box scale attribute. Trouble is, this attribute is a bounding box scaling factor, so you would have to crank you brain a bit to get the desired max displace value. For example, given an object with a bounding box size of 10 x 12 x 20, if you set the bounding box scale to (1.3, 1.4, 1.0), the bbox will be expanded by (3, 4.8, 0), and the max displace value in the resulting MI will be the largest of the three values, 4.8. So you would usually keep the y and z bbox scaling values at 1, and use only the x scaling to calculate the final max displace value.

Custom shaders

You would usually work with Maya's native shaders in the Hypershade, and when it's time to render, the MR interface plugin would quietly translate them to MR shaders. But sometimes you may want to write a custom shader and assign it directly to your objects. There're two ways to go about this. The first way is to use a custom text editor:

  1. Define the environment variable MI_LIBRARY_PATH and set it to the directory paths of base.dll and customs.dll. Define also the variable MI_RAY_INCPATH and set it to the paths of base.mi and customs.mi.

  2. In Maya, make sure the mayatomr.mll plugin is loaded.

  3. First create a 2 x 6 x 2 polygon cube and translate it to (0, 3, 0). Then create a 20 x 20 polygon plane. Also create a pointlight (don't worry about the parameters) and translate it to (-4, 8, 4).

  4. Open the render globals and switch the renderer to Mental Ray. In the mental ray tab, under the translation > customization section, untick export state shader, export light linker and export Maya options; also, under the custom entities > custom globals section, type base.dll customs.dll (note the space between the names) into the links field, and type base.mi customs.mi into the includes field.

  5. Create a Blinn material and assign it to the cube. Select the Blinn node and run window > rendering editors > mental ray > custom text editor. Hit the mapping button next to the custom shader text label, type "plaincolor" ("color" 0 0.4 1, "brightness" 1) into the mi text field, and hit the apply button. Leave the custom text mode at replace.

  6. Create another Blinn material and map any of its parameters to a checker texture (normal mode, without texture placement). Map any of the checker's parameters to a ramp texture (again normal mode and without texture placement). Create custom texts for all three shading nodes like you did in the previous step, typing in respectively

    # ramp
    "mib_illum_lambert" (
      "diffuse" 0.45 0.6 0.9,
      "lights" ["pointLight1"]
    # checker
    "blureflect" ("shiny" 20)
    # blinn
    "simplemixer" (
      "input1" = "ramp1",
      "input2" = "checker1",
      "factor1" 1,
      "factor2" 0.4
  7. Create a custom text for the pointlight shape as well. Type in

    "mib_light_point" (
      "color" 1 1 1,
      "shadow" true,
      "factor" 0.5,
      "atten" true,
      "start" 0,
      "stop" 50
  8. Run file > export all (options). Make sure the export file type is mentalRay. Save the scene and quit Maya.

  9. Render the exported file with ray.

You may want to examine the exported file with a text editor, to relate its content to the procedure above.

All these text juggling could be fun at first, but it's bound to tug on your nerves sooner or later. We need a better procedure.

  1. Create an environment variable called MAYA_MRFM_SHOW_CUSTOM_SHADERS and set its value to 1.

  2. Copy customs.dll to [MRforMaya's installation path]\lib, and customs.mi to [the same path]\include. Open [the same path]\maya.rayrc with Wordpad and add the following highlighted lines to it:

    registry "{MRMAYA_START}"
      link "{MAYABASE}/lib/base.{DSO}"
      link "{MAYABASE}/lib/physics.{DSO}"
      link "{MAYABASE}/lib/mayabase.{DSO}"
      link "{MAYABASE}/lib/contour.{DSO}"
      link "{MAYABASE}/lib/customs.{DSO}"
      mi "{MAYABASE}/include/mayabase.mi"
      mi "{MAYABASE}/include/base.mi"
      mi "{MAYABASE}/include/physics.mi"
      mi "{MAYABASE}/include/contour.mi"
      mi "{MAYABASE}/include/customs.mi"
      echo "mental ray for Maya - startup done"
    end registry
  3. Use a painting program to create seven 32x32 images of any common format (bmp will do nicely). Make sure you name them properly (no spaces or weird characters; must have file extension), and store them in %MAYA_APP_DIR%\5.0\prefs\icons. Use the imconvert command to convert the images to xpm format. The command's syntax is

    imconvert [input image] render_[shader name].xpm

    For example, you could make an image for the plaincolor shader by running

    imconvert plaincolor.bmp render_plaincolor.xpm

  4. Open the scene you saved in the previous procedure. Use the quick selection field to select mentalrayText* and delete them.

  5. Pick the cube's Blinn material, navigate to its shading group and open its attribute editor. Expand the mental ray section and map its material shader to plaincolor (under the materials section, you should recognize its icon). Set the plaincolor node's color to bright green and brightness to 1.

  6. Open the attribute editor on the ground plane's Blinn shading group. Map its mental ray material shader to simplemixer (under miscellaneous section). Set the simplemixer's factor1 parameter to 1 and factor2 parameter to 0.4. Then map its input1 to a mib_illum_lambert node (under materials), and its input2 to a blureflect node (under miscellaneous). Set the diffuse color of the mib_illum_lambert to sky blue, and the shiny factor of the blureflect to 20.

  7. Open the attribute editor on the pointlight's shape, expand the mental ray section, and map its light shader to a mib_light_point node (under lights). Set the created node's color to white, shadow to on, factor to 0.5, atten to on, start to 0, and stop to 50.

  8. Now we need to manually link the light to the ground's lambert material. Execute the following MEL command:

    connectAttr pointLightShape1.message mib_illum_lambert1.lights[0]

    Obviously mib_illum_lambert1.lights is an array attribute; you can connect more lights to its subsequent elements.

  9. Test render.

The plugin uses the apply flags in the shader declarations to categorize the shaders. If no apply flag is specified, the shader is put under the miscellaneous section.

So, you may wonder, why have the custom text thingy at all? First of all, take note that custom text can be attached to nodes other than shading nodes, and you have the choice to not only replace the MI output content but also to append or "prepend" stuff. This can be a powerful tool for output customization, especially when you have to deal with MR capabilities that the plugin isn't savvy to.

Shading groups have a export with shading engine and a export with volume sampler buttons that you have to tick on if you plan to use your customs shaders with Maya's built-in shaders.

Exercise: attach a panoramic lens shader to the camera.

Parameter hints

In the MI shader declaration, you can insert one or more of the following hints to instruct the plugin to tailor the generated shader UI to your requirements:

#: default scalar [scalar [scalar]]
#: min scalar [scalar [scalar]]
#: max scalar [scalar [scalar]]

For example,

declare shader
  color "plaincolor" (
    color "color",
    #: default 1.0 0.0 0.0
    #: min 0.0 0.0 0.0
    #: max 1.0 1.0 1.0
    scalar "brightness"
    #: default 1.0
    #: min 0.0
    #: max 1.0
  version 1
  apply material
end declare

Area light

You can turn any pointlight or spotlight into an area light by ticking on its area light parameter (located in the light's attribute editor, under mental ray > area light). You can also make the light's area shape visible by ticking on its visible parameter.

And here we have a prime example of the usefulness of custom text -- the current version of mayatomr doesn't support MR 3.1+'s geometric area light; to create such lights, we have to resort to custom text. Try this:

  1. Open the scene file lightbulb.mb.

  2. In render globals > custom entities > custom scene, create a custom global text containing a single line of text,

    $include "D:\mentalray\mis\lightbulb.mi"

    That path was just an example; you should set it to the location of your courseware copy. Remember to click the apply when you're done typing that line. Leave the custom text mode at append.

  3. Select the light shape and run rendering editors > mental ray > custom text editor. Hit the mapping button and type the following line into the custom text field:

    object "lightbulb" 16 16

    Change the custom text mode to append and hit apply.

  4. Export the scene to a MI file.

  5. Go to the Maya-MR software folder. Copy include\mayabase.mi and lib\mayabase.dll to the folder where you put the exported MI.

  6. Render the MI with ray.

Subdivision surface

You can prepare subd surface either at model time or at render time. Model-time subd is simple -- just create and work with the usual subd models, and the plug-in will render them as they are. Note that only cages with quad faces can be rendered, and creases must be full (not partial).

MR will recognize the subd model's texture co-ordinates, but only in the base mesh.

Render-time subd is useful if you don't want to lug a heavy subd model around while you're working, or if you have triangle faces; you create it by opening window > rendering editors > mental ray > approximation editor on a polygon model, and then clicking the create button next to the subdivision approx label. The subd cage must be composed entirely of quads or triangles (not a mixture of both).


MR 3.1 (and above) supports a special hair geometry, but the mayatomr plugin doesn't support it currently.