Any material on this topic, perhaps tutorials?

PS. Finally, we have a forum! Thank you very much!

Hi JAre. I am very interested in this topic too. I have not yet played with Broomstick regarding this topic, but I have experimented a lot with PB3D in my own practice engine called Minimole. I know Broomstick has an extremely flexible AGAL shader system but its a bit difficult to extend or customize. My plan is to study it in detail and see how it could be adapted to make simple custom shaders or even use PB3D.

I had the exact same questions when I started using Broomstick in March. After deep-diving into the code base I realized the easiest way to do custom shaders was to extend class WrapDiffuseMethod.

In Broomstick, shaders are called methods and are found in package away3d.materials.methods. Diffuse methods are your best friends, offering possibilities to place both your vertex and fragment shader codes.

Shader code still has to be plain text AGAL op-code, which is retrieved from methods

arcane override function getVertexCode(regCache : ShaderRegisterCache) : String;
override arcane function getFragmentAGALPreLightingCode(regCache : ShaderRegisterCache) : String;
override arcane function getFragmentCodePerLight(lightIndex : int, lightDirReg : ShaderRegisterElement, lightColReg : ShaderRegisterElement, regCache : ShaderRegisterCache) : String;
override arcane function getFragmentPostLightingCode(regCache : ShaderRegisterCache, targetReg : ShaderRegisterElement) : String;

when Broomstick initializes all shader programs. However, Away3D offers a nice helper class called away3d.materials.utils.AGAL, with static functions like add, sub, mul, mov, m44 and so on that make it much easier to assemble AGAL code by concatenating their String return values rather than by purely typing it by hand.

Those four methods have some documentation added to them:
getVertexCode: Get the vertex shader code for this method.
getFragmentAGALPreLightingCode: Get the fragment shader code that will be needed before any per-light code is added.
getFragmentCodePerLight: Get the fragment shader code that will generate the code relevant to a single light.
getFragmentPostLightingCode: Get the fragment shader code that should be added after all per-light code. Usually composits everything to the target register.

For me that was enough to know that my first few custom shaders would simply return custom code in method getFragmentPostLightingCode, since I didn’t want to fiddle around with lighting code anyway.

Well, that’s good to know, but what’s Broomstick’s magic rendering pipeline doing in the background?

Broomstick, when rendering geometry and applying material to it, initialized all associated shader programs prior to the actual rendering call.
What it does is calling the following methods in sequence:

[...] // all of the before mentioned get*Code methods
override arcane function activate(context:Context3D, contextIndex:uint):void;
arcane override function deactivate(context:Context3D):void;

activate is the place to initialize your shader registers and upload constant data, as well as setting texture registers.
In deactivate you need to un-register any textures set in activate.

Registers? Constant Data? What?
Maybe I should go back one step and explain how data is served to shader programs. When assembling any shader code that needs any pre-calculated data (like matrices or weights), they need to be loaded into available and appropriate registers prior to sending the shader to work so the GPU knows what to feed it with. This is done using Stage3D’s methods like Context3D.setProgramConstantsFromVector or Context3D.setTextureAt and is done in the activate method.

However, all those methods need a register index to know into which register to load the data. This index you can retrieve in the get*Code methods using the method’s parameter regCache : ShaderRegisterCache object. It offers methods like getFreeFragmentConstant() or getFreeTextureReg(), which all return objects of type ShaderRegisterElement. These shader register elements each have a property index, which is exactly the index that you need to load data into the registers.

So, the order of things you need to do to feed your shader with data is as follows:
1. request a ShaderRegisterElement during any get*Code method of your diffuse shader.
2. store it’s index property to a field variable.
3. use the ShaderRegisterElement‘s toString() method in your AGAL code (e.g.

AGAL.add(targetReg + ".xyz", targetReg + ".xyz", tempReg + ".xyz");

).
4. use the stored index in the activate method with Context3D’s register setter methods.

What didn’t I mention yet? Ow, yes, where to actually use your custom diffuse method =)

Diffuse methods can be set to any material extending class away3d.materials.DefaultMaterialBase. The most useful ones are ColorMaterial and BitmapMaterial. First one is a simple color fill, second a texture. When setting property diffuseMethod with your own custom diffuse method, you successfully applied your first custom shader program to a material used to pixel-fill your geometry :D

So, as I’ve already worked a bit with custom shaders, I don’t really have to think hard about this pipeline anymore. However, it may also make some things seem just all too obvious to me that would need more explanation here. If so, let me know.

Cheers!
Alex

So you can not use PB3D without rewriting half of engine? Materials should be implemented as an set of plugins with good flexibility :(

Your information was very helpful - thank you.

excellent break down of how Away3D puts the rubber to the road in AGAL, Alex. I’ve followed the huge kirupa thread on molehill with a lot of interest. My understanding is at the moment there’s no real advantage to using PB3D. It doesn’t simplify things much over using AGAL alone and is reportedly slower. Another thing I see as a big concern with molehill is there’s no looping implemented in either AGAL or PB3D which makes it really difficult to do things like vertex skinning. Adobe may or may not implement them which must be a real conundrum for the Away3D devs as to how to keep all this code they are pumping out flexible enough to incorporate looping shaders quickly if they do become available.