MouseEnabled

  So, how is the engine doing the picking? Basically, the engine is casting a ray from the camera, through the mouse position on the screen, and straight into the scene. This ray could potentially hit several objects within the scene. One way to determine which objects are hit is using raw ray collision mathematics. This involves calculating ray-sphere collisions, ray-aabb collisions, ray-triangle collisions, etc. The most expensive part of this process is finding out if the ray hits the inside of a triangle in a mesh, which could potentially contain a very high poly count. Now, ray tracing is not the only method used for picking. Away3D in fact has 2 different methods and uses this one by default for reasons we will discuss further along in this tutorial. Figure 1 below shows the basics of the ray tracing approach.

  When you set mouseEnabled to true on any object, you make it eligible for these collision evaluations. This property is set to false by default on all objects in order to avoid potentially complex calculations.

  Now, you can have an object mouse enabled but with no listeners attached to it. The result of this is simply occlusion. The object collides with the ray, but triggers no mouse events. You can see an example of this on listing 2, which is just the same as listing 1, but simply comments out the attachment of listeners to one of the objects.

  If the occluding object was not mouse enabled, the picking ray would just pass through it, ignoring it completely. Again, this is the default behaviour for all scene objects unless you specifically set the mouseEnabled property to true. This is pretty much the way it works on the regular Flash 2D display API, but in that case objects are mouse enabled by default because picking is so much cheaper in two dimensions.

Picking Colliders

  You may have noticed on the previous listings that ( after rotating the camera to very specific angles ) picking around the sphere is not very accurate. In listing 1 or 2, if you hover the mouse close enough to the sphere, it will trigger a mouse over, even though the mouse is not actually over the object. Hmmm… not good.

  The engine set’s the picking precision for all objects to BOUNDS_ONLY by default. If you have a look at figure 1 again, what’s happening is that the ray is being “stopped” at the mesh’s bounding volume surface and collision calculations with its potentially hundreds or thousands or millions of triangles is avoided by default. If you do need more precision, you just need to tell the ray to keep going. You change this using the following property:

anObject.pickingCollider = PickingColliderType.BOUNDS_ONLY; // default
//anObject.pickingCollider = PickingColliderType.AS3_FIRST_ENCOUNTERED;
//anObject.pickingCollider = PickingColliderType.AS3_BEST_HIT;
// etc…
    

  BOUNDS_ONLY is the cheapest picking collider that the engine offers and is accurate enough for more cases. If you want more precision, you just chose a different one:

 
•   PickingColliderType.BOUNDS_ONLY ( default value )  
    Calculates ray vs bounding volume collisions.  
 
•   PickingColliderType.AS3_FIRST_ENCOUNTERED  
    Calculates ray vs all mesh triangles collision. This costs considerably more than BOUNDS_ONLY (1) and is heavier as the number of triangles in the mesh increases. The collider stops all tests as soon as a collision with a triangle is found, despite the fact that the hit surface may not be the mesh’s closest surface along the ray to the camera.  
 
•   PickingColliderType.AS3_BEST_HIT  
    As the previous collider, but more detailed, evaluating which of the triangles that collide with the ray is closest to the ray’s origin. This is necessary if you want to accurately know the collision point on a mesh that may have more than one possible collision surface with the ray, such as a mesh representing a coffee cup, for example. AS3_FIRST_ENCOUNTERED (2) could incorrectly resolve a collision with the inner walls of the cup whereas this collider will not stop tests on the first hit, but will instead find all collisions and evaluate the best one overall.  
 
•   PickingColliderType.PB_FIRST_ENCOUNTERED  
    This is the same as AS3_FIRST_ENCOUNTERED (2) but uses pixel bender instead of pure Actionscript for ray collision calculations. This method proves to be faster for higher-poly meshes and slower for lower-poly meshes. Pixel bender takes advantage of multi-threading if it can, meaning that such a collider will tend to be faster on desktops, but may not be on simpler CPU’s like those found on mobile devices. Furthermore, AIR for iOS simply does not support pixel bender at this time, which causes this collider to fail silently.  
 
•   PickingColliderType.PB_BEST_HIT  
    Pixel bender equivalent to AS3_BEST_HIT (3).  
 
•   PickingColliderType.AUTO_FIRST_ENCOUNTERED  
    As AS3_FIRST_ENCOUNTERED (2), but automatically decides wether to use the actionscript or pixel bender version of the collider depending on the poly count of the mesh.  
 
•   PickingColliderType.AUTO_BEST_HIT  
    Same as AUTO_FIRST_ENCOUNTERED (6) but with the behaviour of AS3_BEST_HIT (3).  
 

  As you can see, there’s a lot of choices! Deciding which picking collider to use is a delicate topic, and it is important to understand what each available type does in order to choose the best balance between performance and precision. For instance, you would choose PB_BEST_HIT (5) on an application that paints on a high poly mesh where you want very accurate picking precision, but would definitely not choose such a collider for a game where you need to be able to click on a low poly mesh that occupies a small portion of the screen. Such a poor choice would be a waste of resources, specially if you enable such type of mouse interactivity on many of such low poly objects! Deciding which collider type to use is very important, but an easy task once you understand the basics of each collider type. Hence, to solve our problem on figure 2, we could just set a different collider type on the object:

sphere.pickingCollider = PickingColliderType.AS3_FIRST_ENCOUNTERED;

  Or, we could simply change the sphere’s bounding volume to a BoundingSphere instead of an AxisAlignedBoundingBox (Away3D sets all bounding volumes to AxisAlignedBoundingBox by default). This would allow us to get the precision we want without the need to enter the volume and perform costly ray triangle collision calculations. Of course, this is a very specific, almost theoretical case, but its important to keep in mind that the core of this picking technique is bounding volumes, and different bounding volume shapes can be very helpful. In code, this is how you change a bounding volume:

sphere.bounds = new BoundingSphere();

  Either way we choose to resolve our precision problem, the result can be appreciated in listing 3:

Shader vs. Raycast Mouse Pickers

  As mentioned before, Away3D offers a completely different approach for picking calculations. So far we have been dealing with the properties of individual objects, you can change the way picking works on a global scale too.

view.mousePicker = PickingType.SHADER;

  You can choose from the following set of values:

 
•   PickingType.RAYCAST_FIRST_ENCOUNTERED ( default value )  
    Uses ray tracing as a picking method, stopping tests at the first successful renderable level collision.  
 
•   PickingType.RAYCAST_BEST_HIT  
    Uses ray tracing as a picking method, evaluating the best ( closest ) hit between all the colliding renderables.  
 
•   PickingType.SHADER  
    Uses a shader technique as a picking method, always evaluating the best hit.  
 

  The difference between first 2 options is quite subtle, lets first see the difference between the first 2 and the 3rd option, which represents a completely different picking technique from the others. As explained before, the first two are based on evaluating collisions between a ray and mathematical entities within the scene. The 3rd option ( SHADER ) does not evaluate ray-geometry collisions at all, but instead renders a portion of the scene around the mouse with specific colors into a temporary buffer, and then analyzes the color directly under the mouse to find which object is hit. This technique is harder to visualize, and is in fact what most high end 3D engines use, proving to be very efficient both in accuracy and performance. Unfortunately, this technique is not as performant in Flash’s Stage3D at this time.This is because it involves transferring images from the GPU to the CPU via Context3D’s drawToBitmapData() method, which is slow. You can read about this limitation in this article. The method introduces a severe bottleneck which is the sole reason why Away3D offers a raycast alternative and sets it as the default method. RAYCAST proves to be faster than the SHADER method in Flash. Keep in mind that you can only have one of the two methods per view, but not both, i.e. for the moment, you cannot have shader based picking on some objects and raycast based on other objects.
 
  You may be asking yourself “why does Away3D still offer shader based picking if it is proved to be slower?”. Because it is still necessary in some situations. For example, when dealing with GPU animations and transformations, where geometry can be altered after the CPU stage, in the GPU, where the raycast approach could not possibly know where the vertices are, and will not be able to properly calculate ray-geometry collisions. Since the shader method is not ray-geometry collision based and deals with what is actually visible on screen, it is the method of choice when dealing with animated objects where high picking precision is desired. Listing 4 shows the effect of using ray versus shader based picking on an animated object. In this demo, we will also be using a sphere to trace where the mouse ray hits the mesh, as well as a line segment to trace the mesh’s normal at such point.

  Notice how the raycast method does not see the GPU level vertex transformations and instead hits the static “ghost” mesh. Such a problem does not exist for the shader method. Furthermore, the shader based method ignores the “.pickingCollider” property of objects since this property is only relevant for the raycast based method. The shader picker does however care about the object’s “.shaderPickingDetails” property

anObject.shaderPickingDetails = true;

  This property simply asks the shader picker to evaluate not just whether the mouse events exist, but also the data involved in them such as position, normals, etc. The calculation of such information is not free as it is in the raycast approach, so it should only be requested when needed. If set to false, the mouse events will trigger, but some of the event properties will be null or invalid. The value is set to false by default, and in the case of listing 4, it needs to be enabled so that we can retrieve the location of the mouse event in the scene when using the shader picker.

  Hopefully, Adobe will someday remove the drawToBitmapData() bottleneck and make all our lives easier!

The Raycast Mouse Pickers

  On the previous section, we discussed the difference between the raycast pickers and the shader picker. Its time to talk about the difference between the two available raycast mouse pickers, (1) RAYCAST_FIRST_ENCOUNTERED and (2) RAYCAST_BEST_HIT. These are actually the same picker, but with slightly different settings. The differentiation exists for the same reason “best hit” and “first encountered” options exist in an object’s picking collider: for stopping the collision test as soon as it is needed in terms of precision. In this case, the test stops checking on the criteria of “renderables” instead of triangles.

 
•   PickingType.RAYCAST_FIRST_ENCOUNTERED ( default value )  
    Uses ray tracing as a picking method, stopping tests at the first successful renderable level collision.  
 
•   PickingType.RAYCAST_BEST_HIT  
    Uses ray tracing as a picking method, evaluating the best ( closest ) hit between all the colliding renderables.  
 

  But what is a “renderable”? The raycast picking system relies on an entity’s bounding volume, but an entity can contain multiple sub meshes. These sub meshes share a single bounding volume. A sub mesh is a renderable, and that is what a renderable is: something that can be drawn to screen. Sub meshes exist because Stage3D limits the amount of elements that can be put in a single buffer, so if the poly count increases beyond such limits in a mesh, a new set of buffers is created within a new sub mesh object. Such buffers represent vertices, normals, uvs, etc. With RAYCAST_FIRST_ENCOUNTERED, the picking system will not care about which colliding sub renderable is closest to the camera and stop collision checks as soon as a positive collision is found. RAYCAST_BEST_HIT, instead, will find all collisions amongst renderables and pick the one closest to the camera, and exist solely for this case where there are large meshes subdivided into multiple sub meshes.

  This is in fact a very subtle difference, and to be honest, one not very easy to grasp, but it’s good to be aware of it. The use of RAYCAST_BEST_HIT should not be very common, but it is there in case its needed. For example, you will not have a problem if you align a set of cubes along the path of the mouse ray, RAYCAST_FIRST_ENCOUNTERED will do just fine with that, but if you import a mesh which contains a set of aligned cubes into the same geometry ( yeah, weird case ), you will need RAYCAST_BEST_HIT for proper picking. The following listing illustrates all of this. We added another cone tracer pointing towards the hit location just to help us know where it is.

  As you can see, if the mouse ray hits more than 1 sub mesh with RAYCAST_FIRST_ENCOUNTERED, it will fail to identify the appropriate collision. RAYCAST_BEST_HIT however, always finds the right one. Notice how we used a AS3_FIRST_ENCOUNTERED collider on the mesh, otherwise we would be checking collisions with its bounding volume and not with the triangles of its sub meshes… even worse! Also, notice how we shuffled the z positions of the sub meshes. If we didn’t do this, the order of the sub meshes in the tree would coincide with the order in which they collide with the ray, hence achieving correct results by mere coincidence.

  By now we have covered the 3 available mouse pickers, as well as the 7 available picking colliders for independent objects. Phewww, that’s quite a lot! But don’t worry, we have actually covered most of the topic of picking in the Away3D engine at this point. All this gives you a lot of flexibility when it comes to setting up mouse interactivity on your project as efficiently as possible. It can be confusing if you don’t understand what each feature does, but once you do, the decision becomes pretty obvious on each implementation.