[TOOL] HWRM DAE importer for Blender

(D Kesserich) #41

The split normals have to be set to get the smoothing groups. I’ve figured it out.

Oh and UVs :wink: :

Have to add a remove doubles step at the end, and figure out what to feed the meshbuilder for meshes that don’t have materials (or have materials that I didn’t bother making).

Then animations…


import bpy
import xml.etree.ElementTree as ET
import math
from mathutils import *

C = bpy.context
D = bpy.data

#DAE Schemas#

#Just defining all the DAE attributes here so the processing functions are more easily readable

#Utility Schemas
DAENode = "{http://www.collada.org/2005/11/COLLADASchema}node"
DAETranslation = "{http://www.collada.org/2005/11/COLLADASchema}translate"
DAEInit = "{http://www.collada.org/2005/11/COLLADASchema}init_from"
DAEInput = "{http://www.collada.org/2005/11/COLLADASchema}input"
DAEFloats = "{http://www.collada.org/2005/11/COLLADASchema}float_array"
DAESource = "{http://www.collada.org/2005/11/COLLADASchema}source"

##Material Schemas
DAELibMaterials = "{http://www.collada.org/2005/11/COLLADASchema}library_materials"
DAEMaterials = "{http://www.collada.org/2005/11/COLLADASchema}material"
DAELibEffects = "{http://www.collada.org/2005/11/COLLADASchema}library_effects"
DAEfx = "{http://www.collada.org/2005/11/COLLADASchema}effect"
DAELibImages = "{http://www.collada.org/2005/11/COLLADASchema}library_images"
DAEimage = "{http://www.collada.org/2005/11/COLLADASchema}image"
DAETex = "{http://www.collada.org/2005/11/COLLADASchema}texture"
DAEProfile = "{http://www.collada.org/2005/11/COLLADASchema}profile_COMMON"
DAETechnique = "{http://www.collada.org/2005/11/COLLADASchema}technique"
DAEPhong = "{http://www.collada.org/2005/11/COLLADASchema}phong"

#Geometry Schemas
DAEGeo = "{http://www.collada.org/2005/11/COLLADASchema}geometry"
DAEMesh = "{http://www.collada.org/2005/11/COLLADASchema}mesh"
DAEVerts = "{http://www.collada.org/2005/11/COLLADASchema}vertices"
DAETris = "{http://www.collada.org/2005/11/COLLADASchema}triangles"
DAEp = "{http://www.collada.org/2005/11/COLLADASchema}p"

DAEPath = "F:\\myMod\\"

def makeTextures(name, path):
    D.textures.new(name, 'IMAGE')    
    D.textures[name].image = D.images.load(DAEPath+path)
def makeMaterials(name, textures):
    D.materials[name].texture_slots[0].texture = D.textures[textures[0]]

def meshBuilder(matName, Verts, Normals, UVCoords, vertOffset, normOffset, UVoffsets, pArray):
    print("Building "+matName)
    subMesh = D.meshes.new(matName)
    ob = bpy.data.objects.new(subMesh.name, subMesh)
    #split <p> array to get just the face data
    faceIndices = []
    for i in range(0, len(pArray)):
    faceTris = [faceIndices[i:i+3] for i in range(0,len(faceIndices),3)]
    normIndices = []
    for i in range(0, len(pArray)):
    subMesh.use_auto_smooth = True
    #Add UVs
    for coords in range(0,len(UVOffsets)):
        meshUV = []
        for p in range(0, len(pArray)):
        for l in range(0,len(subMesh.uv_layers[coords].data)):
            subMesh.uv_layers[coords].data[l].uv = meshUV[l]
    #for i in range(0,len(UVCoords)):
    #    D.meshes[matName].uv_textures.new()
    #    for l in range(0,len(D.meshes[matName].loops)):
    #        D.meshes[matName].uv_layers[i].data[l].uv = Vector(UVCoords[i][pArray[i][UVoffsets[i]]])
    return ob

#If it ain't broke don't fix it. This function written by Dom2
def CreateJoint(jnt_name,jnt_locn,jnt_rotn,jnt_context):
    print("Creating joint" + jnt_name)
    this_jnt = bpy.data.objects.new(jnt_name, None)
    pi = math.pi
    this_jnt.rotation_euler.x = joint_rotation[0] * (pi/180.0)
    this_jnt.rotation_euler.y = joint_rotation[1] * (pi/180.0)
    this_jnt.rotation_euler.z = joint_rotation[2] * (pi/180.0)
    this_jnt.location.x = float(jnt_locn[0])
    this_jnt.location.y = float(jnt_locn[1])
    this_jnt.location.z = float(jnt_locn[2])
    return this_jnt
#XML Processing#

#More Dom2 code here
tree = ET.parse("f:\\myMod\\test.dae")
root = tree.getroot()

print(" ")
print(" ")

# Create joints
for joint in root.iter(DAENode): # find all <node> in the file
    # Joint name
    joint_name = joint.attrib["name"]
    # Joint location
    joint_location = joint.find(DAETranslation)
    if joint_location == None:
        joint_location = ['0','0','0'] # If there is no translation specified, default to 0,0,0
        joint_location = joint_location.text.split()
    # Joint rotation
    joint_rotationX = 0.0
    joint_rotationY = 0.0
    joint_rotationZ = 0.0
    for rot in joint:
        if "rotate" in rot.tag:
            if "rotateX" in rot.attrib["sid"]:
                joint_rotationX = float(rot.text.split()[3])
            elif "rotateY" in rot.attrib["sid"]:
                joint_rotationY = float(rot.text.split()[3])
            elif "rotateZ" in rot.attrib["sid"]:
                joint_rotationZ = float(rot.text.split()[3])
    joint_rotation = [joint_rotationX,joint_rotationY,joint_rotationZ]
    # Joint or mesh?
    is_joint = True
    for item in joint:
        if "instance_geometry" in item.tag:
            #print("this is a mesh:" + item.attrib["url"])
            is_joint = False
    # If this is a joint, make it!
    if is_joint:
        CreateJoint(joint_name, joint_location,joint_rotation,C)
#My code starts here - DL

#find textures and create them
for img in root.find(DAELibImages):

#Make materials based on the Effects library
for fx in root.find(DAELibEffects).iter(DAEfx):
    matname = fx.attrib["name"]
    matTextures = []
    for t in fx.iter(DAETex): 
    if len(matTextures) > 0:
        makeMaterials(matname, matTextures)

#Find the mesh data and split the coords into 2D arrays

for geo in root.iter(DAEGeo):
    meshName = geo.attrib["name"]
    mesh = geo.find(DAEMesh)
    blankMesh = D.meshes.new(meshName)
    ob = bpy.data.objects.new(meshName, blankMesh)
    UVs = []
    for source in mesh.iter(DAESource):
        print("Source: " + source.attrib["id"])
        if "position" in source.attrib["id"].lower():
            rawVerts = [float(i) for i in source.find(DAEFloats).text.split()]
        if "normal" in source.attrib["id"].lower():
            rawNormals = [float(i) for i in source.find(DAEFloats).text.split()]
        if "texcoord" in source.attrib["id"].lower():
            rawUVs = [float(i) for i in source.find(DAEFloats).text.split()]
            coords = [rawUVs[i:i+2] for i in range(0, len(rawUVs),2)]
    vertPositions = [rawVerts[i:i+3] for i in range(0, len(rawVerts),3)]
    meshNormals = [rawNormals[i:i+3] for i in range(0, len(rawNormals),3)]
    #for u in range(0,len(UVs)):
    #    print("UV" + str(u))
    #   print(UVs[u])
    subMeshes = []
    for tris in mesh.iter(DAETris):
        material = tris.attrib["material"]
        maxOffset = 0
        UVOffsets = []
        vertOffset = 0
        normOffset = 0
        for inp in tris.iter(DAEInput):
            if int(inp.attrib["offset"]) > maxOffset:
                maxOffset = int(inp.attrib["offset"])
            if inp.attrib["semantic"].lower() == "texcoord":
            if inp.attrib["semantic"].lower() == "vertex":
                vertOffset = int(inp.attrib["offset"])
            if inp.attrib["semantic"].lower() == "normal":
                normOffset =  int(inp.attrib["offset"])
        splitPsoup = [int(i) for i in tris.find(DAEp).text.split()]
        pArray = [splitPsoup[i:i+(maxOffset+1)] for i in range(0, len(splitPsoup),(maxOffset+1))]
        subMeshes.append(meshBuilder(material, vertPositions, meshNormals, UVs, vertOffset, normOffset, UVOffsets, pArray))
    for obs in subMeshes:
        obs.select = True
    ob.select = True
    C.scene.objects.active = ob
    ob.data.use_auto_smooth = True

I’ll throw it on Github later, too.

(Taiidan Republic Mod) #42

Looks good. I still can’t get it to work on the Kad_Swarmer though, even when I strip out the collision mesh and ETSH mesh, which don’t have materials. But I’m glad your able to progress my humble beginnings!

(D Kesserich) #43


Tested with the Kadeshi Swarmer and the Taiidan Interceptor. The Swarmer DAE has glow maps defined for some reason, so you’ll have to manually swap the texture on the main diffuse material to use the diffuse texture. Other than that, UVs work, smoothing groups work. I still have to do the remove doubles and the hierarchy setup step (I’ll just use your function for that, if that’s okay), and of course animations.

(Taiidan Republic Mod) #44

Nice one. If you use the code that I pasted back here…

Then you can get it to display the textures (without having to click on the UV map and select the image after doing the import).

More than okay, you are welcome, thats why I wrote it!

(D Kesserich) #45

Remove doubles and hierarchy are done.

Animations next.

(D Kesserich) #46

Animations are done!

I think…

Taiidan Interceptor and Kushan Support Frigate come in with their animations in place fine. Kadeshi Fuel pod had some weird formatting on the engine glow mesh in library_scenes that broke the initial create node function, so I wasn’t able to test that.

Also the actions look slightly different in the action editor than they do when importing the fbx, so I don’t know if that’s going to be a problem (I don’t think it is).

The other thing is re-exporting the imported DAEs. Since things like dock path nodes don’t have the attributes that the exporter expects I think I’m going to have to re-work that part of the import so they come in the right way (which I kind of wanted to do anyway because I think it would be cool for the navlights to actually come in as lamps instead of empties)

The last step is actually integrating it into the Toolkit and adding the functions that’ll allow the import to be UI driven instead of having to manually edit variables in the script.

Script is on github.

(Taiidan Republic Mod) #47

It is working well:

One problem is that to get those textures to display, I had to change the viewport shading to “Texture” instead of “Solid” and take each object into edit mode, select the appropriate faces and then choose the right image in the uv window. That’s a lot of clicks I would rather not have to do… So why not incorporate my code above that does all that automatically?

Also, it doesn’t seem to like the remote paths:

I think it just needs some way of detecting whether the path is remote or local…

(Christoph Timmermann) #48

A remote path starts with file:// as far as I know. (URI format)

(D Kesserich) #49

You could just set the shading type to GLSL in the side panel Shading options and then add a sun lamp or something. Once I change it so it brings in navlights as lamps you won’t even have to add the sun lamp. Multitexture shading is kind of terrible for exactly the reasons you describe: it completely ignores textures from the material and you have to manually set them that way.

Unfortunately images.load only takes a filepath as an argument, not a URI, and relative paths are evaluated relative to Blender, not the DAE. While the importer is in script form if there are full paths for the textures you can just set DAEPath to “”". Once I switch it over to UI integrated I’ll figure out a better solution for the texture pathing.

(D Kesserich) #50

Updated the script to parse out the custom properties on DOCK and SEG nodes, as well as bring in NAVL nodes as point lamps with their custom properties (unfortunately this doesn’t look quite as cool as I thought it would).

There’s still a manual cleanup step with the names for the DOCK, SEG, and NAVL nodes for an import to be export friendly, but it’s getting there.

(Taiidan Republic Mod) #51

Finally started looking at this again. It is now coded as an addon and has a menu item:

For the most part the imports are now export-friendly, although it sometimes struggles with RODOH dae files and the hgn_carrier cause a CTD for reasons I haven’t manage to trace (@PayDay did you notice anything different about the normals of that DAE?)

Here is the addon, more or less usable:

I will try to chase down the remaining bugs at some point.

(Christoph Timmermann) #52

Hmm, I can’t think of anything different about the normals. But could the badge (second UV channel) be related?

Nice timing by the way. I thought about using COLLADA to import/export meshes in DAEnerys, as OBJ doesn’t support 2 uv channels. And you could even open them with this to edit in Blender. :slight_smile:

(Taiidan Republic Mod) #53

Solved one bug (github updated).

Remaining bugs:

1) normal indices - test cases effected:

  • hgn_carrier.dae (gearbox example)
  • vgr_carrier.dae (RODOH’d HW2)
  • vgr_mothership.dae (RODOH’d HW2)
  • hgn_marinefrigate.dae (RODOH’d HW2)

2) subMesh creation - test cases effected:

  • hgn_battlecruiser.dae (RODOH’d HW2)
  • hgn_gunturret.dae (RODOH’d HW2)

Please feel free to throw some DAEs at it and let me know the errors you get!

(Christoph Timmermann) #54

This is a DAEnerys-generated DAE. For some reason it’s rotated by 90 degrees and all of the dockpath segments are not parented to their dockpaths.

But everything else is fine I think. :slight_smile:

(Taiidan Republic Mod) #55

Did you try the example file of the same ship? Just to test that it isn’t your version of Blender or something… I use 2.76.

Did you get an error message?

(Christoph Timmermann) #56

It is correctly rotated and the dockpath segments are where they should be with the original one.

Visual Studio is loading the DAEnerys-DAE like this:

No error message.

(Taiidan Republic Mod) #57

Argh. Sounds like I need some DAENerys test cases…

(Christoph Timmermann) #58

You can export random ships to test this. ^^
I built the exporter around the original example files that were generated (I think) by 3dsMax, there should be nothing special about them…

(D Kesserich) #59

Oh, cool! I probably wasn’t going to have the brain space to tinker with this again until the end of December (Work. And also games. And part of me kind of wants to do a ground-up re-write of the exporter to use ElementTree, since I think it’s probably faster and writes cleaner xml than the bonkers manual writes of the current version.)

The DAEnaerys problem is most likely purely a fail in the hierarchy sort function. The root nodes are rotated 90 degrees forward, and the children inherit that rotation. If nothing is getting parented correctly, then the meshes will come in with a global y-up orientation.

@payday: if you run Blender from a command prompt, then import the DAEnaerys DAE it might spit out some additional log data in the command window.

(Taiidan Republic Mod) #60

There are some updates in the works:

1) Importer options

  • Import LOD0 only as visual mesh (in case you are making a carrier and you want a fighter or resource collector for scale reference).

  • I can’t find a solution to the split normals bug that causes the crash, but for now there is a check box to turn off split normals, so you can still import the problem ships, e.g. the hgn_carrier example ship

  • Option to import dock path SEG[x] nodes as different shapes other than sphere. “Single Arrow” can be useful to get a directional reference:

  • Another thing I want to do is add an option to merge all goblins into LOD0 meshes, since goblins are no longer supported by HODOR.

2) Other fixes I plan to address…

  • Fixing a problem with the nav light names
  • Fixing a problem with dock seg names and duplicated parameters
  • Fixing some strange errors encountered by RODOH’d ships
  • Fixing problems with DAENerys ships