[TOOL] HWRM DAE importer for Blender


(D Kesserich) #21

It's because the Vertex input and the Normals input both have offsets of 0. So since there are two inputs, offset_per_vertex gets set incorrectly right away. Forcing it to be 0 doesn't work because then the this_offset == offset_per_vertex trips over itself and no triangle data gets written.

I sort of have a fix for this in the exporter, but it's real hacky and I don't like it.


(Taiidan Republic Mod) #22

How can they both have offset of 0? Doesn't the data overwrite itself?


(D Kesserich) #23

Apparently not, The p values are just data ordering for the float arrays. It's not a requirement for the order to be different between the vertices, normals, and UV coords. There are valid reasons for them to be different, but they don't have to be.


(Taiidan Republic Mod) #24

Ok, it was quite easy once I understood the problem. Blender-generated DAEs seem to be importing ok now:

Github updated.

https://github.com/Dom222/hwrm-dae-importer


(D Kesserich) #25

Awesome!

I fell into a real rats nest trying to fix up this vertex duplication bug in the exporter. I think he did it intentionally because he couldn't figure out how to properly merge the p data for the UVs into the p data for the regular geometry when they're wildly different lengths. Frankly I'm damned confused myself. I don't think you can just pad out p with crap, can you? It's really hard to figure it out by checking the Gearbox DAEs because there's so much data in them.


(Taiidan Republic Mod) #26

UV array is in pairs, each pair is a uv coord. There is a nice simple example of a triangle here:

https://www.khronos.org/collada/wiki/glsl_example_effect_(collada_1.4.1)


(D Kesserich) #27

That link goes nowhere.

I figured out where my brain blockage was with the UVs. The winding order is not the same as for the 3D faces, so I have to figure that out, and there's something wrong with how I'm getting the normals now, too. Turns out I do have to use offsets, but I can't figure out how to get the indices for the coordinates. Try to compare with the default Blender dae exporter also doesn't help because it looks like it's culling duplicate normal triads, and if you have a mix of smooth and flat shaded polys it uses both the vertex normals and the face normals, and then the indices are even harder to parse.


(Taiidan Republic Mod) #28

Link should be:

https://www.khronos.org/collada/wiki/GLSL_Example_Effect_(COLLADA_1.4.1)


(D Kesserich) #29

I got it all figured out. It turns out the 'loop' data structure in Blender is super-duper important.

Now I can go back to tinkering with your importer and trying to help figure out how to re-build the UVs and assign materials and stuff.


(Taiidan Republic Mod) #30

Great. I'm in the process of rewriting the mesh part - it's taking quite a while to process larger DAE files...


(Taiidan Republic Mod) #31

Progress is slow:

Now creating materials properly, and applying them to the appropriate faces...

...at least for the Kad_Swarmer. Some other DAEs are now making it choke for reasons I don't yet understand.

The UV mapping is a challenge though.

Latest code here:


(Siber) #32

Man, I just want to step in and say thanks for all the work being done here. I know it's not done yet, but I'm very glad someone's working on this particular problem.


(ajlsunrise) #33

Did you try flipping the uvcoords by doing uvcoord.y = 1 - uvcoord.y?


(D Kesserich) #34

Are the ones it's choking on ones that use badges? Polygons that use a badge material have two UV Layers/Textures (I'm not entirely sure what the difference between those is, for the exporter I'm pulling from uv_layers) on them, My guess is it's choking because it's trying to assign both UV coordinate sets to the same layer and failing.

You should be making a new UV Texture for every UV set in each material group, and just name it the same as the UV ID from the DAE.


(Taiidan Republic Mod) #35

Finally getting somewhere. I created and UV mapped this simple mesh using a python script:

It is only a matter of time before I can get that to happen with DAE data...


(D Kesserich) #36

I'm looking at the latest version of the script, and I think maybe you've overcomplicated a lot of things.

You're going to the trouble of getting a lot of irrelevant material data, in a slightly weird way. I'd treat material creation as a separate step from the mesh parsing and creation. Blender doesn't require a mesh to exist to add materials to its material database, so right off the top of reading the DAE you can just make all the materials.

This would dramatically improve the readability of the code by just having them there instead of the recursive material generation that you're doing. Also probably a little bit faster because right now you're going through this step on every piece of geometry when it comes up, even though sub-meshes tend to use the same materials (think the Taiidan Interceptor's wings. They're using the same material as the main ship, so actually getting that material data again when it's already been made is a waste of time). Then when you get to each <triangles> section of the mesh you feed each of those with the material name into your geometry generator and assign the material at the end of it.

Also you should import math and use math.pi instead of defining pi as 3.14159265359, since math.pi is actual pi, not an approximation.

I need to let my brain percolate around this for a bit and then I might fork your code and try to do a refactor later.


(Taiidan Republic Mod) #37

I agree. I'm also coming to think that it's better to create the uv maps before adding/linking the materials.

Feel free to mess with the code, but it is quite messy...

By the way, did you re-release the exporter?


(D Kesserich) #38

Yeah


(D Kesserich) #39

Okay, so I started a pretty extensive re-do of your importer (with the exception of a couple of functions it’s pretty much a ground up re-write). I’m going to stick the code here if you want to poke at it and see if you can figure out things that I’ve missed. UV generation doesn’t work, and I can’t for the life of me figure out assigning the normals/smoothing groups. mesh.loops[index].normal = theNormalIwant doesn’t work.

Logically the materials assignment should work, but for some reason the material on the first mesh is being overridden by the material of the next mesh in the same geo instance (which is why the join call is commented out right now). Also I need to add trimming the vertex array in the meshBuilder function.

Also there’s some poor error handling if a mesh doesn’t have a material at all, and I don’t bother making materials that don’t have at least one texture on them.

import bpy
import xml.etree.ElementTree as ET
import math
from mathutils import *

C = bpy.context
D = bpy.data

#############
#DAE Schemas#
#############

#Just defining all the DAE attributes here so the processing functions are more easily readable

#Utility Schemas
DAENode = "{http://www.collada.org/2005/11/COLLADASchema}node"
DAETranslation = "{http://www.collada.org/2005/11/COLLADASchema}translate"
DAEInit = "{http://www.collada.org/2005/11/COLLADASchema}init_from"
DAEInput = "{http://www.collada.org/2005/11/COLLADASchema}input"
DAEFloats = "{http://www.collada.org/2005/11/COLLADASchema}float_array"
DAESource = "{http://www.collada.org/2005/11/COLLADASchema}source"

##Material Schemas
DAELibMaterials = "{http://www.collada.org/2005/11/COLLADASchema}library_materials"
DAEMaterials = "{http://www.collada.org/2005/11/COLLADASchema}material"
DAELibEffects = "{http://www.collada.org/2005/11/COLLADASchema}library_effects"
DAEfx = "{http://www.collada.org/2005/11/COLLADASchema}effect"
DAELibImages = "{http://www.collada.org/2005/11/COLLADASchema}library_images"
DAEimage = "{http://www.collada.org/2005/11/COLLADASchema}image"
DAETex = "{http://www.collada.org/2005/11/COLLADASchema}texture"
DAEProfile = "{http://www.collada.org/2005/11/COLLADASchema}profile_COMMON"
DAETechnique = "{http://www.collada.org/2005/11/COLLADASchema}technique"
DAEPhong = "{http://www.collada.org/2005/11/COLLADASchema}phong"

#Geometry Schemas
DAEGeo = "{http://www.collada.org/2005/11/COLLADASchema}geometry"
DAEMesh = "{http://www.collada.org/2005/11/COLLADASchema}mesh"
DAEVerts = "{http://www.collada.org/2005/11/COLLADASchema}vertices"
DAETris = "{http://www.collada.org/2005/11/COLLADASchema}triangles"
DAEp = "{http://www.collada.org/2005/11/COLLADASchema}p"

###########
#Functions#
###########
DAEPath = "F:\\myMod\\"

def makeTextures(name, path):
    D.textures.new(name, 'IMAGE')    
    D.textures[name].image = D.images.load(DAEPath+path)
    
def makeMaterials(name, textures):
    D.materials.new(name)
    
        
    D.materials[name].texture_slots.add()
    D.materials[name].texture_slots[0].texture = D.textures[textures[0]]
        

def meshBuilder(matName, Verts, Normals, UVCoords, vertOffset, normOffset, UVoffsets, pArray):
    print("Building "+matName)
    D.meshes.new(matName)
    ob = bpy.data.objects.new(matName,D.meshes[matName])
    
    
    
    #split <p> array to get just the face data
    faceIndices = []
    for i in range(0, len(pArray)):
        faceIndices.append(pArray[i][vertOffset])
    faceTris = [faceIndices[i:i+3] for i in range(0,len(faceIndices),3)]
    #print(Verts)
    #print(faceTris)    
    D.meshes[matName].from_pydata(Verts,[],faceTris)
    D.meshes[matName].materials.append(D.materials[matname])
    
    normIndices = []
    for i in range(0, len(pArray)):
        normIndices.append(Vector(Normals[pArray[i][normOffset]]))
    
    print(normIndices)
    print(len(normIndices))
    print(len(D.meshes[matName].loops))
    
    for l in range(0,len(D.meshes[matName].loops)):
        print(D.meshes[matName].loops[l].normal)
        #print(Vector(Normals[normIndices[l]]))
        ob.data.loops[l].normal = normIndices[l]
        #D.meshes[matName].loops.foreach_set('normal',rawNorms)
        print(D.meshes[matName].loops[l].normal)
        #D.meshes[matName].update()

    for i in range(0,len(UVCoords)):
        D.meshes[matName].uv_textures.new()
        for l in range(0,len(D.meshes[matName].loops)):
            D.meshes[matName].uv_layers[i].data[l].uv = Vector(UVCoords[i][pArray[i][UVoffsets[i]]])
    
    C.scene.objects.link(ob)
    
    return ob

#If it ain't broke don't fix it. This function written by Dom2
def CreateJoint(jnt_name,jnt_locn,jnt_rotn,jnt_context):
    print("Creating joint" + jnt_name)
    this_jnt = bpy.data.objects.new(jnt_name, None)
    jnt_context.scene.objects.link(this_jnt)
    pi = math.pi
    this_jnt.rotation_euler.x = joint_rotation[0] * (pi/180.0)
    this_jnt.rotation_euler.y = joint_rotation[1] * (pi/180.0)
    this_jnt.rotation_euler.z = joint_rotation[2] * (pi/180.0)
    this_jnt.location.x = float(jnt_locn[0])
    this_jnt.location.y = float(jnt_locn[1])
    this_jnt.location.z = float(jnt_locn[2])
    return this_jnt
        
################
#XML Processing#
################

#More Dom2 code here
tree = ET.parse("f:\\myMod\\test.dae")
root = tree.getroot()

print(" ")
print("CREATING JOINTS")
print(" ")

# Create joints
for joint in root.iter(DAENode): # find all <node> in the file
    # Joint name
    joint_name = joint.attrib["name"]
    # Joint location
    joint_location = joint.find(DAETranslation)
    if joint_location == None:
        joint_location = ['0','0','0'] # If there is no translation specified, default to 0,0,0
    else:
        joint_location = joint_location.text.split()
    # Joint rotation
    joint_rotationX = 0.0
    joint_rotationY = 0.0
    joint_rotationZ = 0.0
    for rot in joint:
        if "rotate" in rot.tag:
            if "rotateX" in rot.attrib["sid"]:
                joint_rotationX = float(rot.text.split()[3])
            elif "rotateY" in rot.attrib["sid"]:
                joint_rotationY = float(rot.text.split()[3])
            elif "rotateZ" in rot.attrib["sid"]:
                joint_rotationZ = float(rot.text.split()[3])
    joint_rotation = [joint_rotationX,joint_rotationY,joint_rotationZ]
    # Joint or mesh?
    is_joint = True
    for item in joint:
        if "instance_geometry" in item.tag:
            #print("this is a mesh:" + item.attrib["url"])
            is_joint = False
    # If this is a joint, make it!
    if is_joint:
        CreateJoint(joint_name, joint_location,joint_rotation,C)
        
#My code starts here - DL

#find textures and create them
for img in root.find(DAELibImages):
    #print(img.attrib["name"])
    #print(img.find(DAEInit).text)
    makeTextures(img.attrib["name"],img.find(DAEInit).text.lstrip("file://"))

#Make materials based on the Effects library
for fx in root.find(DAELibEffects).iter(DAEfx):
    matname = fx.attrib["name"]
    print(matname)   
   
    matTextures = []
    
    for t in fx.iter(DAETex): 
        matTextures.append(t.attrib["texture"].rstrip("-image"))
    
        
    if len(matTextures) > 0:
        makeMaterials(matname, matTextures)

#Find the mesh data and split the coords into 2D arrays

for geo in root.iter(DAEGeo):
    meshName = geo.attrib["name"]
    mesh = geo.find(DAEMesh)
    
    blankMesh = D.meshes.new(meshName)
    ob = bpy.data.objects.new(meshName, blankMesh)
    C.scene.objects.link(ob)
    
    print(meshName)    
    
    UVs = []
    
    for source in mesh.iter(DAESource):
        print("Source: " + source.attrib["id"])
        if "position" in source.attrib["id"].lower():
            rawVerts = [float(i) for i in source.find(DAEFloats).text.split()]
            #print(rawVerts)
        
        if "normal" in source.attrib["id"].lower():
            rawNormals = [float(i) for i in source.find(DAEFloats).text.split()]
        
        if "texcoord" in source.attrib["id"].lower():
            rawUVs = [float(i) for i in source.find(DAEFloats).text.split()]
            coords = [rawUVs[i:i+2] for i in range(0, len(rawUVs),2)]
            UVs.append(coords)
        
            
    vertPositions = [rawVerts[i:i+3] for i in range(0, len(rawVerts),3)]
    meshNormals = [rawNormals[i:i+3] for i in range(0, len(rawNormals),3)]
    #print(vertPositions)
    #print(meshNormals)        
    #for u in range(0,len(UVs)):
    #    print("UV" + str(u))
    #   print(UVs[u])
    
    subMeshes = []
    
    for tris in mesh.iter(DAETris):
        material = tris.attrib["material"]
        maxOffset = 0
        UVOffsets = []
        vertOffset = 0
        normOffset = 0
        for inp in tris.iter(DAEInput):
            if int(inp.attrib["offset"]) > maxOffset:
                maxOffset = int(inp.attrib["offset"])
            if inp.attrib["semantic"].lower() == "texcoord":
                UVOffsets.append(int(inp.attrib["offset"]))
            if inp.attrib["semantic"].lower() == "vertex":
                vertOffset = int(inp.attrib["offset"])
            if inp.attrib["semantic"].lower() == "normal":
                normOffset =  int(inp.attrib["offset"])
        splitPsoup = [int(i) for i in tris.find(DAEp).text.split()]
        pArray = [splitPsoup[i:i+(maxOffset+1)] for i in range(0, len(splitPsoup),(maxOffset+1))]
        #print(pArray)
        
        subMeshes.append(meshBuilder(material, vertPositions, meshNormals, UVs, vertOffset, normOffset, UVOffsets, pArray))
    
    for obs in subMeshes:
        obs.select = True
    
    ob.select = True
    C.scene.objects.active = ob
    #bpy.ops.object.join()

(Taiidan Republic Mod) #40

Looks good. On the Kad_Swarmer I get “KeyError: ‘material’” on line 222 - it seems to be looking for a material in the COL mesh. I think we need a test for materials to filter out meshes without any materials.

In terms of normals, I didn’t think we needed to bother - I thought normals were defined by the order of the vertices of the triangle? I don’t have any evidence for that though, just an assumption…

I have been working with the following very simple example to try to understand UV maps and materials. It makes a simple tetrahedron and applies a couple of UV maps. One of them is set to active and then the texture is displayed. I didn’t yet get as far as implementing all of this in the importer.

#----------------------------------------------------------
# File uvs.py
#----------------------------------------------------------
import bpy
import os
 
def createMesh(origin):
    # Create mesh and object
    me = bpy.data.meshes.new('TetraMesh')
    ob = bpy.data.objects.new('Tetra', me)
    ob.location = origin
    # Link object to scene
    scn = bpy.context.scene
    scn.objects.link(ob)
    scn.objects.active = ob
    scn.update()
 
    # List of verts and faces
    verts = [
        (1.41936, 1.41936, -1), 
        (0.589378, -1.67818, -1), 
        (-1.67818, 0.58938, -1), 
        (0, 0, 1)
    ]
    faces = [(1,0,3), (3,2,1), (3,0,2), (0,1,2)]
    # Create mesh from given verts, edges, faces. Either edges or
    # faces should be [], or you ask for problems
    me.from_pydata(verts, [], faces)
 
    # Update mesh with new data
    me.update(calc_edges=True)
 
    # First texture layer: Main UV texture
    texFaces = [
        [(0.6,0.6), (1,1), (0,1)],
        [(0,1), (0.6,0), (0.6,0.6)],
        [(0,1), (0,0), (0.6,0)],
        [(1,1), (0.6,0.6), (0.6,0)]
    ]
    uvMain = createTextureLayer("UVMain", me, texFaces)
 
    # Second texture layer: Front projection
    texFaces = [
        [(0.732051,0), (1,0), (0.541778,1)],
        [(0.541778,1), (0,0), (0.732051,0)],
        [(0.541778,1), (1,0), (0,0)],
        [(1,0), (0.732051,0), (0,0)]
    ]
    uvFront = createTextureLayer("UVFront", me, texFaces)
 
    # Third texture layer: Smart projection
    bpy.ops.mesh.uv_texture_add()
    uvCyl = me.uv_textures.active
    uvCyl.name = 'UVCyl'
    bpy.ops.object.mode_set(mode='EDIT')
    bpy.ops.uv.cylinder_project()
    bpy.ops.object.mode_set(mode='OBJECT')
 
    # Set Main Layer active
    me.uv_textures["UVMain"].active = True
    me.uv_textures["UVMain"].active_render = True
    me.uv_textures["UVFront"].active_render = False
    me.uv_textures["UVCyl"].active_render = False
 
    return ob
 
def createTextureLayer(name, me, texFaces):
    uvtex = me.uv_textures.new()
    uvtex.name = name
    for n,tf in enumerate(texFaces):
        datum = uvtex.data[n]
        datum = tf[0],tf[1],tf[2]
    return uvtex
 
def createMaterial():
    realpath = os.path.expanduser('C:/Program Files (x86)/Steam/steamapps/workshop/content/244160/403557412/Tur_P1Mothership/Tur_P1Mothership_DIFF.TGA')
    tex = bpy.data.textures.new('ColorTex', type = 'IMAGE')
    tex.image = bpy.data.images.load(realpath)
    tex.use_alpha = True
 
    # Create shadeless material and MTex
    mat = bpy.data.materials.new('TexMat')
    mat.use_shadeless = True
    mtex = mat.texture_slots.add()
    mtex.texture = tex
    mtex.texture_coords = 'UV'
    mtex.use_map_color_diffuse = True 
    return mat
 
def run(origin):
    ob = createMesh(origin)
    mat = createMaterial()
    ob.data.materials.append(mat)
    
    TetMesh = bpy.data.meshes.get("TetraMesh")
    TetMesh.uv_textures["UVMain"].active = True
    for face in TetMesh.polygons:
        TetMesh.uv_textures["UVMain"].data[face.index].image = bpy.data.images['Tur_P1Mothership_DIFF.TGA']
    
    # Make textures visible
    for area in bpy.context.screen.areas: # iterate through areas in current screen
        if area.type == 'VIEW_3D':
            for space in area.spaces: # iterate through spaces in current VIEW_3D area
                if space.type == 'VIEW_3D': # check if space is a 3D view
                    space.viewport_shade = 'TEXTURED' # set the viewport shading to textured
    
    return
 
if __name__ == "__main__":
    run((0,0,0))