without that im pretty sure impacts to the drive train will go into the crankshaft and flywheel rather than being distributed to the engine and transmission mounts, not sure how long you can drive it like that but if you offroaded like that you would probably do permanent damage
I was also thinking about the spacing between transmission input shaft and torque converter... there is probably some amount of head space that is designed into that; now they are that much closer together.. could contact in ways it was not designed.
Plus the starter is now closer to the flywheel ring gear by the thickness of that spacer plate; which might be fine, but it might also cause the starter gear to clash against the ring gear, bottom out, could damage the bendix or internal bearings of the starter over time, or cause the starter to become mis-matched and not mesh properly.
Short term, I don't see an issue.. but over a longer period of time, could be weeks, could be months or even years... I would not feel comfortable knowing I "did the job wrong" and left a part out.