This post attempts to explain and document the issue that the poor fellow here has. I have been bitten by the same behavior, and I have played with the map compiler enough now that I think I know what the root cause of the issue is. It appears to be a bug in the map compiler. I haven't yet found out if it's been fixed in BTS09, but this is definitely present in BTS06R2.
Long story short: If you have schemas that use the attributes defined in the http://www.w3.org/XML/1998/namespace namespace, you're going to have problems if you use those schemas in maps. The reason is the way that the VS BizTalk schema editor and the map compiler handle this special-case namespace.
The easiest way to include the elements in the http://www.w3.org/XML/1998/namespace namespace, like lang, is to add BTS.xml as an import. You can find it if your BizTalk project has an assembly reference to Microsoft.BizTalk.GlobalPropertySchemas, which it should by default. BTS.xml is a copy of what you'll find if you browse to http://www.w3.org/XML/1998/namespace, and it has been placed there during the initial install of BizTalk Server.
This namespace is a special case - it is uniquely and permanently mapped to the prefix "xml". Attempting to assign it to another prefix, or attempting to assign another namespace to the "xml" prefix, can result in errors.
So what exactly is the issue? If you import BTS.xml into your schema (or even if you create your own copy of the http://www.w3.org/XML/1998/namespace schema and import that) and use that schema in a map, the mapper creates a new prefix, like "ns0", and assigns http://www.w3.org/XML/1998/namespace to it. This causes two issues:
1) If you do a Validate Map to generate the XSL transform from the map file, the XSLT that's generated won't run on it's own in the Visual Studio XSLT debugger. You will get a complaint that http://www.w3.org/XML/1998/namespace can't be mapped to a new prefix. This doesn't occur if you use Test Map from the context menu, run the map in BizTalk deployment, or call the Transform method on the Transform property of the compiled map type (for unit testing, as explained here. Only applies to BTS 2006, since BTS2009 has the new test wrappers for schema and map types. If this bug is still present in BTS09, I don't know if it affects the test wrappers or not.).
2). If you are using the map to create one of the http://www.w3.org/XML/1998/namespace attributes in the output document, the resulting xml will be invalid, because BizTalk will drop the http://www.w3.org/XML/1998/namespace declaration into the output file along with its erroneous prefix assignment.
If you scroll to the end of the post I linked to at the beginning of this entry and look at the writer's response to himself, he mentions that he has found 3 workarounds, all of which he notes don't really work that well, and one of which I have spent a little more time analyzing.
His first workaround is to explicitly add the declaration "xmlns:xml="http://www.w3.org/XML/1998/namespace"" to the rootnode of the schema using a text editor. This will cause the namespace to appear with the correct prefix in the map XSLT. Unfortunately, as he notes, this declaration vanishes the next time you save the schema using the VS BizTalk schema editor - it apparently recognizes the redundant declaration and removes it. However, what he didn't examine further is that this only works if the BTS.xml schema import and manual namespace declaration are both done on the target schema - if the import/declaration are only done on the source schema, the map compiler still generates a bad prefix. You can't just add the declaration on the target schema either - you have to add the declaration and import BTS.xml. This has to be a bug in the map compiler.
His second workaround is to extract the map XSLT, fix the declaration there, and use that XSLT as the input to Custom XSL Path. This works, but it's extremely unintuitive if your code has to be maintained, especially if you leave all the functoids and links in the original map so the XSLT can easily be extracted again if changes are made. If you do that, no one will ever notice the Custom XSL Path property and will wonder why map changes don't have any effect!
His third workaround is to use custom XSLT inline scripting to create the xml:lang attribute. He mentions that this still results in a bad prefix, so it doesn't really fix the problem.
Hopefully this bug has been fixed in the BTS09 map compiler. If I find out, I'll update this post.
Wednesday, June 17, 2009
Tuesday, March 3, 2009
BizTalk and Dynamics AX
I'm beginning work on an effort to integrate BizTalk with Dynamics AX 2009 via the BizTalk AIF (Application Integration Framework) Adapter that comes with AX 2009. I don't have a lot to post just yet as we're just getting started, but I'm looking forward to learning more, as there's not a whole lot out there yet about the AIF adapter. AX is also capable of communicating via messages through the AIF to MSMQ queues and the file system, but its cool to see such great support for BizTalk - the adapter install is included on the CD with the product, and includes hooks into the "Add Adapter Metadata" wizard in Visual Studio so you can connect to an AX instance to generate schemas and port types automatically. The adapter and the wizard are very tailored - really all you need are login credentials and the name and port number of an AX instance, and you're off and running.
From what I've done so far (the walkthrough in the newly-updated white paper), from a BizTalk perspective it's pretty simple to get set up and running, and the fact that AX uses fairly simple XML is a big plus. The white paper walkthrough has you creating orchestrations for the purposes of synchronous communication (optional, as async is available as well and probably preferable for most everything) as well as correlation for an asynchronous exchange, but its easy to see how you could do it all without orchestrations for simple one-way messaging.
The white paper is here. If you are at all interested in getting BizTalk and AX 2009 to work together, this should aboslutely be your very first stop. This white paper isn't like most others - it's an instructional walkthrough that covers every step, from AX configuration to building the BizTalk solutions from scratch. Unfortunately, the prerequisites/setup are awful. At the minimum, if you have a valid login at PartnerSource, you'll be downloading many gigabytes worth of Virtual PC images (2 of them), and you'll need decent hardware to run them both at once (I wouldn't touch it with a 10 foot pole if you don't have at least 4 GB RAM, or separate systems you can use to run each image). If you don't have PartnerSource access, you'll need to set up Dynamics yourself, or obtain an already-set-up image from another source. I highly recommend getting the images from PartnerSource if possible - setting up Dynamics is a major activity, requiring a domain and lots of other server resources.
More posts regarding BizTalk and AX are forthcoming.
From what I've done so far (the walkthrough in the newly-updated white paper), from a BizTalk perspective it's pretty simple to get set up and running, and the fact that AX uses fairly simple XML is a big plus. The white paper walkthrough has you creating orchestrations for the purposes of synchronous communication (optional, as async is available as well and probably preferable for most everything) as well as correlation for an asynchronous exchange, but its easy to see how you could do it all without orchestrations for simple one-way messaging.
The white paper is here. If you are at all interested in getting BizTalk and AX 2009 to work together, this should aboslutely be your very first stop. This white paper isn't like most others - it's an instructional walkthrough that covers every step, from AX configuration to building the BizTalk solutions from scratch. Unfortunately, the prerequisites/setup are awful. At the minimum, if you have a valid login at PartnerSource, you'll be downloading many gigabytes worth of Virtual PC images (2 of them), and you'll need decent hardware to run them both at once (I wouldn't touch it with a 10 foot pole if you don't have at least 4 GB RAM, or separate systems you can use to run each image). If you don't have PartnerSource access, you'll need to set up Dynamics yourself, or obtain an already-set-up image from another source. I highly recommend getting the images from PartnerSource if possible - setting up Dynamics is a major activity, requiring a domain and lots of other server resources.
More posts regarding BizTalk and AX are forthcoming.
root_reference and displayroot_reference
If you import a schema into BizTalk, you may get something in the schema editor that looks much different than something you'd get building one in the editor yourself. When you build a schema using the schema editor, you get a statically-defined schema - definitions of nodes exist "in place." This is great for simple schemas, but many more complex schemas rely on more powerful organizational structures made available by the XSD schema language to create more organized and extensible schemas.
This schema is defined structurally - definitions of structures (records, elements, attributes) are all made at the root level and then "referenced" to create the document structure. This is similar to how a DTD is defined - a "root node" exists only incidentally, because its at the top of the reference chain. This type of schema definition can be capitalized on by using complex, named structures to create extremely large and complicated schemas that can be extended very easily. Some industry-standard schemas are so large and well-defined that they actually resemble an inheritance tree that you'd see in an object-oriented language - structures inherit from other structures and are substituted in at various places in the schema as reusable types. Take a look at the documentation for Siemens' version 6.0.2 PLM (Product Lifecycle Management) XML schema standard (warning: The target of that link is an enormous piece of documentation, so large it is capable of crashing browsers). If you download the schemas and take a look at them, you'll see that they are defined in a way similar to how you'd define a tree of classes - classes inherit from other classes, and subclasses can be substituted wherever a parent class is referenced. By defining the schemas in this way, they can be extended infinitely despite their enormous size and scope, and they are very flexible.
In any case, its enlightening to look at a schema this way, but for many tasks it can be a pain. If you just want to take a look at the structure of the schema and verify a few things, it's easier to view it as a proper schema with a single root node. The tools provide a way to fix this and show a "normalized" view of the schema: root_reference and displayroot_reference.
With your schema open, click the top level "Schema" node in the schema editor and take a look at the Properties pane. In the Reference group, one of the available properties is "Root Reference," which provides a drop-down from which you can pick any of the records defined at the top level of the schema. I set this to "Document", which is the root node of my schema.
You can see the result in the schema snippet above: the schemaInfo element gets a new attribute, root_reference. What's not well documented is exactly what this does - by setting the root_reference to the root node of your schema, the view of the schema in the mapper tool will be corrected. If I was to take this schema right now and load it into a map, I'd see the simpler view of the schema. There is a little-documented, manual way to fix the view in the schema editor as well: open the schema file manually using the XML editor, and right next to the root_reference attribute, add another attribute, displayroot_reference, with the same value.
Save and close the schema, and re-open it in the schema editor. The view of the schema is now fixed around the root node, making the schema much easier to view. However, viewing the schema in this way comes with a big caveat. You must be careful when making edits to the schema. If you add an element to a record, what you may be doing under the covers is adding an element to that record's type, changing any other references to that record as well. Things are similarly complicated for all other kinds of edits as well, so tread lightly, be careful what you change, and check the node definition and other node references whenever you make a change. Depending on what you are doing, it may make more sense to work in non-"displayroot_reference" mode, so feel free to switch it on and off. The presence of displayroot_reference does not change the way the schema is parsed or handled by BizTalk at all.
Unlike the BizTalk schema editor tool, many schema tools create their schemas this way by default. Simple schemas that don't take advantage of reusable or substitutable types (and don't even really need to) end up being defined in an obtuse way. root_reference and displayroot_reference provide a way to cut through that definition and provide a simpler view of both a schema definition and a map.
This schema is defined structurally - definitions of structures (records, elements, attributes) are all made at the root level and then "referenced" to create the document structure. This is similar to how a DTD is defined - a "root node" exists only incidentally, because its at the top of the reference chain. This type of schema definition can be capitalized on by using complex, named structures to create extremely large and complicated schemas that can be extended very easily. Some industry-standard schemas are so large and well-defined that they actually resemble an inheritance tree that you'd see in an object-oriented language - structures inherit from other structures and are substituted in at various places in the schema as reusable types. Take a look at the documentation for Siemens' version 6.0.2 PLM (Product Lifecycle Management) XML schema standard (warning: The target of that link is an enormous piece of documentation, so large it is capable of crashing browsers). If you download the schemas and take a look at them, you'll see that they are defined in a way similar to how you'd define a tree of classes - classes inherit from other classes, and subclasses can be substituted wherever a parent class is referenced. By defining the schemas in this way, they can be extended infinitely despite their enormous size and scope, and they are very flexible.
In any case, its enlightening to look at a schema this way, but for many tasks it can be a pain. If you just want to take a look at the structure of the schema and verify a few things, it's easier to view it as a proper schema with a single root node. The tools provide a way to fix this and show a "normalized" view of the schema: root_reference and displayroot_reference.
With your schema open, click the top level "Schema" node in the schema editor and take a look at the Properties pane. In the Reference group, one of the available properties is "Root Reference," which provides a drop-down from which you can pick any of the records defined at the top level of the schema. I set this to "Document", which is the root node of my schema.
You can see the result in the schema snippet above: the schemaInfo element gets a new attribute, root_reference. What's not well documented is exactly what this does - by setting the root_reference to the root node of your schema, the view of the schema in the mapper tool will be corrected. If I was to take this schema right now and load it into a map, I'd see the simpler view of the schema. There is a little-documented, manual way to fix the view in the schema editor as well: open the schema file manually using the XML editor, and right next to the root_reference attribute, add another attribute, displayroot_reference, with the same value.
Save and close the schema, and re-open it in the schema editor. The view of the schema is now fixed around the root node, making the schema much easier to view. However, viewing the schema in this way comes with a big caveat. You must be careful when making edits to the schema. If you add an element to a record, what you may be doing under the covers is adding an element to that record's type, changing any other references to that record as well. Things are similarly complicated for all other kinds of edits as well, so tread lightly, be careful what you change, and check the node definition and other node references whenever you make a change. Depending on what you are doing, it may make more sense to work in non-"displayroot_reference" mode, so feel free to switch it on and off. The presence of displayroot_reference does not change the way the schema is parsed or handled by BizTalk at all.Unlike the BizTalk schema editor tool, many schema tools create their schemas this way by default. Simple schemas that don't take advantage of reusable or substitutable types (and don't even really need to) end up being defined in an obtuse way. root_reference and displayroot_reference provide a way to cut through that definition and provide a simpler view of both a schema definition and a map.
Importing DTD to XSD
Update: Using the trick below to handle the "lang" attribute has been nothing but a headache. BizTalk apparently understands that http://www.w3.org/XML/1998/namespace is a special namespace, and that the xml: prefix is reserved for it, but that doesn't stop a BizTalk map from generating something like "xmlns:ns0="http://www.w3.org/XML/1998/namespace" ns0:lang="EN"". What I ended up doing was creating a new attribute named "lang" in the local namespace and using a custom pipeline component on the send side to tack the "xml:" prefix onto it as it goes out the door. I'm told that adding the "xml:" namespace declaration to the schema that uses the xml:lang attribute in a text editor works, but it gets erased next time you open it and save it in the BizTalk schema editor. I don't consider this a workaround.
I just spent a few hours fumbling around with the DTD -> XSD import tool included in the BizTalk Visual Studio tools, and I thought I'd share my experience. There are a number of small, helpful tips here regarding the tool itself, as well as manipulating schemas and playing around with namespaces (one important one in particular).
The tool is available if you right-click a BizTalk project in Solution Explorer and do Add > Add Generated Items... and use the Generate Schemas wizard. There are three options shown: DTD Schema, XDR Schema, and Well-Formed XML. Two of these tools, DTD and Well-Formed XML, are not available right off of the bat - you need to run a couple of script files to install them.
Before we even get there though, there's an important hotfix here that you must install. The library that contains the DTD converter is completely broken out of the box and needs to be replaced with this hotfix before running the script file to install it.
Once you've obtained and run the hotfix, navigate to %programfiles%\Microsoft BizTalk Server 2006\SDK\Utilities\Schema Generator and run the two .vbs scripts there, InstallDTD and InstallWFX. These scripts will copy the DLLs in that folder (one of which was just updated by the hotfix) to the appropriate location where they can be used by Visual Studio. You may need to restart Visual Studio after running the scripts.
Head back to Add > Add Generated Items... > Generate Schemas and feed the DTD -> XSD tool a DTD schema. What you'll get is a big jumbly mess of node definitions, all at the root level. See my post on root_reference and displayroot_reference for more information about this and how to "fix" it.
So now I've got my DTD imported, but I've got one more problem: The root node on my schema, "Document", has an attribute with the namespace "xml". This isn't represented in the DTD at all, so its understandable that it isn't reflected in the XSD schema, but the sample documents I have all show that the "lang" attribute uses the "xml" namespace prefix. My schema validates just fine, but trying to validate my sample documents fails.
The most important thing to know here is that the "xml" namespace is a special case - XML parsers should universally understand that the "xml" prefix is implicitly reserved to resolve to the namespace "http://www.w3.org/XML/1998/namespace". Defined in this namespace are a couple of attributes, one of them being "lang". So if this namespace is supposed to be implicitly understood, why is schema validation getting hung up on it?
The reason is that BizTalk and its tools understand the namespace, but they don't inherently know what's contained in it. Our schema needs to reference another schema that defines the types in the http://www.w3.org/XML/1998/namespace namespace. If I was to take one of my sample documents and import it using the Well-Formed XML -> XSD generator (a great tool, but be careful - it can only define the nodes present in the particular sample document you use), I would get a second schema, referenced in the first, that defines the attributes available in http://www.w3.org/XML/1998/namespace. The Well-Formed XML -> XSD generator knows about this namespace, and knows it needs a schema that defines those types. Unfortunately, the way it resolves the problem really isn't the best way of going about it - if you deploy the project as-is, you'll get a warning that a schema is already deployed that defines types in http://www.w3.org/XML/1998/namespace.
The BizTalk product team has already defined a schema that contains the types in the http://www.w3.org/XML/1998/namespace. The schema is called "BTS.xml", and it is located in the Microsoft.BizTalk.GlobalPropertySchemas assembly, which by default is referenced in every BizTalk project and deployed to the BizTalk.System application. To reference the schema, open your document schema, click the "Schema" root node, and in the Properties pane, click the Imports property. This will surface the hidden ellipsis button - click this to open the Imports dialog. Select "XSD Import" in the dropdown and click Add to open the BizTalk Type Picker. In the Type Picker, select References > Microsoft.BizTalk.GlobalPropertySchemas > Schemas > BTS.xml and click OK. BTS.xml will be added as an XSD import with a default namespace like "ns0". This is fine, but it is more appropriate to change the prefix to "xml", which is specifically reserved for defining this namespace (note: by using the prefix "xml", no new prefix/namespace declaration will appear in the root xs:schema node, since the "xml" prefix is implicitly understood. Using any other prefix will cause a new namespace declaration to appear).
The reference has been added, but there's still one more thing to fix: the schema still thinks that the "lang" attribute is part of its namespace, not the http://www.w3.org/XML/1998/namespace. To fix this, close the schema and re-open it using the XML editor. Scroll down to where the "lang" attribute is defined and replace the entire type definition with the following: <xs:element ref="xml:lang" use="required">. You can change the value of the "use" attribute depending on your needs, but the key is that you are using "ref" instead of "name", and you have specified the "xml" prefix.
In the XML editor, you will get a blue underline with the message that the attribute is not defined, but this is because the editor can't get to the schema since it's referenced in a remote assembly. Now, if I validate a sample document against the schema, it works perfectly. When I deploy this project to BizTalk, it will automatically reference the BTS.xml schema that has already been deployed.
I just spent a few hours fumbling around with the DTD -> XSD import tool included in the BizTalk Visual Studio tools, and I thought I'd share my experience. There are a number of small, helpful tips here regarding the tool itself, as well as manipulating schemas and playing around with namespaces (one important one in particular).
The tool is available if you right-click a BizTalk project in Solution Explorer and do Add > Add Generated Items... and use the Generate Schemas wizard. There are three options shown: DTD Schema, XDR Schema, and Well-Formed XML. Two of these tools, DTD and Well-Formed XML, are not available right off of the bat - you need to run a couple of script files to install them.
Before we even get there though, there's an important hotfix here that you must install. The library that contains the DTD converter is completely broken out of the box and needs to be replaced with this hotfix before running the script file to install it.
Once you've obtained and run the hotfix, navigate to %programfiles%\Microsoft BizTalk Server 2006\SDK\Utilities\Schema Generator and run the two .vbs scripts there, InstallDTD and InstallWFX. These scripts will copy the DLLs in that folder (one of which was just updated by the hotfix) to the appropriate location where they can be used by Visual Studio. You may need to restart Visual Studio after running the scripts.
Head back to Add > Add Generated Items... > Generate Schemas and feed the DTD -> XSD tool a DTD schema. What you'll get is a big jumbly mess of node definitions, all at the root level. See my post on root_reference and displayroot_reference for more information about this and how to "fix" it.
So now I've got my DTD imported, but I've got one more problem: The root node on my schema, "Document", has an attribute with the namespace "xml". This isn't represented in the DTD at all, so its understandable that it isn't reflected in the XSD schema, but the sample documents I have all show that the "lang" attribute uses the "xml" namespace prefix. My schema validates just fine, but trying to validate my sample documents fails.
The most important thing to know here is that the "xml" namespace is a special case - XML parsers should universally understand that the "xml" prefix is implicitly reserved to resolve to the namespace "http://www.w3.org/XML/1998/namespace". Defined in this namespace are a couple of attributes, one of them being "lang". So if this namespace is supposed to be implicitly understood, why is schema validation getting hung up on it?
The reason is that BizTalk and its tools understand the namespace, but they don't inherently know what's contained in it. Our schema needs to reference another schema that defines the types in the http://www.w3.org/XML/1998/namespace namespace. If I was to take one of my sample documents and import it using the Well-Formed XML -> XSD generator (a great tool, but be careful - it can only define the nodes present in the particular sample document you use), I would get a second schema, referenced in the first, that defines the attributes available in http://www.w3.org/XML/1998/namespace. The Well-Formed XML -> XSD generator knows about this namespace, and knows it needs a schema that defines those types. Unfortunately, the way it resolves the problem really isn't the best way of going about it - if you deploy the project as-is, you'll get a warning that a schema is already deployed that defines types in http://www.w3.org/XML/1998/namespace.
The BizTalk product team has already defined a schema that contains the types in the http://www.w3.org/XML/1998/namespace. The schema is called "BTS.xml", and it is located in the Microsoft.BizTalk.GlobalPropertySchemas assembly, which by default is referenced in every BizTalk project and deployed to the BizTalk.System application. To reference the schema, open your document schema, click the "Schema" root node, and in the Properties pane, click the Imports property. This will surface the hidden ellipsis button - click this to open the Imports dialog. Select "XSD Import" in the dropdown and click Add to open the BizTalk Type Picker. In the Type Picker, select References > Microsoft.BizTalk.GlobalPropertySchemas > Schemas > BTS.xml and click OK. BTS.xml will be added as an XSD import with a default namespace like "ns0". This is fine, but it is more appropriate to change the prefix to "xml", which is specifically reserved for defining this namespace (note: by using the prefix "xml", no new prefix/namespace declaration will appear in the root xs:schema node, since the "xml" prefix is implicitly understood. Using any other prefix will cause a new namespace declaration to appear).
The reference has been added, but there's still one more thing to fix: the schema still thinks that the "lang" attribute is part of its namespace, not the http://www.w3.org/XML/1998/namespace. To fix this, close the schema and re-open it using the XML editor. Scroll down to where the "lang" attribute is defined and replace the entire type definition with the following: <xs:element ref="xml:lang" use="required">. You can change the value of the "use" attribute depending on your needs, but the key is that you are using "ref" instead of "name", and you have specified the "xml" prefix.
In the XML editor, you will get a blue underline with the message that the attribute is not defined, but this is because the editor can't get to the schema since it's referenced in a remote assembly. Now, if I validate a sample document against the schema, it works perfectly. When I deploy this project to BizTalk, it will automatically reference the BTS.xml schema that has already been deployed.
Subscribe to:
Posts (Atom)