Support root level collection serialization

[sdeleuze]

I have done some tests with 2.1.0-SNAPSHOT since on RESThub project (https://github.com/resthub/resthub-spring-stack) we are waiting XML serialization of unwrapped lists for a while.

We use generic SpringMVC controller with Jackson 2.x for serialization.

@RequestMapping(method = RequestMethod.GET, params="page=no")
@responsebody
public List findAllNotPaginated() {
return (List)repository.findAll();
}

For JSON we get this normal output :

[{"id":17,"name":"toto"},{"id":18,"name":"toto"}]

For XML we get this bogus output (the same than is 2.0 I think) :

<ArrayList xmlns=""><id>41</id><name>toto</name></ArrayList><zdef1166645736:ArrayList xmlns:zdef1166645736=""><zdef1166645736:id>42</zdef1166645736:id><zdef1166645736:name>toto</zdef1166645736:name></zdef1166645736:ArrayList>

Is there something to do in order to activate Jackson 2.1 XML unwrapped list support, or is it a bug ?

[cowtowncoder]

Ah. Good question. So, here are some initial suggestions:

1. Extra namespace appears to be caused by Sun SJSXP Stax provider: using Woodstox will remove this. We don't know if there's a way to resolve this for SJSXP
2. Just to make sure: are you using XmlMapper (regular ObjectMapper will not work for all XML aspects). And for configurability, may also want to register JacksonXmlModule on that mapper (see next one)
3. Jackson 2.1 will still require explicit definition of whether to NOT use wrapping (due to backwards compatibility). There are multiple ways to do this; usually either use JacksonXmlModule.setDefaultUseWrapper(false), or annotation @JacksonXmlElementWrapper(useWrapping=false)
4. When using JAXB annotations, wrapping is disabled by default (i.e. no extra work is needed), if @xmlelement (etc) are found, and no @XmlElementWrapper

Third part is messy, but we weren't sure how much 2.0 backwards compatibility would matter.
But as long as you use the module, configure it, things should work.

Output looks funky, either way. If above does NOT help, please update the bug and I will have a look before 2.1.0 release.

[sdeleuze]

Hi,

Thanks for your feedback. I made more test :

1. When I use Woodstox, I get an exception saying it can't output two roots
2. Yes, I use XmlMapper
3. I have modify my code in order to pass an JacksonXmlModule with setDefaultUseWrapper(false) in XmlMapper constructor
4. Nice, thanks for the tip

Do you have some basic XmlMapper + unwrapped List simple unit tests in order to allow me to compare with my code ? We use a lot Generics, perhaps the issue comme from this.

I will also try to provide you later today some unit tests that reproduce my issue.

[cowtowncoder]

Ok. One more thing, if you can reproduce this with a stand-alone, that would be helpful.
Possibly extracting setup code that container uses.

The only other concern I have is that often root-level Collections and Maps are more problematic in XML than with JSON. But it's not necessary problem here.

[sdeleuze]

I will provide you standalone unit tests.

[sdeleuze]

You will find bellow my unit test, I used JDK 1.7 update 5

Without Woodstox, I got this serialization :

<ArrayList xmlns=""><id>123</id><name>Albert</name><description>desc</description></ArrayList><zdef1084405763:ArrayList xmlns:zdef1084405763=""><zdef1084405763:id>123</zdef1084405763:id><zdef1084405763:name>Albert</zdef1084405763:name><zdef1084405763:description>desc</zdef1084405763:description></zdef1084405763:ArrayList>

With Woodstox, I got a javax.xml.stream.XMLStreamException : trying to output second root, ArrayList

My use case seems really simple, I am missing something or is there an issue in Jackson 2.1.0-SNAPSHOT ?

The test

@Test
    public void testListSerialization() {
        SampleResource r1 = new SampleResource();
        r1.setId(123L);
        r1.setName("Albert");
        r1.setDescription("desc");

        SampleResource r2 = new SampleResource();
        r2.setId(123L);
        r2.setName("Albert");
        r2.setDescription("desc");

        List<SampleResource> l = new ArrayList<SampleResource>();
        l.add(r1);
        l.add(r2);

        String result = XmlHelper.serialize(l);
        Assertions.assertThat(result).contains("<id>123</id>");
        Assertions.assertThat(result).contains("<name>Albert</name>");
        Assertions.assertThat(result).contains("<description>desc</description>");
    }

SampleResource :

public class SampleResource {
    private Long id;
    private String name;
    private String description;

    public Long getId() {
        return id;
    }

    public void setId(Long id) {
        this.id = id;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getDescription() {
        return description;
    }

    public void setDescription(String description) {
        this.description = description;
    }
}

XmlHelper :

package org.resthub.web;

import com.fasterxml.jackson.core.type.TypeReference;
import java.io.ByteArrayOutputStream;
import java.io.OutputStream;

import org.resthub.web.exception.SerializationException;

import com.fasterxml.jackson.databind.AnnotationIntrospector;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.introspect.JacksonAnnotationIntrospector;
import com.fasterxml.jackson.dataformat.xml.JacksonXmlModule;
import com.fasterxml.jackson.dataformat.xml.XmlMapper;

/**
 * Helper for XML serialization and deserialization.
 */
public class XmlHelper {

    /**
     * Jackson Object Mapper used to serialization/deserialization
     */
    protected static ObjectMapper objectMapper;

    protected static void initialize() {
        JacksonXmlModule module = new JacksonXmlModule();
        module.setDefaultUseWrapper(false);
        objectMapper = new XmlMapper(module);
        AnnotationIntrospector introspector = new JacksonAnnotationIntrospector();
        objectMapper.setAnnotationIntrospector(introspector);
    }

    /**
     * Return the objectMapper. It can be used to customize serialization/deserialization configuration.
     * @return 
     */
    public ObjectMapper getObjectMapper() {
        if (objectMapper == null)
            initialize();
        return objectMapper;
    }

    /**
     * Serialize and object to an XML String representation
     * @param o The object to serialize
     * @return The XML String representation
     */
    public static String serialize(Object o) {
        if (objectMapper == null)
            initialize();
        OutputStream baOutputStream = new ByteArrayOutputStream();
        try {
            objectMapper.writeValue(baOutputStream, o);
        } catch (Exception e) {
            throw new SerializationException(e);
        }
        return baOutputStream.toString();
    }

    /**
     * Deserialize a XML string
     * @param content The XML String object representation
     * @param type The type of the deserialized object instance
     * @return The deserialized object instance
     */
    public static <T> T deserialize(String content, Class<T> type) {
        if (objectMapper == null)
            initialize();
        try {
            return objectMapper.readValue(content, type);
        } catch (Exception e) {
            throw new SerializationException(e);
        }
    }

    /**
     * Deserialize a XML string
     * @param content The JSON String object representation
     * @param valueTypeRef The typeReference containing the type of the deserialized object instance
     * @return The deserialized object instance
     */
    public static <T> T deserialize(String content, TypeReference valueTypeRef) {
        if (objectMapper == null)
            initialize();
        try {
            return objectMapper.readValue(content, valueTypeRef);
        } catch (Exception e) {
            throw new SerializationException(e);
        }
    }

}

[cowtowncoder]

I will have a look. Most tests so far have focused on Collections (and arrays) as POJO members.
So it is likely that root-level value is an edge-case.

[sdeleuze]

Thanks a lot !

For information, my real life use case is Spring MVC returning a XML collection.
I got the same issue than in this test case.

[cowtowncoder]

Ok good to know.

[cowtowncoder]

Ok. The problem is definitely due to this being a root value, and it is possible that we will have to actually explicitly state that root-level Collections and arrays are not supported. The issue is that modifications that are needed for collection types are handled through POJO property mechanisms, and at root level there is no referring property.

I will leave this issue open just in case we'll figure out a way to make things work, but for now (in 2.0 and 2.1), this use case is not supported.

[sdeleuze]

OK, I hope you will be able to support such functionnality in the future since handling List of elements is a very common use case, at least when you are using Jackson for serialization in REST webservices.

Since it is not supported by JAXB by default, it could make a really good reason for people using Jackson for XML. I know that Spring MVC folks are intersted by that (3.2 release with use Jackson by default but only for json for now)

Is there an existing annotation that we could use to specify the root property, or do you need to create a new one ? If the annotation exists, could it be possible to support this in 2.1.x ? Maybe I can help by contributing a fix with your guideance?

[cowtowncoder]

By root property I simply mean that Collection value needs to be a value of a POJO property. Simple wrapping within (XML) element does not matter. The problem is with the implementation of wrapping (or not) that relies on modifying handling of "bean properties" (BeanPropertyWriter at low level for writing; for reading handling is bit different).
So what is needed for now is simple wrapper object. Spring (et al) could provide such wrappers.

Note that Lists are fully supported as values of POJO properties (and as List elements as well, as long as outermost is a property value).

Ideally root-value lists would be supported, but it would be in 2.2 at earlier just due to work involved.
I mean, unless a simpler work-around is found, which could go in 2.1.x (never say never).

[cowtowncoder]

One last thing: I don't think JAXB supports serialization of Lists directly either. At least I wasn't able to do that for the use case, probably since there is no way to indicate type of elements (i.e. serializes as lists of empty objects).
Although serialization of arrays does work, since arrays retain type unlike Collections (no generic type erasure)..

[sdeleuze]

JAXB with SpringMVC does not support serialization of collections, but JAXB with Jersey does. It don't know how they did that.

Thanks for the tip, I will use plain arrays on my controllers in order to make them work with XML too.

Should I update the title of this issue or create a new issue for root Collection serialization ?

[cowtowncoder]

Updating the title would make sense if you can do it. I guess there is some way to specify full type, since what seemed to fail was simply passing of Collection value type; otherwise Collection itself was serialized correctly.

Interesting bit about Jersey -- would be good to know what they do, even if it does not directly help here.

On Jackson side, I think that with 2.2 we should try to find a better way to deal with Collections in general.
The other current problem is that if the static type is not Collection -- like, when we have plain Object -- same problem exists. One possibility is to add a new hook to allow post-processing non-POJO (de)serializers, similar to how Bean(De)SerializerModifier works. This is necessary for improved Scala support as well.

[sdeleuze]

I updated this issue title.

About Jersey + JAXB support of root level unwrapped collections, when you use it, you have to provide Jersey explicitly the list of classes that will be used in root level collections content, since it can't get the information due to type erasure.

I guess it allows Jersey thanks to collection introspection (and/or JAXB annotations) to find the name of the class and use it as root level XML element. Will try to find the exact source code responsible for this functionality.

[sdeleuze]

Even with plain java arrays, I got the same issues (Trying to output second root exeption when using woodstox for example). Is it expected ?

[tatu-at-salesforce]

From perspective of how things are implemented, yes, unfortunately it is. So Jackson does not have issues with not having all the type info for Collections (it may or may not have, depending); but xml module relies on being able to handle array wrapping via BeanSerializerModifier. And that does not get called unless we have a property.

So no, it should not fail, but yes, it does fail with current version and 2.1.

We have plans to tackle this deficiency for 2.2, but it needs bigger rewrite, and addition of functionality in jackson-databind. But this same (or similar) problem affects Hibernate and Scala modules too, so it is high priority thing now.

[sdeleuze]

I updated the title from "Support non wrapped root level collection serialization" to "Support root level collection serialization", since at root level we need a wrapper, we just want Jackson being able to create it automatically (List = XML root)

[BenGrant]

It will be good if this can happen, i've created a work around where i put the list in a random pojo, just so it can be convert to xml.

[marccarre]

Hi, is 2.2 still the ETA?
It seems still failing in 2.1.1:
Unexpected IOException (of type java.io.IOException): javax.xml.stream.XMLStreamException: Trying to output second root, <ArrayList>
Like sdeleuze, I'm happy to give a hand with your guidance, provided it would be actually helpful to you.
FYI: end goal is to implement a multi-format REST service and avoid duplicated endpoints for XML and JSON
Cheers,
M.

[tatu-at-salesforce]

Yes. The problem is that since List is not referenced via POJO property, current mechanism can not add expected wrapping. It is possible to use root element wrapping to avoid the "single root element" problem, but that will probably not solve the problem completely (since serializer/deserializer handler won't be invoked anyway).

Master branch is now open for 2.2, so it is possible to start solving this, starting with adding hook(s) needed.

[MordechaiTamam]

Hi all,
I just want to mention that XStream supports serialization of collections and arrays quite nicly.

Any way , what is the status of this? I'm using 2.2 version is there a way to achieve something like XStream collection/array serialization?

[cowtowncoder]

@MordechaiTamam You need to wrap Collections in a POJO for things to work: they work at any other level, just not as root values. XStream does support this (partly since XML is its only focus; and partly because it can freely decide on output structure, whereas JAXB/Jackson try to allow customization of both Objects and external XML representation), but that's neither here nor there. Although it is good to know if someone can not use wrapper POJO.

[cowtowncoder]

Lo and behold -- after spending quite a while playing with code, I think I managed to actually implement this feature; unit test now passes.

All entries will be serialized using container element item. Actual name to use could probably be auto-detected for some cases (it's bit tricky as type information may or may not be available); or could add a config method for specifying it. But I don't know if it really matters.

I also fixed some oddities with existing "root name" configuration that comes from jackson-databind -- it really wasn't working well at all for 2.2; and will make more sense for 2.3.
I did realize one short-coming with it: there is no way currently to specify namespace. Will need to think about what (if anything) to do about that (feel free to file separate RFE for that if it seems important).

[MordechaiTamam]

You are the man...
Thanks a lot.
A couple of things:

1. The project wont compile when I pull it (missing com.fasterxml.jackson.core.json.PackageVersion import in XmlMapper,XmlFactory, JacksonXmlModule and XmlSerializerProvider._serializeNull does not override one of the ancestors method).
2. One of the tests fails (TestIndentation.testWithAttr(com.fasterxml.jackson.dataformat.xml.TestIndentation): expected:<...AttrBean2 count="3">[(..))
3. Regarding the item issue, it matters. It will be great if it will be configurable .
   In my case, I'm serializing UserDTO. A single entity serialization will result with:
   < UserDTO >< id/ >< email >testuser_100@shunra.co.il < /email >< / UserDTO >

and serializing a collection results with this:
< >
< item >
< id/ >
< email >testuser_100@shunra.co.il< /email >
< /item >
< item >
< id/ >
< email >testuser_101@shunra.co.il< /email >
< /item >
< / >
Now, I will have to explain my clients that in case of a collection serialization, they will refer the "item" element as the user.
It will be great if instead of item there will be UserDTO.
4)Why does the root element name is missing? You can set it to "items" by default and if you decide to make the "item" configurable, let say we're replacing "item" with "user" elements, the root element of the collection should be "users".
5) where can I get jackson-dataformat-xml:2.3.0-SNAPSHOT jar?

[MordechaiTamam]

Another thing is, that I can not deserialize the generated XML .
I'm generating an XML:

Integer[] ints = new Integer[] { 1, 2, 3 };
String content = xmlMapper.writeValueAsString(ints);
Which results with:
< Integer[] >
< item > 1 < /item >
< item > 2 < /item >
< item > 3 < /item >
< /Integer[] >

Than I'm removing the Integer[]:
< >
< item > 1 < /item >
< item > 2 < /item >
< item > 3 < /item >
< / >

and Converting back to array of integers:
Integer[] Array = xmlMapper.convertValue(content, new TypeReference<Integer[]>() {});

results with:
java.lang.ClassCastException: com.fasterxml.jackson.databind.util.TokenBuffer cannot be cast to com.fasterxml.jackson.dataformat.xml.ser.ToXmlGenerator
at com.fasterxml.jackson.dataformat.xml.ser.XmlSerializerProvider._initWithRootName(XmlSerializerProvider.java:185)
at com.fasterxml.jackson.dataformat.xml.ser.XmlSerializerProvider.serializeValue(XmlSerializerProvider.java:71)
at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2593)
at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2558)
at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2549)

I'm using the next versions:
jackson-dataformat-xml-2.3.0-SNAPSHOT
woodstox-core-asl-4.2.0
jackson-databind-2.3.0-SNAPSHOT
jackson-core-2.3.0-SNAPSHOT
jackson-annotations-2.3.0-SNAPSHOT

[MordechaiTamam]

I just want to say that I'm not able to deserialize nothing with xmlMapper.convertValue(content, new TypeReference() {});
and I'm able to convert single POJOs with : xmlMapper.readValue(xml, Class);
Still, I can not deserialize collections/arrays.

[cowtowncoder]

@MordechaiTamam sounds like you got through the compilation issue -- it does require you to re-build jackson-databind.

Problem with conversion is unrelated, but worth filing another bug about. As a work-around, you can use standard ObjectMapper so it does not try to apply XML-specific work-arounds which trigger the issue.

It also sounds like array-handling is not working as well as Collections. I will add a unit test for this.

Naming part is difficult, as I said, and my main concern is this: while serialization of data is possible to do without generic type information, determination of root name to use is not. There simply isn't anything to go about.
And while root type may be provided explicitly, most users do not do that: they just pass a List instance.
One possibility would be to check out type of the first element; that has its problems in case contents are heterogenous, but I guess it would work for common cases.

[cowtowncoder]

I added a test for array case (SampleResource[]). This did expose issues with naming, and I fixed that part a bit, as XML does not allow use of brackets in names. Instead, I added a trivial pluralization; and with that, modified test passes. I also filed bug #71 for conversion/TokenBuffer issues.

[cowtowncoder]

Ok: convertValue() should now work better, as per fix for #71.

[MordechaiTamam]

Hi again,
I'm trying to deserialize using the convertValue method:

this:
public static < T > T fromXml(String xml, Class < T > clazz) {
XmlMapper xmlMapper = AppContextAware.getBean(XmlMapper.class);
try {
return xmlMapper.convertValue(xml, clazz);
} catch (Exception e) {
throw new RuntimeException(e);
}
}

results with:

Caused by: java.lang.ClassCastException: com.fasterxml.jackson.databind.util.TokenBuffer$Parser cannot be cast to com.fasterxml.jackson.dataformat.xml.deser.FromXmlParser
at com.fasterxml.jackson.dataformat.xml.deser.WrapperHandlingDeserializer._configureParser(WrapperHandlingDeserializer.java:140)
at com.fasterxml.jackson.dataformat.xml.deser.WrapperHandlingDeserializer.deserialize(WrapperHandlingDeserializer.java:108)
at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2628)
at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2559)

Am I missing something here?

[cowtowncoder]

Looks like handling worked for root-level Lists and arrays (for which I added tests). But case of list-as-property had additional work-arounds; I added a test, and resolved problems there. So now conversions should work better.
I also created issue #71 to cover convertValue() issues specifically, so let's use that for any follow-up work -- in the meantime, regular vanilla ObjectMapper can do all object conversions so we can focus on resolving use case of Lists.

Also note that I could not verify sample code above since I do not know what it was called with.

[juzerali]

Seems this issue got fixed for XML, did it got fixed for JSON too?

[cowtowncoder]

@juzerali I don't think this ever affected JSON; it is XML specific.