Array element and field token spans include previous comma

[latkin]

Consider JSON blob

[1,    2,    3]

The spans from getTokenLocation to getCurrentLocation when sitting on each VALUE_NUMBER_INT are currently

1
,    2
,    3

I would expect them to each be 1-char wide, i.e.

1
2
3

Is this expected/per-spec?

[cowtowncoder]

That does not look right.

getTokenLocation should point to the first character that is logically part of the token (first digit in this case of numbers). Current location is less well defined, and it can only be relied on giving useful information for error reporting, mostly showing where an unexpected input character was found. During normal operation, current location should point to the character past end of the logical token, as long as lazy parsing is not affecting the result: for example, JSON Strings are not fully decoded until they are actually requested. But additionally current location will be unlikely to give end location of property names, because parsing may eagerly decoded part of the value token that follows.

So, first things first: getTokenLocation() seems wrong if it points to leading comma.

[latkin]

@cowtowncoder thanks for the confirmation. Here's a repro in Scala (using Jackson 2.6.3).

val json = "[1,     2,     3]"
val parser = new JsonFactory().createParser(json)
while(parser.nextToken() != null) {
  val tok = parser.getCurrentToken
  val tokStart = parser.getTokenLocation.getCharOffset.toInt
  val currLoc  = parser.getCurrentLocation.getCharOffset.toInt
  val slice = json.substring(tokStart, currLoc)

  println(f"$tok%-20s Start: $tokStart%2d End: $currLoc%2d Slice: {$slice}")
}

Output:

START_ARRAY          Start:  0 End:  1 Slice: {[}
VALUE_NUMBER_INT     Start:  1 End:  2 Slice: {1}
VALUE_NUMBER_INT     Start:  2 End:  9 Slice: {,     2}
VALUE_NUMBER_INT     Start:  9 End: 16 Slice: {,     3}
END_ARRAY            Start: 16 End: 17 Slice: {]}

[latkin]

And for reference, the corresponding behavior for object fields also occurs, and is mentioned here #37 (comment)

[cowtowncoder]

@latkin Thank you! Field name case was known already, and can hopefully be fixed for 2.7.0. Array element one is new to me.

[cowtowncoder]

Yes, I can reproduce this, and can see where it goes wrong, as well as how it can be fixed.
Now the only (?) question is how to coordinate fixing this, along with fix for #37.