Json encode / decode questions

Mark

Apologies for such a basic question. This is the first time I've used the json.encode / json.decode functions, and I've clearly missed something basic. Though the help document indicates that the encode statement will take any object that implements the metamethod, it gives no example of how to do so. I've searched Jason on the forum, and also did not see an example.

I tried implementing as I have simple things like a change in _index, but it doesn't seem to be effective as I'm still getting an error stating encode won't that "user data."

So,,, someone give me a smack and explain the blindingly obvious thing I missed. Thanks.

yojimbo2000

My entry to the gamejam uses json for saving levels. Have a look at the Level IO tab in this:

https://gist.github.com/Utsira/8a364a35f397f26aefad

Mark

@yojimbo2000 Thanks, but it looks as if you're encoding simple tables. That works fine for me, as well. My question is implementing the __tojson method within a complex object so that it can properly encoded. I can turn the object into the save string I've been using to file them away, or a set of key-value pairs and encode that, but there must be a neater way. After all, that seems like the work I expect out of encode / decode.

yojimbo2000

Oh I see. You mean adding json encode/decode support to, say, a vec2? Can you be more specific about which objects you want to encode?

Mark

These are classes I created in the current application (in this case, a class called "Location" which is actually a work location within an office / work site and contains a number of fields, including a series of coordinates).

However, just an example of how to implement __tojson so that it supports json.encode would be fine. Even on something like a Vec2. All I can find is the notation in the help file that indicates .encode supports any object that implements the metamethod __json encode, but I can't find any examples here or elsewhere showing what that implementation looks like.

Thanks.

yojimbo2000

So you're trying to encode the instances of the location class? If they're held in a table, you can encode that table, unless it contains "user data" (ie vectors, matrices etc). I'd also like to know how to implement json encoding for vectors etc. Do you know specifically within the location class, what part of it is causing the "user data" error? What I'm saying is, I don't think you need to implement a custom json encode for your location class as such, you just need json encoding for the objects contained within it (from your description, vectors).

yojimbo2000

Eg. Instances of this class can be encoded, as long as they don't contain "user data" (vectors).

--# Main
-- Json Test

-- Use this function to perform your initial setup
function setup() --cant use vec2s

    objects={
    MyClass(17), MyClass(23), MyClass(9)
    }

    tabSave()
    tabLoad()
end

function tabLoad()
    str=readProjectTab("Test")
    tab=json.decode(string.sub(str,3,-3))
    for k,v in pairs(tab) do
        print(v)
    end
   -- tab[1]:draw() --doesnt work
end

function tabSave()
     str=json.encode(objects)
    saveProjectTab("Test", "--"..str.."--")
end

--# MyClass
MyClass = class()

function MyClass:init(x)
    -- you can accept and set parameters here
    self.foo = "bar"
    self.x=x
    -- self.pos=vec2(50,50) --vec2 cant be used with json
end

function MyClass:draw()
    print (self.foo)
end

function MyClass:touched(touch)
    -- Codea does not automatically call this method
end

--# Test
--[{"x":17,"foo":"bar"},{"x":23,"foo":"bar"},{"x":9,"foo":"bar"}]--

dave1707

I'm not sure the encode, decode works properly, or else I'm wrong in what I'm expecting. In the example below, table tab1 is created. I encode it and print the result. I then decode it into table tab2. I encode it and print the result so I can compare it to the original table. The results are the same, but maybe out of order which shouldn't matter. I can print values from tab1, but I get errors when I try to print it from tab2.

function setup()
    tab1={a=123,b=456,{3,4,5}}
    z=json.encode(tab1)
    print(z)

    tab2=json.decode(z)
    z=json.encode(tab2)
    print(z)

    print("\ntab1 --------")
    for a,b in pairs(tab1) do
        print(a,b)
    end

    print("\ntab2 --------")
    for a,b in pairs(tab2) do
        print(a,b)
    end

    print("\ntab1 --------")
    print(tab1[1])
    print(tab1[1][1],tab1[1][2],tab1[1][3])

    print("\ntab2 --------")
    print(tab2[1])
    print(tab2[1][1],tab2[1][2],tab2[1][3])    
end    

juce

@dave1707, tab2 doesn't have an integer key 1, but a string "1". That's because JSON only allows string keys, so when you encode/decode tab1 into tab2, the [1] = {3,4,5} becomes: ["1"] = {3,4,5}. You can see that in the first print of z.

yojimbo2000

@dave1707 mixing key-values with an array is a terrible idea IMO. You shouldn't expect to be able to index it by number.

yojimbo2000

@Mark instead of adding the tojson method to whatever object you need encoding, how about using the "exception" field to catch the parts of the class that are causing json to fall over. If it's just vec2s that should be quite straight forward. Can you show us the code for the class whose instances you're trying to encode?

yojimbo2000

@Mark here's my example from above, but using the "exception" field to catch the vec2s and convert them into a table/json object: "pos":{"x":9.0,"y":50.0}. I haven't worked out how to do the equivalent decode though yet. I need to work out how to indictate that this is a type vec2.

Edit: v1.2 more elegant jsonException class, now indicates that table-object is a vec2 "pos":{"vec2":{"x":9.0,"y":50.0}}. No decoding of vec2s yet.

--# Main
-- Json Test

-- Use this function to perform your initial setup
function setup() 

    objects={
    MyClass(17), MyClass(23), MyClass(9)
    }

    tabSave()
    tabLoad()
end

function tabLoad()
    local str=readProjectTab("Test")
    local tab=json.decode(string.sub(str,3,-3))
    tabIterate(tab, "")
   -- tab[1]:draw() --doesnt work
end

function tabIterate(tab, indent)
    for k,v in pairs(tab) do
        if type(v)=="table" then print(indent..k.."=") tabIterate(v, indent.."  ") else print(indent..k,v) end
    end
end

function tabSave()
     str=json.encode(objects, {exception = jsonException})
    saveProjectTab("Test", "--"..str.."--")
end

function jsonException(reason, v, state, message)
    if v.x and v.y and not v.z then --vec2
        return json.encode({vec2={x=v.x,y=v.y}})
    end
end

--# MyClass
MyClass = class()

function MyClass:init(x)
    -- you can accept and set parameters here
    self.foo = "bar"
    self.x=x
    self.pos=vec2(x,50) --vec2 cant be used with json without an exception class
end

function MyClass:draw()
    print (self.foo)
end

function MyClass:touched(touch)
    -- Codea does not automatically call this method
end

--# Test
--[{"pos":{"vec2":{"x":17.0,"y":50.0}},"foo":"bar","x":17},{"pos":{"vec2":{"x":23.0,"y":50.0}},"foo":"bar","x":23},{"pos":{"vec2":{"x":9.0,"y":50.0}},"foo":"bar","x":9}]--

yojimbo2000

Here's a version that attempts to decode the vec2s. It's not particularly elegant, as there's no corresponding equivalent to the exception function in json.decode (this is where implementing the __tojson meta method might be the way to go, as @Mark said in his original enquiry).

Edit: here's a working version of the code

--# Main
-- Json Test

-- Use this function to perform your initial setup
function setup()

    objects={
    MyClass(17), MyClass(23), MyClass(9)
    }

    tabSave()
    tabLoad()
end

function tabLoad()
    local str=readProjectTab("Test")
    local result=json.decode(string.sub(str,4,-2))
    result=tabIterate(result)
    result=tabIterate(result) --re-iterate to see if decoding has stuck
end

function tabSave()
    str=json.encode(objects, {exception = jsonException, indent = true})
    saveProjectTab("Test", "--["..str.."]")
end

local underscore = {"=","~","-","."} --underscore characters indicate table depth

function tabIterate(tab, i)
    local new = {}
    local i = i or 0
    local indent = string.rep(" ", i*3) --depth of indentation
    for k,v in pairs(tab) do
        if type(v)=="table" then
            if v.vec2 then --convert back to vec2
                new[k] = vec2(v.x, v.y)
                print (indent..k..": decoding vec2"..tostring(new[k]))
            else
                local under = string.rep(underscore[math.min(#underscore, i+1)], string.len(k))
                print(indent..k.."\n"..indent..under)
                new[k]=tabIterate(v, i+1) --increase indent upon recursion
            end
        else
            if type(v)=="userdata" and v.x and v.y then --check whether vec2 decoding has worked
                print(indent..k..": vec2"..tostring(v))
            else
                print(indent..k..": "..v)
            end
            new[k]=v
        end
    end
    return new
end

function jsonException(reason, v, state, message)
    if v.x and v.y and not v.z then --vec2
        return json.encode({x=v.x,y=v.y,vec2=true}) --set a vec2 flag
    end
end

--# MyClass
MyClass = class()

function MyClass:init(x)
    -- you can accept and set parameters here
    self.foo = "bar"
    self.x=x
    self.pos=vec2(x,50) --vec2 cant be used with json without an exception class
end

function MyClass:draw()
    print (self.foo)
end

function MyClass:touched(touch)
    -- Codea does not automatically call this method
end

--# Test
--[[{
    "foo":"bar",
    "pos":{"vec2":true,"y":50.0,"x":17.0},
    "x":17
  },{
    "foo":"bar",
    "pos":{"vec2":true,"y":50.0,"x":23.0},
    "x":23
  },{
    "foo":"bar",
    "pos":{"vec2":true,"y":50.0,"x":9.0},
    "x":9
  }]]

Jmv38

here is how to implement _toJjson to vec2

--# Main
-- Json Test

-- Use this function to perform your initial setup
function setup() --cant use vec2s
    objects={
    MyClass(17), MyClass(23), MyClass(9)
    }
    Vec2ToJson()
    tabSave()
    out = tabLoad()
    setmetatable(out,{__tostring = function(obj)
        return json.encode(obj,{indent=true}) 
        end})
    print(out)
end

function Vec2ToJson()
    local meta = getmetatable(vec2())
    meta.__tojson = function(t)
        local a = {x=t.x,y=t.y}
        return json.encode(a)
    end
end

function tabLoad()
    local str=readProjectTab("Test")
    str = string.sub(str,4,-4)
    return json.decode(str)
end

function tabSave()
     local str=json.encode(objects,{indent=true})
    saveProjectTab("Test", "--["..str.."]--")
end


--# MyClass
MyClass = class()

function MyClass:init(x)
    -- you can accept and set parameters here
    self.foo = "bar"
    self.x=x
    self.pos=vec2(50,50) --vec2 cant be used with json
end

function MyClass:__tostring()
    return json.encode(self,{indent=true}) 
end



--# Test
--[[{
    "foo":"bar",
    "pos":{"x":50.0,"y":50.0},
    "x":17
  },{
    "foo":"bar",
    "pos":{"x":50.0,"y":50.0},
    "x":23
  },{
    "foo":"bar",
    "pos":{"x":50.0,"y":50.0},
    "x":9
  }]]--

yojimbo2000

@Jmv38 that's awesome, thanks. Though might I suggest adding a vec2=true flag to the json-vec2 so that the decode process can convert it back to a genuine vec2 (rather than leaving it as a pseudo-vec2 table)? Unless someone can think of a more elegant way to handle the decode?

Jmv38

@yojimbo2000 i agree. I had not enough time to rework and post your last version, so i just posted this one to show you how to use __toJason (as you asked for it earlier). Thanks.

yojimbo2000

Ok, here's my version using @Jmv38 's meta methods. It now supports vec2, vec3, vec4, matrix, with support for conversion back to the proper types. In the code below I show it recreating the instances of the class in the objects table from the saved code.

--# Main
-- Json Test

-- Use this function to perform your initial setup
function setup()
    VecToJson() --establish metamethods for encoding vec2, vec3, vec4, matrix as json objects
    objects={
    MyClass{pos=vec3(17,18,19), angle=45, foo="bar"}, 
    MyClass{pos=vec4(123,124,127,1), angle=0, marzipan="tasty", foo="boo"}, 
    MyClass{pos=vec3(257,263,278), angle=-90, boolean=true, foo="bah"}
    }

    for i=1,2 do --do this twice to check back-conversion
        tabSave(objects, "Test")
        result=tabLoad("Test")
        result=tabIterate(result) --decode json objects back to vec2, vec3, vec4
        result=tabIterate(result) --re-iterate to see if decoding has stuck
        objects = {} --clear objects
        for i,v in ipairs(result) do
            objects[i]=MyClass(v) --recreate them from loaded table
        end
    end --rinse and repeat
end

function VecToJson()
    local meta = getmetatable(vec2())
    meta.__tojson = function(t)
        return json.encode({x=t.x,y=t.y,vec2=true})
    end
    meta = getmetatable(vec3())
    meta.__tojson = function(t)
        return json.encode({x=t.x,y=t.y,z=t.z,vec3=true})
    end
    meta = getmetatable(vec4())
    meta.__tojson = function(t)
        return json.encode({x=t.x,y=t.y,z=t.z,w=t.w,vec4=true})
    end
    meta = getmetatable(matrix())
    meta.__tojson = function(t)
        return json.encode({values={t[1],t[2],t[3],t[4], t[5],t[6],t[7],t[8], t[9],t[10],t[11],t[12], t[13],t[14],t[15],t[16]},mat=true}) --a matrix.unpack function would've been handy here!
    end

end

function tabSave(tab, loc)
    local str=json.encode(tab, {indent = true}) --exception = jsonException, 
    saveProjectTab(loc, "--["..str.."]")
end

function tabLoad(loc)
    local str=readProjectTab(loc)
    return json.decode(string.sub(str,4,-2))
end

local underscore = {"=","~","-","."} --underscore characters indicate table depth

function tabIterate(tab, i)
    local new = {}
    local i = i or 0
    local indent = string.rep(" ", i*3) --depth of indentation
    for k,v in pairs(tab) do
        if type(v)=="table" then
            if v.vec2 then --convert back to vec2
                new[k] = vec2(v.x, v.y)
                print (indent..k..": decoding vec2"..tostring(new[k]))
            elseif v.vec3 then --convert back to vec3
                new[k] = vec3(v.x, v.y, v.z)
                print (indent..k..": decoding vec3"..tostring(new[k]))
            elseif v.vec4 then --convert back to vec4
                new[k] = vec4(v.x, v.y, v.z, v.w)
                print (indent..k..": decoding vec4"..tostring(new[k]))
            elseif v.mat then --convert back to matrix
                new[k] = matrix(table.unpack(v.values)) 
                print (indent..k..": decoding matrix"..tostring(new[k]))
            else --genuine table
                local under = string.rep(underscore[math.min(#underscore, i+1)], string.len(k))
                print(indent..k.."\n"..indent..under)
                new[k]=tabIterate(v, i+1) --increase indent upon recursion
            end
        else
            print(indent..k..": "..tostring(v))         
            new[k]=v
        end
    end
    return new
end

--[[ --not needed when using meta-methods
function jsonException(reason, v, state, message)
    if v.x and v.y and not v.z then --vec2
        return json.encode({x=v.x,y=v.y,vec2=true}) --set a vec2 flag
    end
end
  ]]

--# MyClass
MyClass = class()

function MyClass:init(t)
    for k,v in pairs(t) do
        self[k]=v
    end
    self:draw()
end

function MyClass:draw() --some pseudo drawing
    translate(self.pos.x, self.pos.y, self.pos.z)
    rotate(self.angle)
    self.matrix=modelMatrix()
    print(self.foo)
    resetMatrix()
end

--# Test
--[[{
    "angle":45,
    "foo":"bar",
    "pos":{"y":18.0,"x":17.0,"vec3":true,"z":19.0},
    "matrix":{"values":[0.70710676908493,0.70710676908493,0.0,0.0,-0.70710676908493,0.70710676908493,0.0,0.0,0.0,0.0,1.0,0.0,17.0,18.0,19.0,1.0],"mat":true}
  },{
    "foo":"boo",
    "angle":0,
    "matrix":{"values":[1.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,1.0,0.0,123.0,124.0,127.0,1.0],"mat":true},
    "pos":{"y":124.0,"x":123.0,"w":1.0,"vec4":true,"z":127.0},
    "marzipan":"tasty"
  },{
    "pos":{"y":263.0,"x":257.0,"vec3":true,"z":278.0},
    "angle":-90,
    "foo":"bah",
    "boolean":true,
    "matrix":{"values":[-4.3711388286738e-08,-1.0,0.0,0.0,1.0,-4.3711388286738e-08,0.0,0.0,0.0,0.0,0.99999994039536,0.0,257.0,263.0,278.0,1.0],"mat":true}
  }]]

Jmv38

you are faster than me! I was preparing this: a version to manage any class with "Class" keyword. How do you like it?

--# Main
-- Json Test

-- Use this function to perform your initial setup
function setup() --cant use vec2s
    objects={
    MyClass(17), MyClass(23), MyClass(9)
    }
    Vec2ToJson()
    Vec3ToJson()
    tabSave()
    out = tabLoad()
    setmetatable(out,{__tostring = function(obj)
        return json.encode(obj,{indent=true}) 
        end})
    print(out[1].pos)
    print(out[1].mis)
end

function Vec2ToJson()
    local meta = getmetatable(vec2())
    meta.__tojson = function(t)
        local a = {Class="vec2",x=t.x,y=t.y}
        return json.encode(a)
    end
end
function Vec3ToJson()
    local meta = getmetatable(vec3())
    meta.__tojson = function(t)
        local a = {Class="vec3",x=t.x,y=t.y,z=t.z}
        return json.encode(a)
    end
end

function tabLoad()
    local str=readProjectTab("Test")
    str = string.sub(str,4,-4)
    local t = json.decode(str)
    t = checkForClass(t)
    return t
end

function checkForClass(t)
    if type(t) ~= "table" then
        return t
    end
    for key,val in pairs(t) do
        if type(val) == "table" then
            t[key] = checkForClass(val)
        end
        local Class = t["Class"] 
        if Class then
            local obj = _G[Class]()
            for k,v in pairs(t) do
                if k~="Class" then
                    obj[k] = v
                end
            end
            t = obj
        end
    end
    return t
end

function tabSave()
     local str=json.encode(objects,{indent=true})
    saveProjectTab("Test", "--["..str.."]--")
end


--# MyClass
MyClass = class()

function MyClass:init(x)
    -- you can accept and set parameters here
    self.foo = "bar"
    self.x=x
    self.pos=vec2(50,50) 
    self.mis =vec3(1,2,3) 
end

function MyClass:__tostring()
    return json.encode(self,{indent=true}) 
end



--# Test
--[[{
    "x":17,
    "mis":{"x":1.0,"y":2.0,"z":3.0,"Class":"vec3"},
    "pos":{"x":50.0,"y":50.0,"Class":"vec2"},
    "foo":"bar"
  },{
    "x":23,
    "mis":{"x":1.0,"y":2.0,"z":3.0,"Class":"vec3"},
    "pos":{"x":50.0,"y":50.0,"Class":"vec2"},
    "foo":"bar"
  },{
    "x":9,
    "mis":{"x":1.0,"y":2.0,"z":3.0,"Class":"vec3"},
    "pos":{"x":50.0,"y":50.0,"Class":"vec2"},
    "foo":"bar"
  }]]--

Jmv38

i found this more general than vec2=true.

yojimbo2000

@Jmv38 Ok. Actually, I think my preferred method would be, rather than a keyword, nesting the data in a sub table with an appropriate key, eg a vec3 called pos looks like this: "pos":{"vec3":{"y":18.0,"x":17.0,"z":19.0}}. Edit, added exception method, has feature parity with meta-method:

--# Main
-- Json Test

-- Use this function to perform your initial setup
function setup()
   -- VecToJson() --establish metamethods for encoding vec2, vec3, vec4, matrix as json objects
    objects={
    MyClass{pos=vec3(17,18,19), angle=0, foo="bar"}, 
    MyClass{pos=vec2(123,124), angle=-45, marzipan="tasty", foo="boo"}, 
    MyClass{pos=vec3(257,263,278), angle=-90, boolean=true, foo="bah"}
    }

    for i=1,2 do --do this twice to check back-conversion
        tabSave(objects, "Test")
        local loadBack=tabLoad("Test")
        loadBack=tabIterate(loadBack) --decode json objects back to vec2, vec3, vec4
        print("---------------------\nDECODING COMPLETE, CHECKING\n---------------------")
        loadBack=tabIterate(loadBack) --re-iterate to see if decoding has stuck
        print("=====================\nRECREATING OBJECTS FROM DECODED DATA, CHECKING\n=====================")
        objects = {} --clear objects
        for i,v in ipairs(loadBack) do
            objects[i]=MyClass(v) --recreate them from loaded table
        end
    end --rinse and repeat
end

--meta-methods
function VecToJson()
    local meta = getmetatable(vec2())
    meta.__tojson = function(t)
       return json.encode({vec2={x=t.x,y=t.y}})
    end
    meta = getmetatable(vec3())
    meta.__tojson = function(t)
       return json.encode({vec3={x=t.x,y=t.y,z=t.z}})
    end
    meta = getmetatable(vec4())
    meta.__tojson = function(t)
        return json.encode({vec4={x=t.x,y=t.y,z=t.z,w=t.w}})
    end
    meta = getmetatable(matrix())
    meta.__tojson = function(t)
    return json.encode({mat4={t[1],t[2],t[3],t[4], t[5],t[6],t[7],t[8], t[9],t[10],t[11],t[12], t[13],t[14],t[15],t[16]}}) --a matrix.unpack function would've been handy here!
    end   
end

 --encode exception func can be used instead of meta-methods
function jsonException(reason, t, state, message)
    if t.x and t.y and t.z and t.w then --vec4
        return json.encode({vec4={x=t.x,y=t.y,z=t.z,w=t.w}})
    elseif t.x and t.y and t.z then --vec3
        return json.encode({vec3={x=t.x,y=t.y,z=t.z}})
    elseif t.x and t.y then --vec2
        return json.encode({vec2={x=t.x,y=t.y}})
    elseif t[16] then --matrix, presumably. nb #t not allowed. what other "userdata" types are there?     
        return json.encode({mat4={t[1],t[2],t[3],t[4], t[5],t[6],t[7],t[8], t[9],t[10],t[11],t[12], t[13],t[14],t[15],t[16]}}) --a matrix.unpack function would've been handy here!    
    end
end

function tabSave(tab, loc)
    local str=json.encode(tab, {indent = true, exception = jsonException}) 
    saveProjectTab(loc, "--["..str.."]\n")
end

function tabLoad(loc)
    local str=readProjectTab(loc)
    return json.decode(string.sub(str,4,-3))
end

local underscore = {"=","~","-","."} --underscore characters indicate table depth

function tabIterate(tab, i)
    local new = {}
    local i = i or 0
    local indent = string.rep(" ", i*3) --depth of indentation
    for k,v in pairs(tab) do
        if type(v)=="table" then --check first whether its a pseudo-table representing a vec2, vec3, vec4, mat4
            if v.vec2 then --convert back to vec2
                local vec = v.vec2
                new[k] = vec2(vec.x, vec.y)
                print (indent..k..": decoding vec2"..tostring(new[k]))
            elseif v.vec3 then --convert back to vec3
                local vec = v.vec3
                new[k] = vec3(vec.x, vec.y, vec.z)
                print (indent..k..": decoding vec3"..tostring(new[k]))
            elseif v.vec4 then --convert back to vec4
                local vec = v.vec4
                new[k] = vec4(vec.x, vec.y, vec.z, vec.w)
                print (indent..k..": decoding vec4"..tostring(new[k]))
            elseif v.mat4 then --convert back to matrix
                new[k] = matrix(table.unpack(v.mat4)) 
                print (indent..k..": decoding matrix"..tostring(new[k]))
            else --genuine table
                local under = string.rep(underscore[math.min(#underscore, i+1)], string.len(k))
                print(indent..k.."\n"..indent..under)
                new[k]=tabIterate(v, i+1) --increase indent upon recursion
            end
        else
            print(indent..k..": "..tostring(v))         
            new[k]=v
        end
    end
    return new
end

--# MyClass
MyClass = class()

function MyClass:init(t)
    for k,v in pairs(t) do
        self[k]=v
    end
    self:draw()
end

function MyClass:draw() --some pseudo drawing
    translate(self.pos:unpack()) --(self.pos.x, self.pos.y, self.pos.z)
    rotate(self.angle)
    self.matrix=modelMatrix()
    print(self.foo)
    resetMatrix()
end

--# Test
--[[{
    "pos":{"vec3":{"y":18.0,"z":19.0,"x":17.0}},
    "angle":0,
    "matrix":{"mat4":[1.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,1.0,0.0,17.0,18.0,19.0,1.0]},
    "foo":"bar"
  },{
    "foo":"boo",
    "matrix":{"mat4":[0.70710676908493,-0.70710676908493,0.0,0.0,0.70710676908493,0.70710676908493,0.0,0.0,0.0,0.0,1.0,0.0,123.0,124.0,0.0,1.0]},
    "pos":{"vec2":{"y":124.0,"x":123.0}},
    "angle":-45,
    "marzipan":"tasty"
  },{
    "foo":"bah",
    "angle":-90,
    "boolean":true,
    "pos":{"vec3":{"y":263.0,"z":278.0,"x":257.0}},
    "matrix":{"mat4":[-4.3711388286738e-08,-1.0,0.0,0.0,1.0,-4.3711388286738e-08,0.0,0.0,0.0,0.0,0.99999994039536,0.0,257.0,263.0,278.0,1.0]}
  }]]

Jmv38

Good idea. However, it means you cant have a field in a table with the same name as an existing userdata. This forbids many keywords!

yojimbo2000

Ok, I see what you mean. How about I preface each keyword ie "uservec4", "usermat4". Edit: this method is handy with matrices where adding the keyword ruins the array (table.unpack stops working).

Is there a comprehensive list somewhere of all the kinds of "userdata" ? I guess it's all the extra non-Lua APIs that Codea provides, eg Mesh, physics.body etc. I wonder how far this could be taken? Would be quite cool to be able to save a physics body or a mesh in this way. Will need to check whether you can iterate through all the values of a physics body or a mesh using pairs...

Mark

Wow. I looked a way for a bit, and when I looked back, you two had beaten the problem into submission.

Thanks, @jmv38 and @yojimbo2000. Exactly what I wanted to see.

yojimbo2000

You can't use pairs on a mesh or a physics body, so you'd have to go through all the keys by hand. That's not too bad I guess, a cut-and-paste job...

yojimbo2000

@Mark you're welcome. I learned a lot. I'm new to meta-methods, don't really use them in my own code. Depending on the complexity of the object, I think an encode exception function might be simpler (ie my last post above). If you have lots of different kinds of objects though, meshes, physics bodies etc, then meta methods is probably the way to go

yojimbo2000

@Jmv38 or how about 2 underscores before each special key, eg __mat4?

Jmv38

great idea! I think you've nailed it.

yojimbo2000

On a related note, I'm having issues here: http://codea.io/talk/discussion/6679/how-do-i-get-the-meta-table-of-a-physics-body#latest
@Jmv38 any ideas?

Jmv38

here is a possible solution. It is not 'finished' but you get the idea.

--# Main
-- Json Test

-- Use this function to perform your initial setup
function setup() --cant use vec2s
    objects={
    MyClass(100,500), MyClass(200,500), MyClass(400,500)
    }
    Vec2ToJson()
    Vec3ToJson()
    body2ToJson()
    tabSave()
    print("tap screen to load saved data")
end
function draw()
    background(0)
    for i,o in pairs(objects) do
        o:draw()
    end
end
function touched(t)
    if t.state==BEGAN then
        for i,o in pairs(objects) do
            o.body:destroy()
        end
        objects = tabLoad()
    end
end
function body2ToJson()
    local b = physics.body(CIRCLE,10)
    local meta = getmetatable(b)
    b:destroy()
    meta.__tojson = function(t)
        local a = {Class="Body",x=t.x,y=t.y}
        return json.encode(a)
    end
    Body = function()
        return physics.body(CIRCLE,10)
    end
end
function Vec2ToJson()
    local meta = getmetatable(vec2())
    meta.__tojson = function(t)
        local a = {Class="vec2",x=t.x,y=t.y}
        return json.encode(a)
    end
end
function Vec3ToJson()
    local meta = getmetatable(vec3())
    meta.__tojson = function(t)
        local a = {Class="vec3",x=t.x,y=t.y,z=t.z}
        return json.encode(a)
    end
end

function tabLoad()
    local str=readProjectTab("Test")
    str = string.sub(str,4,-4)
    local t = json.decode(str)
    t = checkForClass(t)
    return t
end

function checkForClass(t)
    if type(t) ~= "table" then
        return t
    end
    for key,val in pairs(t) do
        if type(val) == "table" then
            t[key] = checkForClass(val)
        end
        local Class = t["Class"] 
        if Class then
            local obj 
                if Class == "MyClass" then
                    obj = MyClass(t.body.x,t.body.y)
                else
                    obj = _G[Class]()
                end
            for k,v in pairs(t) do
                if k~="Class" then
                    obj[k] = v
                end
            end
            t = obj
        end
    end
    return t
end

function tabSave()
     local str=json.encode(objects,{indent=true})
    saveProjectTab("Test", "--["..str.."]--")
end


--# MyClass
MyClass = class()

function MyClass:init(x,y)
    self.Class = "MyClass"
    self.foo = "bar"
    self.pos=vec2(50,50) 
    self.mis =vec3(1,2,3) 
    self.body = physics.body(CIRCLE,10)
    self.body.x = x
    self.body.y = y
end

function MyClass:__tostring()
    return json.encode(self,{indent=true}) 
end

function MyClass:draw()
    ellipse(self.body.x,self.body.y,self.body.radius)
end



--# Test
--[[{
    "body":{"y":500.0,"x":100.0,"Class":"Body"},
    "Class":"MyClass",
    "pos":{"y":50.0,"x":50.0,"Class":"vec2"},
    "mis":{"z":3.0,"y":2.0,"x":1.0,"Class":"vec3"},
    "foo":"bar"
  },{
    "body":{"y":500.0,"x":200.0,"Class":"Body"},
    "Class":"MyClass",
    "pos":{"y":50.0,"x":50.0,"Class":"vec2"},
    "mis":{"z":3.0,"y":2.0,"x":1.0,"Class":"vec3"},
    "foo":"bar"
  },{
    "body":{"y":500.0,"x":400.0,"Class":"Body"},
    "Class":"MyClass",
    "pos":{"y":50.0,"x":50.0,"Class":"vec2"},
    "mis":{"z":3.0,"y":2.0,"x":1.0,"Class":"vec3"},
    "foo":"bar"
  }]]--

dave1707

@juce That's because JSON only allows string keys. JSON works with numeric keys, see the example below. If I add a string key to tab1, it doesn't. So it looks like you can have numeric keys or string keys, but not both in the same table. I haven't needed to use json for anything, but just wanted to see how it worked.

function setup()
    tab1={1,2,{7,8,9},3,4}
    z=json.encode(tab1)
    print(z)

    tab2=json.decode(z)
    z=json.encode(tab2)
    print(z)

    print("\ntab1 --------")
    for a,b in pairs(tab1) do
        print(a,b)
    end

    print("\ntab2 --------")
    for a,b in pairs(tab2) do
        print(a,b)
    end

    print("\ntab1 --------")
    print(tab1[1])
    print(tab1[3][1],tab1[3][2],tab1[3][3])

    print("\ntab2 --------")
    print(tab2[1])
    print(tab2[3][1],tab2[3][2],tab2[3][3])    
end

yojimbo2000

@Jmv38 this is my version of json encoding of a physics scene. You can save a snapshot of the scene, and then load it in again. Could be useful for a level editor in an Angry Birds style physics game:

https://gist.github.com/Utsira/da4fda9b95e9112d705a

Here's the class with the various meta-methods:

--meta-methods
function AddToJson()
    local meta = getmetatable(vec2())
    meta.__tojson = function(t)
        return json.encode({__vec2={x=t.x,y=t.y}})
    end
    meta = getmetatable(vec3())
    meta.__tojson = function(t)
        return json.encode({__vec3={x=t.x,y=t.y,z=t.z}})
    end
    meta = getmetatable(vec4())
    meta.__tojson = function(t)
        return json.encode({__vec4={x=t.x,y=t.y,z=t.z,w=t.w}})
    end
    meta = getmetatable(matrix())
    meta.__tojson = function(t)
        return json.encode({__mat4={t[1],t[2],t[3],t[4], t[5],t[6],t[7],t[8], t[9],t[10],t[11],t[12], t[13],t[14],t[15],t[16]}}) --a matrix.unpack function would've been handy here!
    end
    meta = getmetatable(color())
    meta.__tojson = function(t)
        return json.encode({__color={r=t.r, g=t.g, b=t.b, a=t.a}})
    end
    meta = getmetatable(mesh())
    meta.__tojson = function(t)
        return json.encode({nil}) --({__mesh=t.id})
    end

    local dummy = physics.body(CIRCLE, 10)
    meta = getmetatable(dummy)
    dummy:destroy()

    meta.__tojson = function(t)
    return json.encode({
    __rigidbody={shapeType=t.shapeType, radius=t.radius, points=t.points},
    __rigidbodyProperties={type=t.type, position=t.position, angle=t.angle, linearVelocity=t.linearVelocity, angularVelocity=t.angularVelocity, restitution=t.restitution, friction=t.friction, linearDamping=t.linearDamping, angularDamping=t.angularDamping
    }})
       end 
    --add extra categories from below according to needs
    --gravityScale=t.gravityScale, info=t.info, density=t.density, sensor=t.sensor, categories=t.categories, mask=t.mask, fixedRotation=t.fixedRotation, active=t.active, awake=t.awake, sleepingAllowed=t.sleepingAllowed, interpolate=t.interpolate, joints=t.joints, bullet=t.bullet, 

end

yojimbo2000

@Jmv38 thanks for showing us how to set the toJson metamethod (for physics bodies too!)

juce

@dave1707, you are right: i should've been more specific - json only allows string keys in dictionaries. The edge cases ultimately come from data structure differences: Lua's table can have keys of arbitrary types and also has the notion of having an array part and a hash (dictionary) part, while json has two different container types: dictionary and list.

I think what happens is that json.encode looks for special case when encoding tables: if a table only has integer keys, with the smallest one being 1, then it converts such table to a list, putting nulls into missing slots. Otherwise, it converts the table into a dictionary, collating numeric keys into string keys.

--# Main

function setup()
    t = {[1]="foo",[10]="bar"}
    print(json.encode(t))

    t = {[1]="foo",[20]="bar"}
    print(json.encode(t))

    t = {[-1]="foo",[0]="bar", [1]="baz"}
    print(json.encode(t))

    -- print array part of the table
    for i,v in ipairs(t) do
        print(i,v)
    end
end

But even that description is not entirely accurate: notice how second table has "too many" nil slots, compared to first one - and gets converted into a dictionary, not a list.

yojimbo2000

@juce that's interesting how it decides whether to go with dictionary or list. I avoid mixed dictionary-lists in Lua, it just seems like too much potential for unexpected behaviour

dave1707

@juce Interesting. I guess you really need to know how the decode is going to handle a table that's encoded. You could write a program that uses a table without any problems only to have it crash when you try to recreate a new table with decode.

juce

@yojimbo2000, i agree that mixing numeric keys with keys of other types is dangerous, and best avoided if possible. However, even if we use only numeric keys, this table-to-list conversion algorithm can play unexpected tricks: say you have a sparse table with integer keys - it can probably change back and forth between list and dictionary, as you add more values to it, depending on the exact distribution of keys.

juce

Like that:

--# Main

function setup()
    t = {[1]="a", [4]="b", [9]="c"}
    t[10]="d"
    t[5]="e"
    print(json.encode(t))
    t[13]="f"
    print(json.encode(t))
    t[11]="g"
    t[12]="h"
    print(json.encode(t))
end

On my iPad 2 it goes from list to dictionary and back to list:

["a",null,null,"b","e",null,null,null,"c","d"]
{"1":"a","9":"c","10":"d","4":"b","5":"e","13":"f"}
["a",null,null,"b","e",null,null,null,"c","d","g","h","f"]

yojimbo2000

I also try to avoid sparse lists, by using table.remove to make sure there's no holes. @dave1707 try my code at the gist. I'll put my neck on the block and say it's reasonably stable at recreating a physics scene. :-SS

https://gist.github.com/Utsira/da4fda9b95e9112d705a

juce

@dave1707, i think decode is ok, because it is fully deterministic. It's the encoding step that can have unexpected results. But yes, the end result of encode + decode will not necessarily recreate the table as it was before.

dave1707

@yojimbo2000 I'm not saying it won't work and I'm sure yours does, but for someone who doesn't understand how encode/decode works could be in for a surprise if they use it. Just like me, I thought encode/decode would take a table, convert it to a string that could be saved, then recreate the table exactly like the original. But it turns out there's limits, even with simple tables. I don't have any code that uses encode/decode and probably never will. It's just that when it was brought up in this discussion, I thought I'd try it to see how it works and ran into some surprises that I pointed out above. If someone uses this in an App Store program without understanding everything about it, their program could crash.

yojimbo2000

I guess I tend to think of dictionaries as being somewhat unpredictable anyway - they have no order as such, in that you can't predict what order pairs will return the items in.

AxiomCrux

@yojimbo2000 @dave1707 I am perpetually baffled by the fact that pairs() seems to be so inconsistent in its traversal of a table.

@Simeon @John I am curious how it is possible for pairs() based for loops to occur in different order each run?

I was using them to parse and layout GUI objects, and it was practically random order each run. This goes against everything I have ever seen in programming, usually the same scenario produces the exact same result..

I alwasy was assuming that pairs() would work alphabetically, or in the order that the values were created.. Is this related to available memory locations / garbage collection / frame rate? It may be worth adding a note in the built in documentation, I was going bannanas trying to figure out why things were always showing up in a different order and ended up having to switch to ipairs to keep from losing my mind :P`

Super useful thread btw, some of this seems worthy of inclusion into a built in demo / example of some sort.

dave1707

@AxiomCrux I think one of the reasons pairs works the way it does is because pairs will iterate thru a table even if it contains empty entries. For example if tab[5]=100 and tab[1000]=345, pairs will show both entries, but ipairs wont show either because it's not consecutive starting at 1. You're probably correct when you say that the tables are probably created based on available memory locations. I did a google search for pairs and found this:

Description

Returns an iterator function(next) for a for loop that will return the values of the specified table in an arbitrary order.

AxiomCrux

heheh yeah! @dave1707 "arbitrary order" thats the thing I can't seem to fully wrap my head around..

I've been using computers since I could form memories, and I have recently come to understand cryptography and how 'random' number generation works (which opened my mind to profound philisophical insight into chaos theory and the nature of the universe).

"arbitrary order" and programming are opposing concepts IMHO. The notion that a key component of this awesome Lua programming language functions that way.. . scrambles my brain into "arbitrary order" a bit.

dave1707

@AxiomCrux Maybe you'll find this interesting. It contains something about tables but I don't know if it really answers anything about arbitrary order.

https://www.lua.org/gems/sample.pdf

yojimbo2000

Don't think of it as "arbitrary order", just unordered. JSON dictionaries are unordered as well. If order is important, you need to put the elements into an array.

dave1707

@yojimbo2000 @AxiomCrux The comment I mentioned in the above post and what I show below is wrong for pairs. It won't return the values in arbitrary order, but will create the table in arbitrary order. If you run the example code below and keep tapping the screen to read the table, it will show the same results in some order each time. If you restart the program, it will create the table in a different order.

Description

Returns an iterator function(next) for a for loop that will return the values of the specified table in an arbitrary order.

function setup()
    tab={a=1,b=2,c=3,d=4,e=5}
    for a,b in pairs(tab) do
        print(a,b)
    end
end

function touched(t)
    if t.state==BEGAN then
        output.clear()
        for a,b in pairs(tab) do
            print(a,b)
        end
    end
end