Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

http request XML

edited March 2014 in Questions Posts: 22

Hey guys

i have this piece of code, where i try to load an eve-central API key (which is a server located XML file) and have it save it locally for further use. (i want to read from that XML, parse it / use values)

I have it now save the entire part of the XML system they use, based on the URL i give it, but then i have trouble loading the XML's parameters (headers). Now the code i am building on seems a bit overdone, but it's mainly for testing purposes, to see what my options are and what to built on. Question is: what are my options? End goal: download a LARGE amount of data, formatted in an XML order (eve central.com) and use parts of it for further calculations. (so for instance, i now ask for ID 34, stats from region 10000002. it gives me buy, sell and all volumes, min, max, median, etc and writes it to an xml file for later use)

let me note i am an amateur coder with limited C and JAVA knowledge and basically only know a bit about the structure of LUA and some basic coding. Almost totally new to codea and trying to learn as much as i can the next few weeks

code i use now:

-----------------------------------------------

local VALUE = 1
function setup()
    print("-- run check: loading eve central")
    market = nil
    http.request("http://api.eve-central.com/api/marketstat?&typeid=34&regionlimit=10000002", loadxml, failxml)
end
function loadxml(inp)
   market = inp
local file = os.getenv("HOME").."/Documents/market.xml"
local xml = inp
print("-- created file "..file)
writefiles(file,xml)
readFiles(file)
   end
function writefiles(file,xml)
    print("-- start write -- ")
    print("write in "..file)
    io.output(io.open(file,"w"))
    io.write(xml)
    io.close()
    print("-- write done --")
    end
function readFiles(file)
    print("--start read --")
    print("read from "..file)
    io.input(io.open(file,"r"))
    local li
    for i in io.lines() do print(i) end
    print("-- read done --")
    end
    function failxml()
        print("xml download failed")
        end
function draw()
    background(40, 40, 50)
    strokeWidth(5)
    if market ~= nil then
        text(market, WIDTH / 2, HEIGHT/2)
        fill(255)
        textWrapWidth(WIDTH)
        else
        text("loading", WIDTH / 2, HEIGHT/2)
end
end

---------------------------------------------
Tagged:

Comments

  • IgnatzIgnatz Mod
    Posts: 5,396

    @Numenni - to make your code format nicely, put three ~ on a line before and after it. Also, use ' for apostrophes if you don't want weird text formatting.

    Can you clarify whether your problem is getting the XML headers downloaded at all, or reading them using Codea?

  • edited March 2014 Posts: 1,976

    Not any experience with XML, but I think I know what you need. Function loadFiles can accept actually three parameters. First is data received from the website, second is status (200, 404, etc.), and third is headers. I believe the third is what you want. loadFiles should look like this, I think:

    function loadxml(inp, status, headers)
        if inp ~= nil and status == 200 then
            market = inp
            local file = os.getenv("HOME") .. "/Documents/market.xml"
            local xml = inp
            print("-- headers:")
            for k, v in pairs(headers) do -- Loop through the headers and print all of them
                print("    " .. k .. ": " .. v)
            end
            print("-- created file " .. file)
            writefiles(file, xml)
            readFiles(file)
        else -- The page returned nothing, or gave an error.
            print("-- error getting xml, status: " .. status) -- Note that this kind of error is different from the failed function. The failed function is for something like a network error, like no WiFi. This kind of error is like 404 Not Found, where data is received, but the data tells you the page couldn't be found.
        end
    end
    

    Tested.

  • edited March 2014 Posts: 22

    Reading them using Codea after i created the xml file, but i rather have the direct approach opted by SkyTheCoder here. will test this afternoon and let you guys know!

    it does not print anything between Headers: and Created file, but also gives back no error. does that have to do with the setup of the source file?

    i love the additions though, status check is great for error handling, didn`t think of that one. but somehow the script now does not recognise the headers (i believe).

    PS. you can CLICK the link in my original post to view what it reads.

    Eventually i want to be able to call ID > BUY > VOLUME for instance or ID > SELL > MAX.

    having trouble loading this in lua. in .NET it`s much easier, better XML integration and can't get the XML parsers to work for codea, so figured to try and hardcode it myself, but i have to be honest and say it's much harder than it looked at first.

    @ignatz thank u, will do next time!

    oh and can i delete a comment?

    EDIT:
    reading my code carefully, i think we are now reading from the created XML file. I think thats where i go wrong somehow, i might not be creating the xml properly, hence it does not find the headers? And would it then not be easier reading directly form the URL provided file?

  • Posts: 22

    it does not print anything between Headers: and Created file, but also gives back no error. does that have to do with the setup of the source file?

    i love the additions though, status check is great for error handling, didn`t think of that one. but somehow the script now does not recognise the headers (i believe).

    PS. you can CLICK the link in my original post to view what it reads.

  • edited March 2014 Posts: 22

    Eventually i want to be able to call ID > BUY > VOLUME for instance or ID > SELL > MAX.

    having trouble loading this in lua. in .NET it`s much easier, better XML integration and can't get the XML parsers to work for codea, so figured to try and hardcode it myself, but i have to be honest and say it's much harder than it looked at first.

  • Posts: 22

    @ignatz thank u, will do next time!

  • edited March 2014 Posts: 1,976

    Woops! My bad. Should've used pairs, not ipairs. Edited the above code.

    You can either update with the edited code, or remove the i in ipairs, so it's just pairs.

    Edit: Tested it, it works now.

  • Posts: 22

    Great. Now i need to find a way to have it read and display the subsections correctly.

  • Posts: 22

    Is there a way to use luaxml?

  • edited March 2014 Posts: 22

    Or otherwise integrate string.match(i, "< min>(d-)< /min>") into io.lines?
    Like
    ~ for i in io.lines do local z = string.match(i, '< min>(d-)< /min>') end ~

    I would then want to draw the d- number on screen, or for now at leaSt have it printed

    Ps formatting fails for me

  • BriarfoxBriarfox Mod
    edited March 2014 Posts: 1,542

    @Numennal I didn't read the whole discussion but here is an xml lib that I've used created by @hyrovitalyprotago

    XML = class()
    --[[
    -- @Author : Hyro Vitaly Protago
    -- @Version : 1.1
    
    -- This class is very simple to use :
    
    --     - If you have an object, and you want to export as xml :
    --         tostring(XML(obj, nameOfObj))
    
    --     - If you have an xml, and you want to remake table :
    --         XML(obj).racine
    
    --     This doesn't support empty element like "<br />".
    --     This doesn't support namespace.
    --     This doesn't support attribute. (because is totally useless in this case)
    
    --     Errors message are not really precise, but sometimes, it can help
    --]]
    
    XML.templates = {}
    XML.templates.header = "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\" ?>"
    XML.patterns = {}
    XML.patterns.header = "<\?xml[^\?]*\?>"
    
    function XML:init(obj, name)
        if obj ~= nil and type(obj) == "table" then -- mode to xml
            self.xml = self:toXml(obj, 0, name)
        elseif obj ~= nil and type(obj) == "string" then -- mode to tab
            self:tableConstruct(obj)
        else
            error("IllegalArgumentException: @Param (1) obj must be a table or a string")
        end
    end
    
    function XML:toXml(tab, level, name)
        level = level or 0
        local str = ""
        if name then
            str = "<" .. name .. ">\n"
            level = level + 1
        end
        for k,v in pairs(tab) do
            for i = 1, level do
                str = str .. "\t"
            end
            if type(v) == "table" then
                str = str .. "<" .. k .. ">\n"
                str = str .. self:toXml(v, level + 1)
                for i = 1, level do
                    str = str .. "\t"
                end
                str = str .. "</" .. k .. ">\n"
            else
                str = str .. "<" .. k .. ">" .. tostring(v) .. "</" .. k .. ">\n"
            end
        end
        if name then
            str = str .. "</" .. name .. ">\n"
        end
        return str
    end
    
    function XML:tableConstruct(xml)
        local _i, _j = string.find(xml, XML.patterns.header)
        if _i and _j then
            -- header manipulations (namespace)
            xml = string.sub(xml, _j+1)
            xml = xml:gsub(">[\s\b\t\r\n]*<", "><")
            xml = xml:gsub("^%s*(.-)%s*$", "%1")
        end
    
        self.racine = {}
        self.stack = {{name="racine", table=self.racine}}
    
        while string.len(xml) > 0 do
            local i, j = string.find(xml, "^<[^/<>]*>[^<>]*</[^<>]*>")
            if i ~= nil and i == 1 then
                xml = self:simpleNode(xml)
            else
                i, j = string.find(xml, "^</[^<>]*>")
                if i ~= nil and i == 1 then
                    xml = self:endOfComplexNode(xml)
                else
                    xml = self:complexNode(xml)
                end
            end
        end
    
        assert(#self.stack == 1, "Xml malformed !")
    
    end
    
    function XML:simpleNode(xml)
        local b,e,b2,e2,name,name2,value
        b, e = string.find(xml, "<[^<>]*>")
        name = string.sub(xml, b + 1, e - 1)
        assert(string.len(name) > 0, "Xml malformed ! you cannot use tag with no name...")
        b2, e2 = string.find(xml, "</[^<>]*>")
        name2 = string.sub(xml, b2 + 2, e2 - 1)
        assert(name == name2, "Xml malformed ! Opening and Closing Tag incorrect: ("..name..") ~= ("..name2..")...")
        value = string.sub(xml, e + 1, b2 - 1)
        self.stack[#self.stack].table[name] = value
        return string.sub(xml, e2 + 1)
    end
    
    function XML:complexNode(xml)
        local b,e,name
        b, e = string.find(xml, "<[^<>]*>")
        name = string.sub(xml, b + 1, e - 1)
        self.stack[#self.stack].table[name] = {}
        table.insert(self.stack, {name=name, table=self.stack[#self.stack].table[name]})
        return string.sub(xml, e + 1)
    end
    
    function XML:endOfComplexNode(xml)
        local b,e,name,name2
        b, e = string.find(xml, "</[^<>]*>")
        name = self.stack[#self.stack].name
        name2 = string.sub(xml, b + 2, e - 1)
        assert(name == name2, "Xml malformed ! Opening and Closing Tag incorrect: ("..name..") ~= ("..name2..")...")
        table.remove(self.stack, #self.stack)
        return string.sub(xml, e + 1)
    end
    
    function XML:__tostring()
        local toXmlString
        toXmlString = function(tab, level)
            level = level or 0
            local str = ""
            for k,v in pairs(tab) do
                for i = 1, level do
                    str = str .. "\t"
                end
                if type(v) == "table" then
                    str = str .. k .. ":\n" .. toXmlString(v, level + 1)
                else
                    str = str .. k .. ": " .. tostring(v) .. "\n"
                end
            end
            return str
        end
        if self.racine then
            return toXmlString(self.racine)
        elseif self.xml then
            return XML.templates.header .. "\n" .. self.xml
        end
    end
    
    
  • Posts: 22

    Your class seems to handle, sort of, what i want. But i got it so that it works, but returns xml malformed error, line 121 in e xml class

  • Posts: 22

    There seems to be a rogue closer downloaded, /type. Have to remove that from the string first seems to fix the problem. Will have to try for different entries though

  • Mmmmm, I have write that code a "long" time ago. I'll try to make something better in the near future (maybe tomorrow or this weekend).

  • edited March 2014 Posts: 22

    Its workng my friend, on the database end it does < type id34> and closes with < type> which i think confuses the code. It wants < /marketstat> in that spot ( run my code or follow the url to see what i mean. Substracting the type level solved the problem, but its a bit messy that way.

  • Yeah... too messy for my taste ;) I've a lot of ideas for support attributes, empty elements and others things, so, when i'll found time, i upgrade it.

  • edited March 2014 Posts: 304

    [Duplicate post]

  • BriarfoxBriarfox Mod
    Posts: 1,542

    @HyroVitalyProtago I had started re working your old code. I'm looking forward to seeing what you come up with.

  • Thanks @Briarfox, this is my first work on it (just for read) :

    XML2 = {}
    
    -- @Author : Hyro Vitaly Protago
    -- @Version : 2.0
    
    -- Warnings:
    --  - this utilities have not been made for check if a xml is malformed or not, so, check before if it's possible.
    
    -- Support: attributes
    
    -- This doesn't support: namespaces, empty elements (for the moment), mixed elements
    
    -- How use it ?
    --  - XML2.toTableFromString(str) : return a table who contains the elements of the xml text (str)
    
    -- Examples:
    -- XML2.toTableFromString(...).marketstat.type.buy.volume
    -- XML2.toTableFromString(...).__attribute.version
    -- XML2.toTableFromString(...).__type == "simple" || "complex"
    
    -- ToDo:
    -- - empty (and mixed) elements
    -- - toStringFromTable(xml)
    
    local function trim(s)
      return (s:gsub("^%s*(.-)%s*$", "%1"))
    end
    
    local function startsWith(s, pattern)
        return not (not s:find("^"..pattern))
    end
    
    local mt
    mt = {
        __index = function (tbl, key)
            if (startsWith(key, "__")) then
                tbl = rawget(tbl,key)
                return tbl[key]
            end
            assert(rawget(tbl,"__type") ~= "empty", "Empty elements have no value.")
            tbl = rawget(tbl,"__values")
            if rawget(tbl[key],"__type") == "simple" then
                return rawget(tbl[key],"__value")
            end
            return tbl[key]
        end,
        __newindex = function(table, key, value)
            error("Read-only table")
        end
    }
    
    local function removeXmlDefinition(xml)
        local _i, _j = string.find(xml, "<\?xml.-\?>")
        if _i and _j then
            -- header manipulations (namespace)
            xml = string.sub(xml, _j+1)
            xml = xml:gsub(">[\s\b\t\r\n]*<", "><")
            xml = xml:gsub("^%s*(.-)%s*$", "%1")
        end
        return xml
    end
    
    local function parse(xml, acc)
        xml = trim(xml)
    
        local i, j, element, elementName
        local space, endOfElement
        local attributes = nil
    
        i, j = string.find(xml, ".->")
        element = string.sub(xml, i, j)
    
        space = string.find(element, "%s") or j
        endOfElementName = math.min(space, j)
    
        elementName = string.sub(element, 2, endOfElementName-1)
    
        acc[elementName] = {}
    
        if space ~= j and space < j then -- attributes
            attributes = {}
            local attributesString = trim(string.sub(element, endOfElementName, j-1))
            local ii, jj = string.find(attributesString, '.-="')
            while (ii) do
                local attrName = string.sub(attributesString, ii, jj-2)
                local attributesString2 = string.sub(attributesString, jj+1)
                local iii,jjj = string.find(attributesString2, '"')
                local attrValue = string.sub(attributesString2, 1, jjj-1)
                attributes[attrName] = attrValue
                attributesString = trim(string.sub(attributesString, jj+jjj+1))
                ii, jj = string.find(attributesString, '.-="')
            end
            acc[elementName].__attributes = attributes
        end
    
        -- ToDo
        if string.sub(xml, j-1, j-1) ==  "/" then -- empty element
            -- acc[elementName].__type = "empty"
        end
    
        xml = string.sub(xml, j+1)
    
        i2, j2 = string.find(xml, "</.->")
        local nextEndBalise = string.sub(xml, i2, j2)
        local nextEndBaliseName = string.sub(nextEndBalise, 3, string.len(nextEndBalise)-1)
    
        if elementName == nextEndBaliseName then -- simple
            acc[elementName].__type = "simple"
            acc[elementName].__value = string.sub(xml, i, i2-1)
            setmetatable(acc[elementName],mt)
            return string.sub(xml, j2+1)
        else -- complex
            acc[elementName].__type = "complex"
            acc[elementName].__values = {}
            xml = parse(xml, acc[elementName].__values)
            while (xml and string.len(xml) > 0) do
                i2, j2 = string.find(xml, "</.->")
                local nextEndBalise = string.sub(xml, i2, j2)
                local nextEndBaliseName = string.sub(nextEndBalise, 3, string.len(nextEndBalise)-1)
                if nextEndBaliseName == elementName then
                    setmetatable(acc[elementName],mt)
                    return string.sub(xml, j2+1)
                else
                    xml = parse(xml, acc[elementName].__values)
                end
            end
        end
    end
    
    function XML2.toTableFromString(xml)
        xml = removeXmlDefinition(xml)
        local racine = {}
        parse(xml, racine)
        setmetatable(racine,mt)
        return racine
    end
    
    
  • Posts: 22

    Hey there. Have to say, im impressed. Can i use this customized parser in my app? Will be non commercial, but will still ask. Its perfect for what i need.

  • Use it as you wish for all what you want ;)

  • Posts: 22

    Just a silly question, why are we using this as namespace instead of just as a new class? Any particular reason?

  • In fact, the real question behind that is: why we use classes ?

    In first, classes in Lua is just a table with a metatable (you can watch the code of "class.lua" in Codea.app on your iPad if you want).

    Also, we use classes when we want stock a state of an object (for example, I have 3 square, with attributes {x,y,width,height}, same pattern, values differents, so, the class is used for create object of the same type with different states), and when we want inheritance.
    In this case, no inheritance, no states, just a "static" function.

    My previous implementation use a class, but in real, I didn't need it.

  • Posts: 22

    Ah roger roger. I thought namespaces were just used to store values and variables, don't know where i got that idea

  • Posts: 22

    Btw u were still working on the code (besides the missing tostringfromtable)? since it doesnt work (yet for me). tried to find the holes in it but cannot seem to fix them. It either does not identify XML2 properly when i try to use it on my xml (that is if i use for instance XML2.toTableFromString(file).marketstat.type.buy.volume as you mention), or it (obviously) asks for a number when you leave it blanc (just use XML2.toTableFromString(file). I decided to work on the rest of the app that (thank god) goes much better. though that is just some simple math on right now made up preset static variables (that hopefully gets worked out in the end with this xml parser). awsome job so far anyway. it is more then appreciated and if you get this to work i`ll definitly throw you down in the credits page on top.

  • edited March 2014 Posts: 22

    PS check http://www.evedustry.com/download.html for the windows (.net) version of the app that im redoing in lua for ipad.

  • lua function "tonumber(string)" is probably what you want (if I have understand)

  • Posts: 22

    Uh, no sorry, i was just confirming that it is still a work in progress and that my attempts to help on it were futile, void, failed and i stated u need to be credited for the good work

  • Posts: 22

    i am really looking forward to the Tostringfromtable half of the code :)

  • Posts: 22

    Hows it coming around? What i tried myself is write it so that it reads i.lines and stores the values in a table, which worked great BUT since it reads per line, it loses the context of th ID before it, so it can insert everything between buy and volume per id in the right table, but unless i have it call id 1-gazillion and in that order, it is near impossible to store it in the right ID. In the table. Or yet at least above my ability so far.

  • Posts: 22

    Unless i can have it look back at a line, it cannot be done ....

  • I try to make it soon, but I have a lot of work currently x)

  • Posts: 1,255

    Not to revive a long dead thread...

    But did anyone ever come across a nice XML parser that works neatly in Codea? The last example in this thread looks like the right approach, but fails frequently when I try to parse data coming from SOAP services that seem well formed. Thanks.

  • @Mark Always a lot of work... but I'll try to do something when I can

Sign In or Register to comment.