Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Implement blocking http.request() ?

in General Posts: 81

Does anyone have a solution to make a call to http.request a blocking/synchronous call? I’d like to wait for the success function to complete before proceeding with code after the http request. If not, has anyone set up a notification message from success, or a way to poll of the request data has been received?

Thanks!

Tagged:

Comments

  • dave1707dave1707 Mod
    Posts: 7,810

    @iam3o5am Just set a variable when you do the request and don’t reset that variable until you receive the success. As long as that variable is set, you don’t do any processing of the data. Do a forum search for http request and see what you find. There’s a lot of info for it.

  • dave1707dave1707 Mod
    Posts: 7,810

    @iam3o5am Here’s some code that downloads 5 mp3 files. I have the function that saves the files commented out since I don’t know if you want that or not or you just want to see how to do a request and wait for a completion before starting the next download. Theses are large files, so it takes a few seconds to download them. Just tap the screen to start the download.

    displayMode(FULLSCREEN)
    
    function setup()
        rectMode(CENTER)  
        downloadedTable={}
        sizeTable={}
        queTable={
            {"http://www.amigaremix.com/files/2894/laki_-_Shadow_of_the_Beast_-_Death_Scene.mp3","Death_Scene"},
            {"http://www.amigaremix.com/files/2910/EXON_-_Lotus_III_Theme_.mp3","Lotus_III_Theme"},
            {"http://www.amigaremix.com/files/2869/Darkman007_-_Dazzler_Cover.mp3","Dazzler_Cover"},
            {"http://www.amigaremix.com/files/2212/Cisskey_-_Turrican_2_-_In_the_water.mp3","In_the_water"},
            {"http://www.amigaremix.com/files/2010/Sven_Storm_-_Firehawk_Loader.mp3","Firehawk_Loader"}, }
    end
    
    function draw()
        background(0)
        fill(255,0,0)
        text("To download",WIDTH/2,HEIGHT-40)
        text("Downloaded",WIDTH/2,HEIGHT-300)
        text("Tap screen to start download",WIDTH/2,200)
        fill(255)
        for z=1,#queTable do
            text(queTable[z][2],WIDTH/2,HEIGHT-30*z-60)
        end 
        if #queTable>0 and start then
            if not downloading then            
                downloading=true
                http.request(queTable[1][1],didLoadMusic,notFound)
            end
            text("Downloading --  "..queTable[1][2],WIDTH/2,100)
        end
    
        for a,b in pairs(downloadedTable) do
            text(string.format("%s  %d bytes",b,sizeTable[a]),WIDTH/2,HEIGHT-320-a*30)
        end
    end
    
    function touched(t)
        if t.state==BEGAN then
            start=true
        end
    end
    
    function notFound()
        print("error with file..."..queTable[pos][1])
        downloading=false
        table.remove(queTable,1)
    end
    
    function didLoadMusic(data,status,headers)
        writeFile(queTable[1][2]..".mp3",data)
        downloading=false
        table.insert(downloadedTable,queTable[1][2])
        table.insert(sizeTable,#data)
        table.remove(queTable,1)
    end
    
    function writeFile(fileName, data)
        --[[
        local file = os.getenv("HOME").."/Documents/Dropbox.assets/"..fileName
        wFd, err = io.open(file, "w")
        if wFd == nil then
            print("Error creating file "..fileName.."\n\nError: "..err)
        else
            wFd:write(data)
            wFd:close()
        end
        --]]
    end
    
  • Posts: 81

    Thanks @dave1707 for the sample code. I suppose I’ve been simplistic in my thinking. I was hoping there was a way to just prevent execution from continuing to the line after the http.request() call, and instead rely on continuing execution from the http.request success function. Instead it looks like I just have to use some logic to loop around and control were execution happens until I get my requested data.

  • dave1707dave1707 Mod
    Posts: 7,810

    @iam3o5am The way Codea works is it keeps on executing. If you don’t want something to run, you skip around it. Once you do your request, you don’t execute the request again until you received what you requested or after a certain time limit you set. Once you get used to how Codea runs, it is easy to code things like that. I remember when I coded in basic, there were pause or delay commands that stopped the execution at that point until you wanted to continue. Commands like that would never work with Codea because it needs to constantly run.

  • Posts: 502

    You can make a blocking api yourself (or a "pool" like you mentioned). The way you do this, is you queue your function calls as coroutines and then let the coroutines work one after another by checking whether the current routine done its job or not and the continue with the next.

    I posted an implementation some time ago. Please see here: https://codea.io/talk/discussion/8215/cutscenes-with-coroutines

  • Posts: 81

    Along the same vein, here’s some code:

    function setup()
        print(coroutine.running())
    
        function callback(data)
            print(coroutine.running())
        end
    
        co = coroutine.create(
            function()
                print(coroutine.running())
                http.request("http://www.google.com/home", callback, callback)
            end
        )
    
        coroutine.resume(co)
    end
    

    The output is:

    thread: 0x1281adc08 true
    thread: 0x1c41bacc8 false
    

    I put callback() as both the success and fail arguments, to be sure I catch a call from either. Problem is, nothing prints from that function. The main thread resumes after ouputting what you see above.

    I suppose the second return value (boolean) is whether the thread is running, though PIL does not mention it. Yet, if I add more code to the coroutine function (say, before the http.request call) then that code executes.

    So my questions are:
    - Why isn’t callback() called?
    - What does the true/false return value mean?
    - Under the Codea hood what is happening when http.request is called?

    Thanks.

  • edited July 2018 Posts: 81

    @se24vad Thank you for the link. Between your code and that of @dave1707, I am getting a good picture of this approach. I have a question about your thread queue in particular:

    Could you show how you would write a function, called by exec(), that itself would call Codea’s http.request() function? In other words, if I have this function:

    function request()
        local function success(data)
            print(data)
        end
    
        http.request("http://www.google.com/home", success)
    end
    

    And then add it to your thread queue:

    exec(request)
    

    Then the request function finishes not after success() is called, but as soon as the http.request() line is complete. The coroutnie yields and I never get the data printed in the success() function.

    I would love to have Codea’s http.request() execute in another coroutine and not yield until the success function is called and complete.

    On that note, does anyone know how the http.request function works under the hood?

  • dave1707dave1707 Mod
    edited July 2018 Posts: 7,810

    @iam3o5am I don’t know about under the hood, but once you call the request function, it does it’s own thing until it’s done. When it’s done, it calls the success function you set. So once your request starts running, Codea continues to run your regular code until the success function is called. It then runs the success code until it’s complete. Then Codea continues with your regular code.

    Edit: I don’t think the request function will work if the coroutine it’s in doesn’t yield. I might have an example of a coroutine that will start a request. When the coroutine ends, the request function executes. Once the request is successful, it starts another coroutine that runs some code, yields to Codea, Codea yields to the coroutine, etc. until that coroutine exits. Then Codea runs without yielding. I don’t know if your after something like that.

  • Posts: 81

    I ask about under the hood, because as my example above shows (two comments up, where I use the coroutine.running() call) there are cases when the success callback function doesnt get called. It either happens when I use http.request in a separate coroutine, or if I use it in the main thread but i enter a loop and it appears that the success function is never given a chance to run. I wanted to loop and just check if I got my data yet, but the while loop never allowed success() to execute.

    My guess is that @Simeon and team are using some sort of multithreaded C under the hood to handle the actual http request? I just don’t understand how that plays into the execution path that the Lua interpreter is going through....

  • edited July 2018 Posts: 81

    @dave1707 In your music download code above, what if you had follow-up code to execute - where would you put that? In other words:

    1) Do something
    2) Break, to download music (what your code does now)
    3) Continue on doing something else...

    Thanks for being patient with me...not sure why I’m having such a mental block with this one. My ultimate goal is a general solution for blocking http.requests. In the middle of other code I may want to make some request and I want to hold everything else until I get what I want and then move on.

  • dave1707dave1707 Mod
    Posts: 7,810

    @iam3o5am Without knowing exactly what you’re doing, here’s something simple. The first sprite tree runs all the time. The second sprite Observatory runs only during a download. The third sprite Explosion runs after the download completed. Tap the screen to start a download. Not sure if this is what you’re after.

    function setup()
        x1,y1=WIDTH/2,HEIGHT/2
        xv1,yv1=2,3
        x2,y2=WIDTH/2,HEIGHT/2
        xv2,yv2=-3,2
        x3,y3=WIDTH/2,HEIGHT/2
        xv3,yv3=3,-3
        fill(255)
        str="tap screen to start download"
    end
    
    function draw()
        background(0)
        text(str,WIDTH/2,HEIGHT-100)
        firstSprite()
        if runSecond then
            secondSprite()
        end
        if runThird then
            thirdSprite()
        end
        sprite("Small World:Tree Round",x1,y1)
        sprite("Small World:Observatory",x2,y2)
        sprite("Small World:Explosion",x3,y3)
    end
    
    function firstSprite()
        x1=x1+xv1
        y1=y1+yv1
        if x1<0 or x1>WIDTH then
            xv1=-xv1
        end
        if y1<0 or y1>HEIGHT then
            yv1=-yv1
        end
    end
    
    function secondSprite()
        x2=x2+xv2
        y2=y2+yv2
        if x2<0 or x2>WIDTH then
            xv2=-xv2
        end
        if y2<0 or y2>HEIGHT then
            yv2=-yv2
        end
    end
    
    function thirdSprite()
        x3=x3+xv3
        y3=y3+yv3
        if x3<0 or x3>WIDTH then
            xv3=-xv3
        end
        if y3<0 or y3>HEIGHT then
            yv3=-yv3
        end
    end
    
    function touched(t)
        if t.state==BEGAN and not downloading then
            runSecond=true
            runThird=false
            print("http request started\nrun secondSprite")
            http.request("http://www.amigaremix.com/files/2894/laki_-_Shadow_of_the_Beast_-_Death_Scene.mp3",success,fail)
            downloading=true
        end
    end
    
    function success(data)
        print("http request ended\nrun thirdSprite")
        runSecond=false
        runThird=true
        downloading=false
    end
    
    function fail()
        print("http failed")
    end
    
  • dave1707dave1707 Mod
    Posts: 7,810

    @iam3o5am The http.request is probably a coroutine in itself because you can run a request without it stopping the normal Codea code. If you do a request that loads a lot of data that you need to process, then the processing code should be put in a coroutine. That way you can process the data without interrupting the normal Codea code and when you’re done with the process, then you can do whatever you want with the processed data.

  • dave1707dave1707 Mod
    edited July 2018 Posts: 7,810

    @iam3o5am Here’s another example. A sprite moves around the screen. When you tap the screen to start the http.request, the sprite continues to move. When the request is complete, a process routine starts. The non coroutine process stops the sprite because it’s taking all of the cpu process. When you do the request with the runCoroutine slider on, the coroutine process will run when the request is complete. The coroutine is set up to yield every 100000 count to allow the sprite to continue to move. The coroutine will take longer, but it won’t completely stop the sprite. Changing the 100000 count to a larger or smaller number will vary the time the process routine run, but it will also vary the movement of the sprite.

    function setup()
        parameter.boolean("runCoroutine",false)
        x1,y1=WIDTH/2,HEIGHT/2
        xv1,yv1=2,3
        fill(255)
        limit=100000000
        str="tap screen to start download"
    end
    
    function draw()
        background(0)
        text(str,WIDTH/2,HEIGHT-100)
        firstSprite()
        sprite("Small World:Tree Round",x1,y1)
        if done then
            done=false
            process()   -- run the non coroutine function
        end
        if yy then
            coroutine.resume(yy)    -- run the coroutine process
        end
    end
    
    function firstSprite()
        x1=x1+xv1
        y1=y1+yv1
        if x1<0 or x1>WIDTH then
            xv1=-xv1
        end
        if y1<0 or y1>HEIGHT then
            yv1=-yv1
        end
    end
    
    function process()
        print("loop process started")
        print("about 25 second delay")
        a=0
        for z=1,limit do
            a=a+math.sin(z)
        end
        print("loop process ended")
        print("total",a)
    end
    
    function processCo()
        print("coroutine loop process started")
        print("about 33 second delay")
        a=0
        for z=1,limit do
            a=a+math.sin(z)
            if z%100000==0 then
                coroutine.yield(yy)
            end
    
        end
        yy=nil
        print("coroutine loop process ended")
        print("total",a)
    end
    
    function touched(t)
        if t.state==BEGAN and not downloading then
            output.clear()
            print("http request started")
            http.request("http://www.amigaremix.com/files/2894/laki_-_Shadow_of_the_Beast_-_Death_Scene.mp3",success,fail)
            downloading=true
        end
    end
    
    function success(data)
        print("http request ended")
        print("data size ",#data)
        downloading=false
        if runCoroutine then
            print("coroutine process started")
            yy=coroutine.create(processCo)
            coroutine.resume(yy) 
        else
            print("non coroutine process started")
            done=true
        end   
    end
    
    function fail()
        print("http failed")
    end
    
  • edited July 2018 Posts: 502

    @iam3o5am you can't execute the success function BEFORE you sent a request. So if I understand you right, then what you want is to fire the success function right after the request received data. then later after success is ready, you want call some other routine. - if so then here's how I would do it.

    The key here, is that you call your functions through the exec() call, to queue them up and run after each other (in sequence). Because if you were to call it the Codea-way then success is like a coroutine and will fire only when it got data. By that time your other functions will be already called - thus will run BEFORE the success function, which is what you try to avoid as I understand.

    local function success(response)
        while not response do -- block until data from http request comes in
            coroutine.yield()
        end
        print(response) -- when data arrived, processing it
        print("response received. done.")
    end
    
    local function request()
        http.request("http://google.com/home", success)
        print("request sent. wait...")
    end
    
    local function continue_with_some_other_stuff()
        print("now that we completed the request and processed the response, it's time to continue with the rest of the queue")
    end
    
    
    function setup()
        request() -- start request
        exec(sucess) -- create a queue to work through
        exec(continue_with_some_other_stuff)
    end
    
    function draw()
        thread_update()
    end
    
  • Posts: 81

    @dave1707 Thanks for your recent post - it did get me over a few obstacles in my thinking - especially your done variable. @se24vad, also - thank you, though I still have some issues with the code you posted. If I flesh it out with an exec and thread_update functions (from your previous code, then the print order is still not what I expect/hope for. Here’s the full code:

    local thread_queue = {}
    
    local function exec(func, ...)
        local params = {...}
        local thread = function(self) func(self, unpack(params)) end
        table.insert(thread_queue, coroutine.create(thread))
    end
    
    local function thread_update()
        if #thread_queue > 0 then
            if coroutine.status(thread_queue[1]) == "dead" then table.remove(thread_queue, 1)
            else coroutine.resume(thread_queue[1], thread_queue[1]) end
        end
    end
    
    function draw()
        thread_update()
    end
    
    local function success(response)
        while not response do -- block until data from http request comes in
            coroutine.yield()
        end
        print(response) -- when data arrived, processing it
        print("response received. done.")
    end
    
    local function request()
        http.request("http://google.com/home", success)
        print("request sent. wait...")
    end
    
    local function continue_with_some_other_stuff()
        print("now that we completed the request and processed the response, it's time to continue with the rest of the queue")
    end
    
    function setup()
        request() -- start request
        exec(success) -- create a queue to work through
        exec(continue_with_some_other_stuff)
    end
    

    Here’s the response order I get:

    request sent. wait...
    
    thread: 0x1c01bb9e8
    
    response received. done.
    
    now that we completed the request and processed the response, it's time to continue with the rest of the queue
    
    <!DOCTYPE html>
    <title>Google</title>
    <script>
      var url = 'https://madeby.google.com/home/';
      url += window.location.search;
      url += window.location.hash;
      window.location = url;
    </script>
    
    response received. done.
    

    Reflection:
    The response data (arbitrary google.com request) comes back after the exec() calls are made. In order words, when Insee “Response received. Done.”, I would hope to have the google data already printed.

    The exec(success) call causes the thread ID to print, again before getting the google data.

    The goal is for the print statements to be in this order:
    1) request sent
    2) google data prints
    3) response received
    4) continue

    Thanks for everyone throwing code my way! Every snippet helps.

    In fact, that’s the challenge. (“Programming challenge! everyone join in!”). I would love to see the shortest code snippet (just the essentials, so its easy to see the blocking logic) that can:
    1) print a starting message
    2) make a request and print the response data when received (I’ve been using google html data, for no reason - its arbitrary)
    3) Only after step 2, print an ending message

    Nothing else. 3 MUST wait until 2 prints its response data.

    The drawing or other messages in some of the examples makes things more interesting, but also more complex and clouds the essential blocking logic. I just want to understand steps 1 - 3 in their purest form.

    Thanks again.

  • Posts: 502

    oh, that's really weird. Here's my complete test code with a screenshot of the final result I get on my iPad Pro.

    local thread_queue = {}
    
    local function exec(func, ...)
        local params = {...}
        local thread = function(self) func(self, unpack(params)) end
        table.insert(thread_queue, coroutine.create(thread))
    end
    
    local function thread_update()
        if #thread_queue > 0 then
            if coroutine.status(thread_queue[1]) == "dead" then table.remove(thread_queue, 1)
            else coroutine.resume(thread_queue[1], thread_queue[1]) end
        end
    end
    
    local function success(response)
        while not response do -- block until data from http request comes in
            coroutine.yield()
        end
        print(response) -- when data arrived, processing it
        print("response received. done.")
    end
    
    local function request()
        http.request("http://google.com/home", success)
        print("request sent. wait...")
    end
    
    local function continue_with_some_other_stuff()
        print("now that we completed the request and processed the response, it's time to continue with the rest of the queue")
    end
    
    function setup()
        request() -- start request
        exec(sucess) -- create a queue to work through
        exec(continue_with_some_other_stuff)
    end
    
    function draw()
        thread_update()
    end
    
  • dave1707dave1707 Mod
    Posts: 7,810

    @iam3o5am Here’s the code to do your 4 steps above. You never said what you’re doing before you want to do the request, or what you want to do with the data received, or how many request you want to do, or what you want to do when the request is done. It doesn’t take a lot of code to do the request, but what else you want to do will change the code below. It’s easy to write small code that does one thing, but it will take more code to make it work with the other code you don’t mention.

    function setup()
        request()
    end
    
    function request()
        print("request sent")
        http.request("http://google.com/home", success)
    end
    
    function success(data)
        print(data)
        print("response received")
    end
    
  • edited July 2018 Posts: 81

    @dave1707 - its not that I am holding back on what I am trying to do. I am just working on the concept of a blocking call. I just dont see it yet. The code you posted relies on the success function for the final (‘im moving on step’). The only reason this works is because there is nothing left in the setup function. But this isn’t realistic. In any larger program there will be more to do where the original call was made (setup in this case, but likely somewhere else in a larger program). In my mind, setup is just a callback to handle the response - not where the main code path would be.

    I apologize if Im coming across as frustrating on not communcating my intention, or now I suppose I could be accused of first wanting small code, and now saying something is not realistic and wouldnt be that way in a larger program. I recognize that, and so possibly my error is in my thinking. I need to change how I am approaching the problem.

    That said, what I mean by my 3-step goal is:
    1) main thread (doesnt REALLY have to be a separate coroutine, but I’ll call it that since its the main code path)
    2) the request is made, data received, using a callback whose job is to handle the request (print it, i suppose, for now)
    3) continue on the main thread

    To demo that NOT working:

    function setup()
        request()
        print("continue")
    end
    
    function request()
        print("request sent")
        http.request("http://google.com/home", success)
    end
    
    function success(data)
        print(data)
        print("response received")
    end
    

    The “continue” line is out of order above.

    Now, that is obvious to all of us. That is why everyone gave their original, longer suggestions. Either way, thats what I would like to solve in its purest form / shortest code.

    @dave1707 I do appreciate your contributions and I apologize again if this has been a frustrating discussion thread for you. Please dont feel obligated to respond if I keep hammering at this. Maybe the bottom line is, there isn’t a simple/pure/short way to block and that it requires looping around and checking variables and its implmentation is case by case specific.

    @se24vad I haven’t look yet at your updated post. On it now...

  • Posts: 81

    @se24vad Yours seems to work, but its because ‘success’ is misspelled in the exec(success) call. You have it as ‘sucess’.

  • edited July 2018 Posts: 502

    @iam3o5am good catch. so does it work now?

  • dave1707dave1707 Mod
    Posts: 7,810

    @iam3o5am It’s not that it’s frustrating, it’s just hard to give you an example when I don’t know everything you’re trying to do. Here’s another example of a program that does different things along with the http.request. Since I don’t know what all you want to do, I kind of just put in dummy routines.

    function setup()
        func=menu
    end
    
    function draw()
        background(40, 242, 191, 255)
        fill(255,0,0)
        func()
    end
    
    function request()
        print("requesting data")
        http.request("http://google.com/home", success)
        func=wait
    end
    
    function success(data)
        print(data)
        print("response received")
        func=process
    end
    
    function menu()
        text("This is a menu screen",WIDTH/2,HEIGHT/2)
        text("Buttons could be here to do different things.",WIDTH/2,HEIGHT/2-50)
        text("Since I don't know what that is\njust tap the screen to start a download.",WIDTH/2,HEIGHT/2-100)
    end
    
    function process()
        text("Processing received data",WIDTH/2,HEIGHT/2)
        text("Since I dont know what you want to do\nfor processing the data received",WIDTH/2,HEIGHT/2-50)
        text("just tap the screen to simulate end of processing to continue.",WIDTH/2,HEIGHT/2-100)
    end
    
    function continue()
        text("Here is a continue screen",WIDTH/2,HEIGHT/2)
        text("This is where whatever your program is going to do",WIDTH/2,HEIGHT/2-50)
        text("Since I dont know what that is\njust tap the screen for the menu again.",WIDTH/2,HEIGHT/2-100)
        output.clear()
    end
    
    function wait()
        text("Waiting for download",WIDTH/2,HEIGHT/2)
    end
    
    function touched(t)
        if t.state==BEGAN then
            if func==menu then
                func=request
            elseif func==process then
                func=continue
            elseif func==continue then
                func=menu
            end
        end
    end
    
  • edited July 2018 Posts: 502

    @iam3o5am ok, please try this one. The idea behind it is much simpler but the code has to be simplified yet. You create a variable for the response-data (and the response-status if you want to use it). Then start a blocking coroutine wait() which is blocking everything until the request variable got assigned, which again happens only after the request got a response from the server.

    local http_status
    local http_response
    local thread_queue = {}
    
    local function thread_update()
        if #thread_queue > 0 then
            if coroutine.status(thread_queue[1]) == "dead" then table.remove(thread_queue, 1)
            else coroutine.resume(thread_queue[1], thread_queue[1]) end
        end
    end
    
    local function exec(func, ...)
        local params = {...}
        local thread = function(self) func(self, unpack(params)) end
        table.insert(thread_queue, coroutine.create(thread))
    end
    
    local function listener(response, status)
        http_status = status
        http_response = response
    end
    
    local function process()
        print(http_response) -- do something with the http response data
        print("response received. done.")
        http_status = nil
        http_response = nil
    end
    
    local function wait()
        while not http_response do
            print("wait...")
            coroutine.yield()
        end
        process()
    end
    
    local function request()
        http.request("http://google.com/home", listener)
        print("request sent. wait...")
    end
    
    local function do_something_else()
        print("now continue doing other work...")
    end
    
    function setup()
        request()
        exec(wait)
        exec(do_something_else)
    end
    
    
    function draw()
        thread_update()
    end
    
Sign In or Register to comment.