Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

http.request Accept-Encoding: gzip in headers...

edited July 2013 in Questions Posts: 7

Hi.

http.request adds Accept-Encoding: gzip into headers so some sites send me gzipped content and data I get from them seems to be empty...

I tried to rewrite this header but could not...

What can I do? Can I disable adding this header or rewrite or unzip content to get htmls?

Tagged:

Comments

  • Posts: 7

    The concrete example

    -- Downloading data
    function setup()
        textData = nil
        params={}
        headers={}
        headers["Host"]="new.mcko.ru"
        params["method"]="GET"
        -- changing useragent does not solve the problem...
        params["useragent"]= "Mozilla/4.0(compatible; MSIE 7.0b; Windows NT 6.0)"
    
    
    
        params["headers"]=headers
    
        --   Request some data
        http.request( 'http://new.mcko.ru',
         didGetData, fail, params)
    
    end
    
    -- Our callback function
    function didGetData( data, status, headers )
        print(status..": ")
        -- This line prints empty string
        -- But I checked - server sends zipped data
        print(data)
        for k,v in pairs(headers) do print(k..": "..v) end
    
    
    end
    
    function fail(error)
        print("error")
    end
    
  • BriarfoxBriarfox Mod
    edited July 2013 Posts: 1,542

    Well data being returned is not nil. When I try to access it as a table it shows it's a string. I'll keep playing with it.

    edit: It looks like there are options for reading gzip data. I'm not familiar with gzip so let me do some more research.

  • BriarfoxBriarfox Mod
    Posts: 1,542

    Well if I encode data into Base64 I get a printable string.

  • edited July 2013 Posts: 7

    Briarfox, thanks!
    It works, but it is unuseful.

    How to unzip...

    Btw, how you encoded that?
    Has Codea standart functions for that?

  • BriarfoxBriarfox Mod
    Posts: 1,542

    I don't think the Base64 helps. I was just trying to see the string that was there. I'm still working on this. Anyone else have for knowledge on compression?

  • edited July 2013 Posts: 175

    Besides setting the Accept header on the request for just HTML : http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

    There is also the (reasonably unpleasant ;) ) option of implementing gzip/deflate in pure lua :)
    A quick google turned up a pure lua implementation (although may need some modification for codea) :

    https://github.com/davidm/lua-compress-deflatelua
    https://github.com/davidm/lua-compress-deflatelua/blob/master/lmod/compress/deflatelua.lua

    and for interest heres a pure JavaScript implementation :)

    https://github.com/beatgammit/gzip-js
    https://github.com/beatgammit/deflate-js

    The specs may also be of interest :)

    DEFLATE Compressed Data Format Specification version 1.3
    http://tools.ietf.org/html/rfc1951

    GZIP file format specification version 4.3
    http://tools.ietf.org/html/rfc1952

    Edit: I also happened across this which you could possibly coax to decode it for you :)
    http://i-tools.org/gzip

    or you could write a little Php script and use gzdecode to decode the data
    http://php.net/manual/en/function.gzdecode.php

  • Posts: 7

    Thanks, XanDDemoX!

    Please explain, how to use some of the choices you described.
    First about Accept header. I tried to disallow that, but could not.
    And that is the fastest way to solve the problem.

    Second about pure lua. That will be slow, but enough for me, I think.
    You were right. That code is not compiled by Codea...
    I got in line
    local crc32 = require "digest.crc32lua" . crc32_byte
    error that that module not found.
    How Can I add it to Codea?

    How to use JavaScript fo my purpose?
    I have no idea...

    And finally yes, I have thoughs about writing main code in PHP
    and then get Queries from Codea program and send to Codea the results only
    from the php script.
    But this does not seem to me a good Idea, because we need to organize special server - webpage for the program, and it will be difficult for many users...

    And Thanks, again!

  • edited July 2013 Posts: 175

    The accept header is probably your best bet for a simple solution. Setting it up like so should tell the server you want plain text or HTML instead of gzipped :) (it actually says to the sever i only support/accept these formats of data)


    local params ={ headers = {Accept="text/*, text/html"} } local success = function() end local fail = function() end http.request("url",success,fail,params)

    The other solutions i wouldn't recommend unless you absolutely have to as there is considerably more work involved :). The pure lua one would need the "inflate" part of the algorithm to be extracted or its dependencies to be included with it in Codea. The JavaScript one is really just for a secondary reference. The same goes for the actual specs of gzip and the deflate algorithm, just for reference of how gzip works :).

  • edited July 2013 Posts: 7

    Unfortunatelly, that header Accept does not help.

    This code shows 200: without any symbols after..

    -- Downloading data
    function setup()
        textData = nil
        params={}
        headers={}
        headers["Host"]="new.mcko.ru"
        params["method"]="GET"
        headers["Accept"]="text/*, text/html"
        params["headers"]=headers
    
        --   Request some data
        http.request( 'http://new.mcko.ru',
         didGetData, fail, params)
    
    end
    
    -- Our callback function
    function didGetData( data, status, headers )
        print(status..": ")
        print(data)
        for k,v in pairs(headers) do print(k..": "..v) end
    end
    
    function fail(error)
        print("error")
    end
    
  • Posts: 175

    Having a look at this it seems to work just fine and the accept headers are completely unnecessary also. It actually seems to be print is not printing the data which is 26611 characters long. Try this for the didgetdata function :) I'm just printing the first 255 characters.

    function didGetData( data, status, headers )
        print(status..": ")
    
        local count =#data
        print(count)
        print(string.sub(data,1,255))
        for k,v in pairs(headers) do print(k..": "..v) end
    
    end
    
Sign In or Register to comment.