Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

GLSL talk and demo

BortelsBortels Mod
edited January 2012 in General Posts: 1,557

So, GLSL. It's been in the buzz, mentioned in various threads - I've certainly been frothing about it. TLL has mentioned it may show up as a feature. What is it, and should you care? Let's find out.

First, a link to a free app: http://itunes.apple.com/us/app/paragraf/id422685475?mt=8&ls=1

That's "Paragraf", an app that lets you write GLSL stuff, much as Codea let's you write Lua stuff. Rather than wasting your time describing things, i suggest you download it and mess around for 5 minutes, looking at the examples. I hate their color scheme, but I love what they've done, and free is downright generous. 5 stars.

In a nutshell - GLSL let's you write special programs called "shaders" that execute on the GPU instead of the CPU. The GPU is optimized to do this sort of stuff very, very fast, massively in parallel, so some graphics effects can be thousands of times faster than they would be implemented on the CPU in any normal language. "shader" is a historic term - they used to be only for shading, but are now general purpose, usable for all kinds of graphical manipulation, including 3d transforms, filters, and even oddball stuff like Conway's Life game, which blows me away to this day. The Pargraf app examples are a good overview of the variety of effects you can achieve.

GLSL is insanely powerful, and profoundly crippled at the same time; it does one thing - graphics manipulation and effects - hugely well, and other things (like general purpose computing, or simple input/output) poorly or not at all. This is why a marriage of Codea and GLSL is so exciting - each can cover for the others shortcomings.

Fair warning - if there ever was an "advanced users" feature, this is it; I won't pretend I even understand how most of the examples work! But - learning is fun, and I'm perfectly ok swiping bits of code while I do.

Anyway - found this while spelunking, wanted to share. It's not in Codea, but I think it's firmly in the "desirable and likely" category - it was first suggested by @Dylan, one of the Codea devs. Plus, I've been raving like a lunatic about it - go look and see for yourself if I'm blowing it out of proportion.

Best of all - I can think of no conceivable reason why Apple would balk at its inclusion; there's even a sample iOS project up in github (easy to google for, but let me know if I should post a link), so it's hopefully straightforward to implement as part of Codea as well.

Tagged:

Comments

  • SimeonSimeon Admin Mod
    Posts: 5,061

    I'm not against the inclusion of GLSL, but there's a lot of editor UI stuff to handle, and exactly how we allow you to communicate with GLSL shaders from the Codea API:

    How do you set uniforms and attributes? How do you tell a shader to use an image as a uniform sampler2d? How do you multi-pass. And so on. The shader API (not the language) in OpenGL is esoteric, passing it off directly inside Codea seems ugly. I think there is an elegant solution, but it's one we will probably explore after some of the more requested features.

  • edited January 2012 Posts: 273

    Mr. Bortels I thank you for the pointer to Paragraf. This, dear sir, is trés cool. Love the camera input!

    Edit: @Simeon, if anyone can do elegant, you/TLL/Codea can.

  • BortelsBortels Mod
    Posts: 1,557

    I'd be comfortable simply passing them a string with the GLSL code. If you wanted to get fancy, special-handle a GLSL tab. As I said - advanced feature. But this stuff being hard doesn't make it bad - it makes it FUN, especially given the power it adds. Not everyone wants to ski the bunny slopes.

    I would suggest quick-and-dirty to begin with - if you call glsl(string), it won't dirty up things for people who don't want to use it. Then once we have some experience with it, a more refined implementation can be introduced. I like to think the various font libraries people wrote fed into your design decisions for native font support - a "preview" implementation may give you data for how to implement it elegantly down the line.

    Disclaimer: when I ski for real, it's always the bunny slopes... I am teh wimp.

  • SimeonSimeon Admin Mod
    Posts: 5,061

    Yes, but then how do you get data into your shaders? If you want to run your shader on an image, you need a way of telling OpenGL to link the texture object to a uniform in the fragment shader. This would have to be represented somehow in the Codea API. I'm sure it can be represented, but we would have to design that.

    We can do a basic version first if you like, one that loads strings. But we'd still need to design the API to provide input to the shader. And this would still come after the features we've already promised.

  • beebee
    Posts: 381

    Agree with @simeon. I think 2LL should focus first on features that can be used by EVERY users instead of features that can only be used by SOME (advance) users. ;)

  • edited January 2012 Posts: 273

    Hm... while it is undoubtedly true that (as users) our concrete desires can't all be met at once... it seems to me that (as users) we are best served when we are encouraged in our excitement and dreams.

    Edit: I read once how Mr. Jobs disliked 'focus groups of users' because he maintained that 'people don't know what they want.'

    Something to keep in mind when we love (and kill) our visionaries/crazy ones.

    PS: But yes, if I had to state my own druthers, I'd want to see text, text input and more graphic primitives first... but only because I currently have clear (albeit limited) dreams of how I would put them to use. Recognizing my limitations I'm very grateful to Mr. Bortels for his effort in introducing something I didn't know about and broaching the subject with Simeon.

  • SimeonSimeon Admin Mod
    Posts: 5,061

    @Blanchot most of the features I'd like to include first are features that I personally want (physics, custom sprites, text). GLSL is nice, too. And we'll get there. Most of the delays are due to Apple's review times anyway.

  • edited January 2012 Posts: 273

    @Simeon good to hear that you are "stepping to the music" which you yourself hear! At least that's what's most important in my book. Cheers!

  • edited January 2012 Posts: 2,161

    paragraf looks nice. They could learn from the Codea guys with regard to the editor, though. I found myself looking for the cursor keys ... say no more!

    Being able to interleave this with lua would be great.

    I notice that paragraf has code sharing - you can email a shader (not sure about import as yet).

  • Posts: 384

    That was pretty cool... :) leaves one a bit dizzy though! It would be a great feature down the track to allow the camera or other arbitrary input data to feed a shader. Thanks, @Bortels for explaining it.

  • BortelsBortels Mod
    Posts: 1,557

    Yes - Codea has crippled me keyboard-wise, I habitually look for cursor control characters that aren't there in other apps. They seem to be using the bog-standard text input box, which we all know sucks for coding. (total aside - if you don't already have it, get the free wolfram alpha app - check out THAT keyboard.)

    Paragraf is also crashy, unfortunately. I don't know if that's inherent in GLSL or it's the app; but I'm inclined to point at the app, since I'm just trying their "help" examples. Still - you get more than you paid for it, so judge lightly.

    How to get an image to a shader? Presumably you pass a pointer (or more exactly, an image() object) to it, along with the GLSL code, with nil meaning the screen? More interesting to me is output - presumably for maximum flexibility, you'd render back to another image(), not just to the screen - you could then have multiple GLSL pipelines for multiple effects. I liked the idea you had in the "we want video" thread, about having the camera live feed or a video decoding into an image, that you could then sprite() as usual. Can GLSL do that (support multiple pipelines)? I suspect so, I can run multiple quartz composer projects on the mac in different windows, but I'm by no means sure - thus the research and this thread, where I hope to nail down details and expand mindshare (aieeeeee I sound like my work). Point is - how we pass in and out data are not necessarily things I'd get to pick - they depend on architectural details of the internals of Codea, and lua, and GLSL, none of which I can speak to with authority.

    But I'll tell you this - right now, the only thing I found on the app store that let you do GLSL was Paragraf, and it's a toy. If you added GLSL, especially in a manner like the above (multiple inputs and outputs, ooh chained maybe) you'd be all alone in a hot, hot niche - this technology is really new, and a lot of graphics-heads miss quartz composer on the iPad, and this would blow QC away in terms of flexibility and ease-of-use. (One of QCs biggest assets was also its biggest flaw -visual programming. Big projects get complex, and look like spaghetti - trying to keep code legible is hard enough with text, with pictures it gets much worse)

    As for when - I won't tell you this should bump things like native fonts or camera access (just the opposite!) - but I also know we're mostly waiting on apple anyway. I merely suggest that, done right, this could make Codea something more than simply self-hosted lua with some useful libraries and a nice editor. Those are sufficient and good, but lua and a good editor don't make people say "holy crap, how'd they do THAT?". I'd prioritize that higher than some of the other things suggested, myself.

    I think this could have game changing potential. VisiCalc single-handed ly made the Apple II a requirement for business. I think this has the potential to be that important to some people. Of all the features suggested, this has crazy "bang for the buck".

  • edited January 2012 Posts: 273
    But I'll tell you this - right now, the only thing I found on the app store that let you do GLSL was Paragraf...
    

    There is also an app named RenderDuck (also free, and also, like Paragraf, filed in my appstore under 'Education').

  • BortelsBortels Mod
    Posts: 1,557

    Neat - good find on RenderDuck.

  • Posts: 2,161

    Aaaggghhh ...

    Too many years of Perl, PHP, TeX, and now Lua have ruined what little C skills I ever had. I've lost count of the number of times paragraf has complained about me writing stuff like if (a > 1) {...} where a is a float. And I've only written one shader so far.

  • NatNat
    Posts: 143

    Wow! Paragraf even runs on the iPhone. That is amazing.

  • NatNat
    Posts: 143

    I've spent today's commutes writing GLSL code on my phone with Paragraf. Bloody amazing app! I'm now sold on Bortel's idea of GLSL tabs in Codea. I want it!

    I wonder if GLSL input/output variables could be bound to Lua globals in the same way that Codea's watches and parameters are at the moment. That could make for some really cool effects.

  • BortelsBortels Mod
    Posts: 1,557

    That's the idea, kinda. You wouldn't want them all bound, but a mechanism by which you can either pass in or bind variables would be important - Note, for example, Paragraf makes the current touch (up to 3 multitouch, actually, I think) available. As well as the image from the camera, of course.

    I'd also like to see outbound binding as well. Thinking out loud, assume we have a glsl() routine that accepts as parameters some code and an input context (ie. an image). You might do something like this:


    sepiatone = "some GLSL to do a sepia tone shading" lens = "some GLSL to do a lens effect" sprite(glsl(sepiatone, glsl(lens, glsl(camera))), WIDTH/2, HEIGHT/2)

    Idea there being you have a "camera" bit of glsl (or hardcoded stuff) that simply returns the current camera image. To apply a GLSL effect, you call glsl with the code, and the input - so you can nest them to chain them. glsl returns an image - that's either used as input to another GLSL, or composited to the screen.

    We'd want some way to refer to a few "magic" contexts - "frontcamera", "backcamera", and "screen" come to mind. (we could use setContext(), but I can see it being messy, not that the above isn't).

  • Posts: 2,161

    Bortels (or anyone else): How does one do an array in GLSL? I've been trying all sorts of things but I can't get paragraf to accept any of my attempts. I really want an array of vec2 objects, but I'd settle for two arrays of floats.

  • JohnJohn Admin Mod
    edited January 2012 Posts: 587

    Found another one: GLSL Studio. It came out yesterday.

    Edit: I just noticed they have Code Export, via Document sharing in iTunes. It will be interesting to see if apple cares about sharing GLSL code.

  • SimeonSimeon Admin Mod
    Posts: 5,061

    Sharing GLSL code is significantly more of a security risk, given that your code is compiled and run directly on the hardware and can potentially violate the sandbox.

  • Posts: 2,820

    Haha. Things are looking up for Codea iTunes project sharing... or not...

  • BortelsBortels Mod
    edited January 2012 Posts: 1,557

    @Simeon - I don't agree, but I'm open to the argument - can you cite a case of GLSL actually being used to expoit a system in the real world? Or even in theory in literature? I've been unable to find one. But I can find TONS of examples of a general purpose language like Lua (and javascript and C and every other one, really) being used to get around security.

    Google's NACL basically runs at the raw hardware level - and yet is one of the most demonstrably secure sandboxes to date. Closeness to hardware doesn't make things less secure, necessarily.

    More to the point - GLSL and it's C-like syntax do not compile down into a machine code - they're compiled by the GL drivers from the GPU's manufacturer into instructions for their cards, which are interpreted by the low level hardware running a local machine code (in other words, as I understand it, internally they're a p-code or virtual machine - there is, in effect an OS and hypervisor running that deals with things like distributing work and arbitrating access to the bus and so on). And given the cards themselves don't have access to main memory, or filesystems... I don't see how any program could "get out of the sandbox" - the sandbox is the GPU or graphics card. Presumably, an exploit would have to be against the systems getting the GLSL to the GPU - the OS graphics drivers - and presumably those are open to abuse the same as any other system service, ie. no added risk.

    Could malicious GLSL hose your video, forcing a reboot (or restart of the video at least)? Of course - hell, I do it accidentally. But being able to make something break doesn't imply you can exploit a security hole.

  • SimeonSimeon Admin Mod
    Posts: 5,061

    I don't know enough about the hardware to say, but it just seems that with a shared-memory video system any possible exploit in the shader hardware might be able to bypass memory protection.

    One of the reasons Firefox was (and is) hesitant to adopt WebGL is due to security issues. I don't know whether there's any merit to them, but some are detailed here: http://www.contextis.co.uk/resources/blog/webgl/

  • BortelsBortels Mod
    Posts: 1,557

    Those security concerns, from what I understand, are from attacks on the GLSL and WebGL drivers - not security concerns from within. Yes - that a weird fine line. I think the key point is that until recently (webgl), there was no real potential for outside code to get to poke at the video card drivers at that level (again, at the "load some GLSL" stuff, not when executing it, because the stuff executing it is fairly removed from the system), and so the drivers had virtually no real hardening - they were an easy target.

    And - that may still be the case in iOS - I note mobile safari does not have WebGL yet, and you can bet Apple is both looking at doing it, and aware of the issues the PC browsers have had.

    Point being - I'll worry for real about something being a security threat when I hear of someone in the real world, researcher or otherwise, actually suggesting (or demonstrating) that it's plausible. I don't see anyone, ever, using GLSL code to exploit anything. Attacks against the drivers are just regular machine code attacks - a buffer overflow on the GLSL compiler in the driver or such - that happen on the regular old CPU). Doesn't mean Apple isn't susceptible, but I'd wager their attack surface is way reduced, and they are approving apps that do GLSL, as we've seen. (I want to try the new one, but TWO BUCKS!!!! highway robbery. :-) )

  • SimeonSimeon Admin Mod
    Posts: 5,061

    Just to make it clear, I'm not worried about the security issues of GLSL or WebGL. I think they're fine. But I think that Apple could be (and I might be if I was in Apple's position).

  • BortelsBortels Mod
    Posts: 1,557

    We see 3 apps that do GLSL approved in the last few weeks (I lie - I don't know how long paragraf's been there). So - they seem to get by.

    From a security engineering standpoint (and that's a lot of what I do 'for reals', which is why I argue about it, not that I need an excuse), setting aside my desire to see GLSL, I see this as significantly less a threat than Codea itself. :-) So it seems unlikely Apple will, either.

    Having said that - show me one exploit, even theoretical, and I reserve the right to change my mind.

  • Posts: 622

    I sort of hesitate discussing the issues because one has to be careful about giving people ideas.

    I think I can explain it in a way that is of no pratical use but gets the point across.

    Two avenues of exploits off the top of my head are firmware access and cache and transfer.

    Everything has firmware and it is meant to be upgraded. Just about anything can be told to respond to the rest of the system as if it was something that it isn't. It doesn't matter what the current iteration of GSGL does, it matters if it will ever have the addition of some kind of cross platform firmware access. At that point it crosses a line into chaos and a target for ultra coordinated MFery. (FYI any motherboard on a server with built in video is an accident waiting to happen, btw this is most of them, servers should emulate video not have a real one)

    For cache and transfer it's best to start by talking history. In the days of yore CRTs by their physics alone broadcasted what they displayed omnidirectionally. You had to be close and it would be blurry but you could read screen over the air though a (unshelled) wall. Video memory is meant to be shared for speed, it's a gap in and of itself. You cache what you see and you store it for later distribution. But all that is dependent on compromising other components so it really a secondary issue. It's like "someone could steal our passwords" (right after they broke down the front door and don't need them).

  • BortelsBortels Mod
    Posts: 1,557

    Mmmm... a stretch, at best. The first talks about an imaginary vulnerability, maybe, someday. The second is a data leak, not a security exploit in the sense we're using here (or, to put it another way, the second one is exploitable even if we don't have GLSL access).

    I keep coming back to exposure - we're talking about bolting the 3rd deadbolt on the upstairs attic window, when the garage door is wide open, comparatively. We're talking about maybe someday exploitable stuff in a hugely limited language, sitting on top of a full blown general computing language on a platform that's been jailbroken (and so is presumably vulnerable to reverse engineering of exploits). If I can compromise the base OS - I can do whatever GLSL I want.

    I think the chances of GLSL exposing a vulnerability of any real consequence that was not exposed in some other manner (ie. by virtue of running on the GPU) is about as close to "vanishingly small" as you are going to find in the real world. And as always - open to other evidence (or speculation...)

    (I always assume the people smart enough to exploit already know what I'm going to say - but I understand, I keep dancing around the things I could do with sockets. Crazy powerful also means potential for abuse...)

  • Posts: 2

    Hi! I created GLSL Studio, found this page while google-stalking "GLSL Studio" and figured I'd jump in to let you know some of what I dealt with in getting approved. :D
    I submitted around mid December last year and got approved 2 days ago. I got rejected twice for the usual "external code" then took it to the appeals board who finally approved. Where this is relevant is that I am only allowed to export - iTunes File Sharing is enabled but only one way - any kind of import (i.e. "downloading code") is still forbidden.
    This is really unfortunate since sharing of files would be a huge benefit to everyone and I also agree that GLSL is probably the least likely point of entry for any exploit. That said, I totally understand their caution. I'm hoping that at some point in the future they will relax this stance a little, or at the very least treat things on a case by case basis (which it looks like they may be since they've let apps like Codea + GLSL Studio into the store).

  • BortelsBortels Mod
    Posts: 1,557

    @kode80 - welcome! Heh - I sprung the 2 bucks last night, had to poke your app and see what I could see.

    You're really in the same boat here as Codea is - I notice I can cut and paste into your app, so there's the "back door" import right there.

    I can tell you from experience that when you can simply import the file, as opposed to making a new project and cutting and pasting, the end-user experience is entirely different. To be blunt, Apple's text selection specifically does a poor job of allowing users to easily select a chunk of code to copy and paste - there are a million things common done on web sites (including github!) that break it. When it's just 'click on this in mobile safari and choose import to your app', you'll see a ton more activity and sharing with your users.

  • Posts: 1,255

    Apple's policy does nothing but put a fig leaf over security, but it puts a serious hurt on ease of use.

  • SimeonSimeon Admin Mod
    Posts: 5,061

    @kode80 interesting! I'm glad you were able to sort out your issues with the appeals board.

    We had similar one-way file sharing through iTunes but were rejected due to "downloadable code," as well. Even though, as you say, it's not downloading because it can't import code into the app.

  • Posts: 2

    @Simeon wow, sorry to hear that. If it's only exporting there's no way that could be interpreted as "downloading code" I'd take that to appeal, my experience definitely renewed my confidence in the reviews process somewhat.

  • Posts: 622

    Thanks for coming by @kode80

  • Posts: 2,820

    Alright. Now that I've seen what you can do with GLSL like making 3D objects, I REALLY want it in Codea. Now another suggestion for sharing code: Will apple allow it if you just export the project as a zipped .txt files (and sprites when you get to it?) It's normal file types and anybody could read it. Or how about adding a feature that when you create a new project or class/blank page, it gives you an option to import from clipboard.

  • SimeonSimeon Admin Mod
    Posts: 5,061

    @kode80 yeah, thanks for sharing your experience. It was really informative.

  • edited January 2012 Posts: 2,820

    Getting his app soon to try it out.

  • Posts: 502

    Nice that you like my old app Paragraf! Hopefully Codea will replace the need for it with the next release. Then it's just an iPhone version that's missing. :)

Sign In or Register to comment.