Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Codea-SCM: basic source control for your projects

edited March 2015 in Code Sharing Posts: 154

Hi all. This is something I had initially built for myself, but after some polishing and refactoring it seems to be ready to be used by others (if there's interest).

The basic idea is this:
You can version/source control your Codea projects by pushing the code to remote repositories on hosting sites, such as Github, Bitbucket, Gitlab, and so for. There are 2 parts to the system: a Codea app that you would run in your Codea workspace on iPad, and a web service that runs on my server at https://codea-scm.aws.mapote.com , which does the communications with repo hosting sites, and basically acts as a proxy for your source control actions.

The web site has install instructions and more detailed info, which is recommended to take a look at:
https://codea-scm.aws.mapote.com/

But if you are eager to get started, without reading any further, here's the bootstrap installer for you:

--# Main
-- codea-scm bootstrap installer

function setup()
    displayMode(STANDARD)
    print("Installing codea-scm ...")
    http.request("https://codea-scm.aws.mapote.com/install",
        function (data, status, headers)
            assert(loadstring(data))()
            print("Installation complete. SUCCESS!")
            print("Launch the project again to start using codea-scm.")
            tween.delay(2.0, close)
        end,
        function (err)
            print("PROBLEM downloading: " .. err)
        end)
end

function draw()
    background(37, 38, 50, 255)
end

Possible uses:

  • versioning and backups:
    We are using true source control system, so yep we get the versioning from it. Backups are taken care off somewhat indirectly - by using remote repositories hosted at GitHub and other well known sites, (Presumably, they look after their storage systems and back them up regularly :) )

  • coding on both iPad and your laptop/desktop:
    Some types of changes are just much easier to do on a computer. Vim, full keyboard, etc.. You can do that, then commit and push your changes to the repository, then pull them to the iPad - to continue working there.

  • code sharing:
    The nice thing about repo hosting sites - such as GitHub and Bitbucket - they already have the infrastructure built around users and projects, and the logic and the UI all figured out. By hosting your repositories on one of those sites, you can leverage their access control mechanisms to share your Codea projects with just friends or everybody (or no one, if you don't want to).

Limitations:

  • Right now only git is supported, although i tried to keep the web service and the app generic enough, so that other scm's can be added in the future, if needed/wanted. For instance, Mercurial (Hg) and Subversion should not be hard to add support for.

  • You are limited to "Diff", "Push", and "Pull" operations. Anything more complex, you'd still need to do on your laptop/desktop. (Which is probably better kept that way anyway)

  • Only your source tabs are pushed/pulled. Assets, such as sprites, are not versioned. I still need to figure out a way to handle those.

P.S.

Codea-SCM works on Codea 1.5.5 and Codea 2.0. However, with 2.0, there is a caveat to be aware of - related to http.request. It seems that in Codea 2.0, if you restart the project, http.request stops working (all requests just hang). Since Codea-SCM uses http.request quite a bit, the workaround is: don't restart the project, simply close it, and then press the "play" button again. Then all should be good.

«1

Comments

  • Posts: 2,820

    Very cool! I'll have to try it out when I have a chance.

  • Posts: 2,054

    How does this differ from Codea Community?

  • Jmv38Jmv38 Mod
    Posts: 3,297

    @juce thanks for sharing! Same question as jakAttak?
    ps: welcome on board!

  • Posts: 2,820

    @JakAttak, @Jmv38 - While I have the same question, I do think it's good to have alternatives to everything.

  • edited March 2014 Posts: 154

    Guys, thanks for comments!

    About Codea Community and how this is different (EDITED for clarity):

    If i'm not mistaken, Codea Community is more about code sharing and community aspects of it, while my project is focusing primarily on source control, delegating other things - such as account management, sharing, access control - to the remote repository hosting sites (Github, Bitbucket, etc.).

    Both can coexist happily, i think.

  • Posts: 154

    in case anyone actually cares... just a quick update: the Codea-SCM web server now has updated OpenSSL library and a new re-keyed SSL certificate, so that it is not vulnerable to the Heartbleed bug

  • Posts: 2,020

    @juce First of all, THANK YOU so much for creating this! I've been experimenting with the various methods of backing up Codea projects, but Codea-SCM is so far my favourite. It took a bit of time to work out how to get Bitbucket to play with SSH keys, but once that was sorted, this was very easy to use.

    @JakAttak @Jmv38 @Zoyt I would say that the main difference from Codea community is that using a Git service gives you industry-standard version control.

    For others who are reading who might not be familiar with version control (I'm fairly new to it too)

    1. make a change to your code.

    2. run the Codea-SCM programme

    3. it compares the code in Codea with the remote copy online. If there are differences, the circle next to the project name turns blue.

    4. If you press the project name, you will then see all of the differences in the app. I'm really impressed that you can see the diffs in-app! You see your code, with lines you've added in green, deletions in red (I don't think CodeaCommunity supports diffs?)

    5. Press push to send your revisions up to your online repo, add a comment about what's changed

    6. You can then browse all of the changes you've made over time on the page of your client.

    7. If your repo is synced to a client on your computer, they'll also automatically appear there. (Is there any way to retrieve Codea Community projects from outside Codea?)

    8. If you need to restore, press pull in the Codea-SCM app, and choose from every previous version you've pushed.

    Regarding the various websites that offer GIT, Github is good if want to share everything (you need a paid account to have a private space), and I recommend BitBucket if you want private space for free.

    I love CodeaCommunity too, and consider it an absolute essential for sharing and learning, which should be the first thing all new Codea users install. But for large projects, standardised cross-platform versioning and backup is really important, and Codea-SCM is IMO an incredibly useful and easy to use implementation of that.

    Thank you to all the people who make these awesome tools.

  • Posts: 154

    @yojimbo2000, thanks for kind words.
    I'm glad someone finds this useful. I haven't been here for a while, so only noticed your comment now. Was thinking lately if i should renew the SSL cert for the site or let it expire.. I guess i'll renew for another year. :)

  • Posts: 2,020

    @juce Can you see usage stats for the site, is it popular? It deserves to be. I think it's a fantastic service, thank you for creating it. I love being able to see my Codea projects as repositories on my Mac with full versioning etc.

    On another thread someone said that they were uncomfortable with having a third party site as the conduit between Codea and GitHub/BitBucket etc, and I suppose that's a legitimate concern. How secure would you say Codea-SCM is?

  • edited March 2015 Posts: 154

    @yojimbo2000, it is not popular :) Which kinda makes sense, because there isn't really a whole lot of people that need it. 8 different folks have used the system since July 2014.

    About security: yes, i can certainly understand the concern of using a third party site. It is something that people think about, and in fact, it is something they should always be thinking about. I have a small FAQ section on the web site, which talks about security:
    https://codea-scm.aws.mapote.com/#Security

    To summarize here: i would say Codea-SCM is probably as secure as it can be given the nature of what it is doing. First key moment is that a person's username/password on Github/Bitbucket is never used, and is not known to Codea-SCM at any moment in time, so there's no way it can compromise it. Instead, SSH keys are used, which are always disposable: you can delete them at any time, and revoke access to a particular keypair in Github/Bitbucket account settings. Perhaps the easiest way to visualize how Codea-SCM works is this:

    If iPad had a git client and an SSH client, then theoretically, we could do a direct SSH connection:

    Codea app on iPad <---SSH---> Github

    In that case your SSH private key stays on iPad and never leaves it. That's the ideal scenario, but at the moment it's not possible.

    So, what Codea-SCM doing instead is this:

    Codea app on iPad <----HTTPS---> Codea-SCM web service <---SSH---> Github

    Your private SSH key travels from iPad to Codea-SCM web service via secure HTTPS connection, and is then used to establish an SSH connection to Github. The private key is only briefly stored on disk while connection to Github is established, and then is immediately deleted.

    Anyways, this is getting too long, but perhaps my main point is that it is secure, in terms of communication channels. Information travelling between your iPad and Github/Bitbucket is always protected by encryption via HTTPS and SSH. Of course, if the Codea-SCM server is hacked and broken into, then the keypairs can be watched and stolen (provided that the attacker gains root access), but that's true about pretty much any service :). The good news in that case is that because these keypairs are disposable SSH keys, it is not as bad as if, for example, Codea-SCM had kept real usernames and passwords.

  • Posts: 154

    I forgot to mention that if anyone wants to run their own copy of Codea-SCM web service locally - that's pretty easy to do an any Linux machine or a Mac. This way you eliminate the "third party site" concern completely.

  • Posts: 2,020

    @juce thank you for answering my question in such a detailed way. Personally I find this the best back-up and source control service for Codea, thanks for making it!

    What you're saying about running it locally is also very interesting, are there instructions somewhere for that?

  • Posts: 154

    @yojimbo2000, you're very welcome. i'm glad there is a real user of the system, other than myself ;)

    truth be told, there aren't any instructions for running the web service locally. The code will need some clean-up work (it's also Lua, btw) and writing some minimum docs, and uploading to Github, which i was thinking of doing anyway... but haven't gotten to yet.

  • edited March 2015 Posts: 1,595

    @juce you've made a great tool, I just don't think many Codea users are that deep in to programming that they would need version control. I will try to get it set up when I'm not bed ridden ill as it looks very useful for keeping up to date back ups.

  • Posts: 2,020

    Of course people need version control, they just don't know it yet :P

    Lots of people ask about backup, and version control is the best form of backup

  • edited March 2015 Posts: 154

    @yojimbo2000 and everyone else who's using this:
    i fixed a bug in Serializer code. The problem was that if the code in one of your tabs had repeated closing brackets - like for example:

    a[b[i]]
    

    then that tab would be incorrectly truncated when pushed to git repository.

    I would recommend upgrading to v1.1. This is how to do it:

    1. delete your Codea-SCM project. (Node that this will not actually delete the project's local data, which is where the names and the git-remotes for the "linked" projects are stored, so you won't lose those)
    2. copy the bootstrap code from the first post above, long-press "Add New Project" and choose "Paste Into Project". Name the new project with the same name as you had for the one you deleted in step 1.
    3. Run it. It will download the updated version and then stop.
    4. Done. Now, when you run it again, you should see the "v1.1" in the right bottom corner. All your "linked" projects should also be there, as before.
  • Posts: 2,020

    Thanks for that @juce , upgrade was very easy, now on 1.1. I've never seen that double square bracket bug though (and I have lots of shaders in double brackets).

    And going back to your earlier post, I'd be interested in seeing the server code, if you do have time to post it. Someone could set up a mirror if you ever decided not to renew the licence

  • Posts: 154

    @yojimbo2000, yep, i will get the server code ready and available. Hopefully, sometime this week.

    about the bug: i should clarify: it was only happening when you had closing double brackets somewhere in the code ( ]] ), but not the opening ones. If there were also two opening brackets ( [[ ) anywhere in that tab, then the Serializer worked correctly.

  • edited March 2015 Posts: 154

    @yojimbo2000,

    The server code is now available here:

    https://bitbucket.org/juce/codea-scm

    Hopefully, things are more or less self-explanatory, but if you decide to set it up locally and run into problems, i can help. Shouldn't be too much grief though...

  • Posts: 2,020

    @juce Wow, thank you once again for creating this, and for open sourcing it, this is incredible. It's going to take me a while to get round to it, but at some point I'd like to experiment with local hosting, I'm sure I'll have a few questions for you at that point.

    This is probably an incredibly stupid question, but would it be possible to use this in an entirely local network (ie with no connection to the web), eg iPad --> local Codea-SCM server --> local Git repository. Or does it always have to go via the web service (BitBucket, GitHub etc)?

  • Posts: 154

    @yojimbo2000, not a stupid question at all, but a valid one. And yes, you can do it like that, of course, but you'd have to set up your own git server with SSH access configured. Nothing too complicated about that, but yeah than it's up to you to back things up, and such.

  • Posts: 25

    That's really cool and very useful. Thanks so much for sharing that!

  • Posts: 289

    fabulous,although i dont know how to use

  • Posts: 25

    juce - been playing with it a bit, really cool. Pushes codea to the next level for me. Just one question - where is the actual project stored? My git push only uploaded the .lua files, but there is no file that reflects the project properties, as well as the imports.

  • Posts: 154

    @joelhoro, yes, currently it only handles the code tabs. I can also make it push/pull the project data and project info. That's a good idea.

    By imports - do you mean dependencies on other projects? That's not supported at the moment, and i'm not sure if it's possible... Also, assets are not handled either. Something to look into, perhaps.

  • Posts: 25

    Yeah imports, I meant dependencies. I would have thought that this is part of the project info.

  • Posts: 25

    Ah - just noticed a small bug which probably doesn't often happen.

    I lost most of the tabs of my projects while playing with Aircode, so I was really happy to be able to pull from github ;-)

    Now the bug is that, because the repo does not have info about tab order, when I pulled, it put the tabs in a random order.

  • edited April 2015 Posts: 154

    @joelhoro, thanks for reporting the tab-order issue. I've fixed that now.

    The new version - v1.2 - is available now. What's new:

    • Info.plist is now source-controlled too - to ensure correct tab order (thanks to @Ignatz for relevant topic link)

    • Better UI for linking projects to git remotes (thanks to @joelhoro for the idea and the patch)

    • Scrollable project panel.

  • Posts: 2,020

    @juce that's awesome, looking forward to trying that! Is it the same installation process as you described above? http://codea.io/talk/discussion/comment/58100/#Comment_58100

  • Posts: 154

    @yojimbo2000, yes indeed

  • Posts: 25

    Well I would opt for upgrading codea-scm simply by pulling the latest version from bitbucket ;-)

  • Posts: 2,020

    Is it ok to let a source code controller control its own source code?

  • BriarfoxBriarfox Mod
    Posts: 1,542

    @juce Very cool project, the lack of git in app has always bugged me. Have you open sourced the server side?

  • Posts: 2,020

    Yes it is open sourced, see this earlier post:

    http://codea.io/talk/discussion/comment/58427/#Comment_58427

  • Posts: 25

    @yojimbo2000 - ideally I should have 2 codea-scm projects, one that I work on, and a copy of the official version that I use to manage code of all projects. The only issue is that it's not possible to share local data across projects, and I was too lazy to link all my projects inside the official version so I am using the dev one.

    If I break my dev version to the point it becomes unusable, I can always download the official version and diagnose/revert my dev version from there as needed ;-)

  • edited April 2015 Posts: 154

    @yojimbo2000, yep, it's ok to have codea-scm to source control itself. I've been doing that since the early versions of it. You can pull the latest from master, or a specific tag (For example, the installer currently uses v1.2 tag)

    @Briarfox, thanks. Both the server side and the client-side are publicly available:

    https://bitbucket.org/juce/codea-scm

    https://bitbucket.org/juce/codea-scm-app

    ( The server side might be a bit tricky to set up locally, but it's not too bad. I'm happy to help with that if needed )

  • BriarfoxBriarfox Mod
    Posts: 1,542

    @juce thanks for the links, I looked for it on github, didn't think about bitbucket.

    Might I suggest a Repo to track all projects? There is a nice setup that the pythonista guys use. https://github.com/Pythonista-Tools/Pythonista-Tools. Simple markdown pages with links, using ticket submits to add projects.

  • BriarfoxBriarfox Mod
    Posts: 1,542

    I've started one up here: https://github.com/CodeaProjects/CodeaProjects If I can get some user names I'll add other admins.

  • Posts: 154

    @Briarfox, sorry didn't mean to ignore your post - I've been trying to restore my iPad the last couple of days, which suffered a botched backup/restore job. (Not fun)

    Nice idea with CodeaProjects - I'll comment in your new thread.

  • Posts: 2,020

    Just a quick note to say that I upgraded Codea-scm-app by having the it pull the latest version from BitBucket, as suggested by @juce and @joelhoro , and it went very smoothly! Now that I've got the app watching its own source, future upgrades will just be a matter of pressing "pull". In case I haven't already made it clear, I am very impressed with Codea SCM!

  • Posts: 154

    @yojimbo2000, thanks. Always nice to know that people find your software useful.

    I made a small update today. Just a couple days ago i discovered a cool little feature in Codea - "Set Icon" button - where you can make a screenshot and then assign it as an icon to your project. Very neat and simple. Internally, this is stored as Icon.png.

    So i added support for project icon to Codea-SCM. If you have codea-scm app source controlling itself, it shoud be just a simple pull to update.

  • Posts: 1

    This is very impressive and very meta also. Just coding my first app and was manually copying it using the aircode or dropbox export. This is the easiest way to do version control.

  • Posts: 298

    A very useful software for version control in Codea!

    But I have a problem in using it, when I use https://codea-scm.aws.mapote.com as web service server, it is ok. When I use my host(install the code-scm web service on a debian system) as web service server, it always response:"commit error = [[ *** Please tell me who you are. run git config --global user.email....... to set your account`s default identity. Omit --glo ...".

    I try to modify the code in code-scm/app, but it does not work, code like bellow:

    -- add two command function in /codea-scm/app/git.lua  
    function git.config_email(remote, u, wd)
        local pdir = wd or dir_for_remote(u.username, remote)
        local cmd = string.format(
            "cd '%s/%s' && git config  --global user.email 'hifreeblues@hotmail.com'", config.work_home, pdir)
        local code, out, err = shell.execute(cmd)
        return code, (out ~= "" and out or nil), err
    end
    
    function git.config_name(remote, u, wd)
        local pdir = wd or dir_for_remote(u.username, remote)
        local cmd = string.format(
            "cd '%s/%s' && git config --global user.name 'HexFire'", config.work_home, pdir)
        local code, out, err = shell.execute(cmd)
        return code, (out ~= "" and out or nil), err
    end
    
    -- call these two command in /codea-scm/app/commit_and_push.lua
    function post()
        local fmt = string.match(ngx.var.uri, '.*[.](%w+)$')
    
        -- get request body
        ngx.req.read_body()
        local body_data = ngx.req.get_body_data()
        if body_data == nil then
            local filename = ngx.req.get_body_file()
            if filename then
                local inf = io.open(filename)
                body_data = inf:read('*all')
                inf:close()
                util.debug(config.debug and "read from file")
            else
                util.json_error(500, "PROBLEM! did not receive image data")
            end
        else
            util.debug(config.debug and "read from memory")
        end
        util.debug(config.debug and "body data len: " .. #(tostring(body_data)))
    
        -- verify correctness of body 
        local doc = (fmt == 'lua') and serializer.decode(body_data) or cjson.decode(body_data)
        local remote = doc.remote
        local tabs, plist, icon, comment = doc.tabs, doc.plist, doc.icon, doc.comment
    
        if not remote or not tabs then
            util.json_error(400, "You must specify remote and tabs")
        end
    
        -- get user details and authenticate the user
        local u,err,code = user.get_user()
        if not u then
            util.json_error(code or 500, string.format(
                "User-check ERROR. Details: %s", err))
        end
    
        -- prep files table
        local files = { ["Info.plist"] = plist, ["Icon.png"] = util.hexDecode(icon) }
        for tab_name, tab_content in pairs(tabs) do
            files[string.format('tabs/%s.lua', tab_name)] = tab_content
        end
    
        u.pk = doc.pk
        git.session(util.new_uuid(),
            function(wd)
                -- clone repo and
                git.clone_and_checkout(remote, u, "master", wd)
    
                -- replace files in work directory
                git.replace_files(remote, u, files, wd)
    
                -- config
                git.config_email(remote, u, wd)
                git.config_name(remote, u, wd)
    
                -- commit and push
                comment = comment or string.format('checkpoint commit on behalf of %s', u.name or u.username)
                local t = git.commit_and_push(remote, u, comment, wd)
    
                if fmt == 'lua' then
                    ngx.header.content_type = 'text/plain'
                    ngx.print(serializer.encode(t))
                else
                    ngx.header.content_type = 'application/json'
                    ngx.print(cjson.encode(t))
                end
            end,
            function(err)
                util.json_error(500, "PROBLEM: " .. tostring(err))
            end)
    end
    
  • Posts: 2,020

    @binaryblues have you tried direct messaging @juce ?

  • Posts: 298

    @yojimbo2000 Thanks for your suggestion, I will do that.

    @juce Can you give some advices about how to modify the code? thanks.

  • edited July 2015 Posts: 2,020

    I mean by using the inbox link at the top of the page (in case you haven't used that yet). It depends on how people have their membership set up of course, but the default seems to be that if you are DMed, you get an email notification, so it can be useful if someone's been away from the board for a while.

  • Posts: 298

    @yojimbo2000 Thank you, I have sent a message to @juce

  • edited July 2015 Posts: 154

    @binaryblues, i believe this happens because of git's distinction between "author" and "committer". Perhaps, you have a newer git version that is more strict about this, than the one i have installed on my codea-scm server.

    In any case, i just pushed a change that should fix this. See here:

    https://bitbucket.org/juce/codea-scm/commits/bf9d7c448acc24098

    So, if you take the latest source (or just incorporate this change) - it should solve the problem. Let me know if it works for you.

    And i should add: thanks for finding this issue! :)

  • Posts: 298

    @juce It is OK now! I can use the local git repo! Thanks a lot!

    On my local host, two files need to be modified, Because the user nobody can not write to the repo, I use the user git nor nobody, so there is a directory for git : /home/git/

    1 Add a file ssh_config In /home/git/.ssh/, the content is:

        StrictHostKeyChecking no
    

    2 Modify /home/git/.gitconfig, append 2 lines:

    [receive]
        DenyCurrentBranch = ignore
    

    Now, we can use Codea-SCM on both local host repo and internet repo! Enjoy it!

Sign In or Register to comment.