Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Utility: Automatic backup and recovery of your project code

IgnatzIgnatz Mod
edited March 2013 in Code Sharing Posts: 5,396

I've created a backup and restore utility that does what I need, which

  • backs up my whole project automatically to Dropbox whenever I change my version number (usually when reaching a checkpoint)
  • allows me to restore backed up projects from Dropbox (including class tabs) automatically, without any copying and pasting
  • requires only one line of code to backup or restore (but you need to create a dependency to this utility, of course)

EDIT The latest code is always below, including full notes. Comments and suggestions welcome. Use with care as it's not exhaustively tested.

http://pastebin.com/VyxjeVmV

Tagged:
«1

Comments

  • Posts: 398

    Blimey - you're on fire Mr Ignatz! Another superb addition! Er.. Do you ever sleep? Lol

  • IgnatzIgnatz Mod
    Posts: 5,396

    Who can sleep when there's Codea to play with? :bz

  • Jmv38Jmv38 Mod
    edited March 2013 Posts: 3,295

    Hi @ignatz I just cant copy your code from safari... In Pastebin i touch download raw but then i cannot select the text. Any solution to do it from the ipad?
    [edit] ok i've done it via goodreader. Not that easy, though.

  • IgnatzIgnatz Mod
    Posts: 5,396

    What do you use for storing code, that is easy to copy from?

  • Jmv38Jmv38 Mod
    edited March 2013 Posts: 3,295

    That is strange: i think i could copy your last posts with no problem. Don't know why i cant select this one. About my storage: i simply upload the txt file to my website. When you click on the link, the text is displayed in safari and seem easy to copy (at leat i think so).

  • That's great. Thank you very much.

  • Posts: 202

    On split


    function split(str, delim, maxNb) if maxNB==nil then maxNB=9999999 end

    The first line has no effect inside the function. Even worse, it has an unadverted global effect. Did you write it? It's not in the referenced link. You didn't read the rest of the function, right?

    Also, the function has two warts you don't need. One is the aforementioned maxNb that you make no use of, the second one is the "bad case", which is at the most a special case, but upon examining it more closely it is not even that.

    Let me present you a more concise split function:


    function split(str, delim) local result = {} local pat = "(.-)" .. delim .. "()" local lastPos = 1 for part, pos in string.gfind(str, pat) do table.insert(result, part) lastPos = pos end -- Handle the last field table.insert(result, string.sub(str, lastPos)) return result end

    Why don't you need maxNb? Either you get the full result or it will be of no use for you.

    Why don't you need the special case? Firstly, let me tell you the the special case is still handled, just not as the very special case that was mention in the original code. Secondly, I assume you will always have at least "end" in a tab.

    On your file format encoding

    At this point I'll just accept the your format looks like this:

    • original contents
    • 10 hash signs that act as some sort of your "format specifier"
    • as much padding with character "0" as you need to get a length that is a multiple of 3

    Your encoding, taken from Backup:Store:


    txt=origTxt..string.rep("#",10) local n=3-string.len(txt)%3 if n>0 then txt=txt..string.sub("0000",1,n) end n=string.len(txt) local s=math.floor((n/3)^.5)+1 img=image(s,s)

    The extra padding with "0"s is awkward. Better you remember the length of the text after adding your format specifier, then add "enough" padding so that the string reaches at least the next 3-byte-boundary.


    -- add format specifier txt=origTxt..string.rep("#",10) -- |n| is the length of the txt to encode local n=string.len(txt) -- padding so that accessing "a few more chars" results in a legal char, not nil txt=txt.."0000" local s=math.floor((n/3)^.5)+1 img=image(s,s)

    The key is to remember the length of the string you want to encode, then the length of the padding only matters so much that it shall not be too short. You can get rid of the padding at all if you modify the string access in the loop:


    -- original, requires padding of |txt| for j=1,3 do e[j]=string.byte(string.sub(txt,i+j,i+j)) end -- without padding of |txt| for j=1,3 do -- pad with "0" if string.sub reads beyond the end of |txt| e[j] = string.byte(string.sub(txt,i+j,i+j)) or "0" end

    On your file format

    Essentially, you have no file format. Instead you split the code into tabs by scanning it for class definitions. That's not very elegant but I aknowledge the effort to try to implement this system. I suggest to use something the the Codea project format for future file formats. Perhaps keep your class splitter so you can rearrange foreign code to your liking.

    Expanding your file format anyway

    Just an idea. You could use the string space after "##########" to store some mata data, e.g. the names and byte positions of the tabs.

    "##########Main,1,125;Other,126,659;Extra,660,1011##########"

    By scanning the meta data you could recreate the tabs' contents.

    Finally

    A great and useful project despite all my ramblings above. Especially the automatic backup when you change the version string is feature to steal.

  • IgnatzIgnatz Mod
    edited March 2013 Posts: 5,396

    Thanks for the comments. No, I didn't write the split function, and I'll try yours instead.

    The hash signs at the end are there so i know where the end is, although I guess if I'm padding with zeros, then any zeros at the end have to be padding and not part of the text, so I don't need the hashes.

    File format - I was being dumb. I can include tab delimiters when I back the code up.

    I'll fix and repost. Thanks for your efforts to improve it, much appreciated.

  • Posts: 502

    unbeatable util! will use it for all my projects!

  • Posts: 202

    It worked, so it wasn't too dumb.

    Before you rush to shake your code up, here some additional suggestions.

    File format

    You have a very special problem with a file in an image. Firstly, it looks like an image. Secondly, the contained data is probably smaller than the image itself because the image is constrained to fill its rectangular region.

    So my suggestion is to use something like a proper file header, e.g. use the first 3 bytes (I've chosen 3 so they make up one pixel (without alpha)) to store a magic number like "Ig1", means "Ignatz Image File Format 1" (no kidding, really). The next 3 bytes tell you how large the actual content is. No padding needed, you know how many bytes to operate on.

    Payload

    You could try to use the Codea project format, this could be an appropriate generator:


    -- Combine the contents of all tabs into one string formatted in the -- Codea Project Format. function Backup:GenerateCodeaProject() local txt = "" local tabs = listProjectTabs() for i,t in pairs(tabs) do txt = txt.."\n--# "..t.."\n" txt = txt..readProjectTab(t) end return txt end

    I think Simeon would be pleased to see this simple format live on. The only disadvantage of this format is if somebody has the habit to start comments with "--#".

    More file formats

    If you're not pleased with it, invent "Ig2", the "Ignatz Image File Format 2". Perhaps this format starts with some meta data to describe the rest. "Main,1,100;Notmain,101,345;;". Indexing starts immediately after the meta data. The end of the meta data is marked by two semicolons in a row.


    These are just suggestions, but you see that it is easy to invent a format for text in an image that is more robust or more practical to implement.

    Recapitulating the main objectives:

    a) Storage of text (or any arbitrary data) in an image.

    b) Combining and splitting tabs.

    c) a) and b) shall be precise. You get exactly that thing out of them that you put it. No more, no less.

  • IgnatzIgnatz Mod
    Posts: 5,396

    @Codeslinger - Good suggestions, thank you. To be honest, I was experimenting to see what would work, and I haven't really sat down and planned it properly, like I should have done.

    If you'd like to collaborate with me, that would be great, because I don't see it as "mine", but as a community utility, and if other people are going to rely on it, then it had better be rock solid

    I like the idea of a version* byte, and storing size, too, to get rid of messing about at the end of the text. I flirted briefly with doing some compression, but there isn't much need because code is so small it doesn't take much space, and I want to keep the backup code small, given it gets loaded together with user code. Because code is small, I'm also not worried about the image format wasting part of the final column.
    *(I come from a business spreadsheet background, and I'm always on at people to include version numbers, so I should have thought of this myself!).

    The tab delimiter is interesting. If pasting the code into Main automatically prompted Codea to split it, then obviously the Codea format would be best, but that doesn't seem to work (in fact, no matter what I do, I've not been able to get Codea to put anything back into tabs from a paste). So I'm assuming we need a tab delimiter that is definitely unique - which the Codea format is not.

    In fact, because Codea doesn't use chars above ASCII 127 - except for 194 and 226 for currency symbols - we could use a single char of (say) 255 as a delimiter. (I initially thought that wouldn't work because colour values can take any value to 255, but colour digits are encoded separately so that's not a problem). However, in case Codea does start using that range in future, maybe it's best to find a chat thar will definitely never be used, such as char 3, which is labelled as "end of text" in the ASCII table (very appropriate) and which will definitely never appear in any user code.

    If that's ok with you, I'll clean up the existing code, which I've started doing (and it's much better already), and post a link soon.

  • IgnatzIgnatz Mod
    Posts: 5,396

    @Codeslinger if you have time, look at this new version

    http://pastebin.com/embed_js.php?i=gc0uGGvb

    It has a file format, a file header, and is simpler than before ,which is good. I've also included a test tab that checks that key components work when you run Backup itself. Before doing so, though, put in the name of one of your projects for two of the tests (it reads from them - but does not write to the!).

    More details on the Notes page in there. I haven't tested exhaustively, that's next.

  • Posts: 202

    I've written the stuff below before you came with your new version. I'll post it anyway and will look into your code later (maybe not even today, sorry).


    In fact the editor's contents are UTF-8 encoded, it just happens to look like ASCII if you don't make use of any special characters. We don't have to deal with a BOM so it's save to use codes like 254 and 255 as tab separators. You may even use 0 to mark the end of all data, so there would be no need to store the length of the data at the beginning. If you make the "Ignatz Image File Format 1" for text only you might even make good use of ASCII control codes.

    Retro corner

    Let's use of ASCII control for the fun of it. Their meaning? We decide upon their meaning, we just look up their names.

    http://en.wikipedia.org/wiki/ASCII#ASCII_control_code_chart

    Perhaps with the codes SOH (1), STX (2) and ETX (3):

    "Mainfunction setup() end function draw() print("Hello") end"

    Where SOH Starts the tab name until STX, content goes until ETX. Then you either have a new SOH for the next tab or NUL for the end of the file.

    You can still add a total length a the beginning of the image for double checking.

    Code ideas:

    (I changed "Ig1" to "Ign1", 4 byte magic numbers are more common. I've initially chosen 3 because they did easily fit into your encoding triple. However, we don't have severe alignment problems to solve.)


    -- Private: Encode all tabs into Ign1 format, including header. -- -- Returns Ign1 encoded string. function Backup:EncodeTabsToIgn1() local SOH, STX, ETX = "\001", "\002", "\003" local encoded = "Ign1" local tabs = listProjectTabs() for i,t in pairs(tabs) do encoded = encoded..SOH..t..STX..readProjectTab(t)..ETX end encoded = encoded.."\000" return encoded end -- Private: Decode one tab of an Ign1 encoded string. -- -- encoded - Ign1 encoded string. -- start - start index inside |encoded|. -- -- Returns tab name, tab contents, next index on success. -- Returns nil on failure. function Backup:DecodeIgn1Tab(encoded, start) local SOH, STX, ETX = "\001", "\002", "\003" local SOHidx = string.find(encoded, SOH, start) local STXidx = string.find(encoded, STX, start) local ETXidx = string.find(encoded, ETX, start) -- assert some basic format properties if not SOHidx or not STXidx or not ETXidx then return nil end if SOHidx ~= start or ETXidx < STXidx then return nil end -- decode local tabname = string.sub(encoded, SOHidx + 1, STXidx - 1) local contents = string.sub(encoded, STXidx + 1, ETXidx - 1) local nextidx = ETXidx + 1 return tabname, contents, nextidx end -- Private: Decode all tabs of an Ign1 encoded string and write them -- into the project. -- -- encoded - Ign1 encoded string, including "Ign1" header. -- -- Returns nil on success. -- Returns error message on failure. -- Tabs may have been saved until that point. function Backup:DecodeAndSaveFromIgn1(encoded) if string.sub(encoded, 1, 4) ~= "Ign1" then return "Not in Ign1 format" end local tabidx = 5 local errmsg = nil repeat -- check for end of data if string.sub(encoded, tabidx, tabidx) == "\000" then tabidx = nil else local tabname, contents tabname, contents, tabidx = self:DecodeIgn1Tab(encoded, tabidx) if tabname ~= nil then saveProjectTab(tabname, contents) else errmsg = "Ign1 decoding error" end end until tabidx == nil return errmsg end

    The image de/serializing and high level backup logic can be taken from your original code.

  • Posts: 202

    Some loose comment on your new version:


    -- In function Backup:init if t~=nil and remind~=false then print("Last backup was",string.format("%.0f",os.difftime(os.time(),t)/60),"minutes ago") else . . . end

    You surely meant to write:


    if t~=nil then if remind~=false then print("Last backup was",string.format("%.0f",os.difftime(os.time(),t)/60),"minutes ago") end else . . . end
  • IgnatzIgnatz Mod
    Posts: 5,396

    Thanks for your latest efforts. We must think alike because I also considered using the ASCII ETX char, but I saw a difficulty because it couldn't be included in a Codea string. However, it could of course just be encoded directly into an image rgb property without ever going into a string, so I'll be interested to see what you've done.

    I very happy for you to improve the existing code, but I think we must try to not both work on the same thing at the same time, to avoid wasting effort. So please let me know if you plan to rewrite any part of it, and I will avoid working on that part.

    What I will do today is look at what you've done, and if it's better than my code, I'll put it in, otherwise I'll come back and discuss.

    Thanks again, much appreciated =D>

  • IgnatzIgnatz Mod
    Posts: 5,396

    @Codeslinger. I've had a look at your code now, and it's great. I didn't realise you could put non printing chars into Codea strings.

    I only have one minor suggestion. Instead of coding "ign1” into every function name and having to do checks that the encoding complies, why not implement different encoding/decoding versions as completely separate classes, eg Backup_1, Backup_2, etc (I prefer to use a number rather than a variant of my name, because as I said before, I don't see this as "mine"), and reducing the current Backup and Restore classes to a simple check of which version applies, then they call the right version class to handle it. Then within each class, you can always assume you are in the right place, and use identical function names across all the class versions, for consistency. It also means that if future versions get messy, their code won't interfere with code for earlier versions.

    I'd be happy if you want to fix up my code, and it is probably better if you do it, because you will probably improve lots of other things. However, I am also happy to do it myself if you prefer. Let me know what suits you. Either way I will share credit for this with you.

  • Posts: 140

    Just wanted to say this is one of the best utilities on this forum. I frequently screw up my code with fresh ideas that crash. Often, I can't remember my errant changes. So this little bit of code is just what I needed. Thanks @Ignatz and @Codeslinger. It would be great if it were built into Codea. For instance when you build a new project, the setup function would automatically add a b=Backup("DropBox:YourProjectName_vers_1) line to the project--of course the YourProjectName would be whatever you typed in when you created the new project.

  • IgnatzIgnatz Mod
    edited March 2013 Posts: 5,396

    @Ric-Esrey thank you!

  • IgnatzIgnatz Mod
    Posts: 5,396

    @Codeslinger

    New version here: http://pastebin.com/embed_js.php?i=hvJ4rRdt

    It incorporates your code, with the following minor modifications

    • I have created version specific tabs for backup and restore. The main Backup and Restore functions test for the version and call the version-specific backup/restore tab. This makes it cleaner if future versions need to work quite differently
    • I have shortened version number to 3 bytes so it can easily be read off from one pixel when restoring and checking version number (as in previous point)

    I have tested it on a couple of projects but I thought I'd share with you in case it needs more work. But the code is much neater and cleaner now, thank you!

  • Hello Ignatz the idea of this utility is great, making a backup works fine, but I can not restore a backup, there is a problem with the local img variable (indexing a nil value). I checked the code I copied and i cant find any mistakes. Can you help me to solve this. I am new at Codea just started a few days ago.

  • IgnatzIgnatz Mod
    Posts: 5,396

    @Dreamdancer

    Try the version immediately above your post. If that fails, please send me your code and I'll try to find the problem. (And welcome, I've only been here a month, and still loving it).

  • @ignatz
    I reinstall the latest version 1.1 but the result is the same I get the following error:

    error: [string "Restore = class()..."]:4: attempt to index local 'img' (a nil value)

    I think it is this line of code : local r,g,b,a=img:get(1,1) at the Restore tab

  • IgnatzIgnatz Mod
    Posts: 5,396

    How are you calling the Restore function? Please post the code.

  • edited March 2013 Posts: 28

    -- testbackup

    -- Use this function to perform your initial setup
    function setup()

    img=readImage("Dropbox:Zeeslag Ver 1.03")

    r=Restore(img)

    end

  • IgnatzIgnatz Mod
    Posts: 5,396

    It sounds like Codea doesn't think it's a proper image in the right format, because all the offending line does is to read the first pixel. That should work for any valid image. If you look at the image file it should be a jumbled mix of pixels of different colours. I'll have a play with it in the morning (it's late here).

  • Oke its here 18.00
    If i look at the back up i see a mix of different colours that looks oke
    I will check the backup part again maybe something wrong there after copiing
    Thanks

  • IgnatzIgnatz Mod
    Posts: 5,396

    If it still doesn't work, perhaps you can put your code where I can find it, and then I can solve it perhaps

  • I think i found the problem the command readimage did not work with the created backup image
    May be the fault is in the creation of the backup image?
    It is strange the program verifys the backupimage with the same restorefunctions

  • IgnatzIgnatz Mod
    Posts: 5,396

    It does use the same functions, but there is one difference. The verify test uses the image in memory, not the one just saved, so there may be a problem in the saving process. I'll look at that. Thanks for the diagnosis.

  • Posts: 2,820

    This is amazingly useful... You have no idea. An it's on the cloud! Your a genius.

  • @Ignatz
    I found the ERROR:-)
    @Simeon
    It is possible in Codea to save an image like this: saveImage("Dropbox:Myproject Ver 1.1",img)
    It is NOT possible to get the image with readImage("Dropbox:Myproject Ver 1.1")
    It is not allowed to use a DOT in the imagename in the function readImage.
    If you use names without DOTS then you have no problem.
    Is this a bug?

  • IgnatzIgnatz Mod
    Posts: 5,396

    Thank you very much for that, I'll include a note in the code

  • Posts: 277

    Ooooh, does this ever look useful...

    Although, am I the only one who gets (after having created a "Backup" project and having pasted the whole code inside it as main) a "0 available tabs" and, once checked, "0 imported tabs" when setting the dependencies in the project I want to backup?

    (As such:
    https://www.dropbox.com/s/239dkla11d5y5hm/2013-07-20 12.02.40-1.jpg )

  • IgnatzIgnatz Mod
    Posts: 5,396

    @Rudolphe - instead of pasting all the code as Main, hold down Add New Project until you get the option Paste into New Project. That will paste into all the tabs.

    Let me know if you have a problem.

  • Posts: 277

    Thanks! I didn't know this, it imports great, and it says my backup is verified! :D

    I'm trying to see the backed up file in my dropbox folder, but can't see it though, and dropbox doesn't look like it's syncing anything...

  • IgnatzIgnatz Mod
    Posts: 5,396

    It should look like a gray picture

  • Posts: 277

    Hmmm, can't see any of that in my Dropbox. It seems odd to me since I get the "backup made" and "backup verified" message when running my project with a new version number. Should it be in a particular folder?

  • IgnatzIgnatz Mod
    Posts: 5,396

    You aren't using a full stop in the name, I hope.

    Codea doesn't like that.

  • Posts: 277

    Nope, here's what I have:
    https://www.dropbox.com/s/5lgezh0sm62t4k4/2013-07-22 14.02.46.png

    I also tried without going through a variable holding the text, with the same result

  • Posts: 2,042

    @Ignatz, this util (SUPER helpful by the way) was working great but this mroning I tried to restore one of my backups and I got "ERROR unknown version 47", do you have any idea what's going on?

  • IgnatzIgnatz Mod
    Posts: 5,396

    The backup software changed at one stage. Is this an old backup?

  • IgnatzIgnatz Mod
    Posts: 5,396

    @Rodolphe - that name works for me, I would have to see your actual code to test it further.

    One thing that sometimes catches me is that Codea sorts images by capital letters, then by lower case, so maybe your backup is down the bottom of the list

  • Posts: 2,042

    @ignatz, no its a backup from earlier today, I just wanted to transfer between devices

  • IgnatzIgnatz Mod
    Posts: 5,396

    @JakAttak- I'll have a look see

  • Posts: 277

    Oh thank you @Ignatz, the grey files were indeed there, I must have overlooked where the code created the images (application/codea/). Everything is there and looking good. Cheers!

  • Posts: 2

    I have used Backup for a short while and it works beautifully for me. Now, I wanted to share projects with my grandson. So, I connected his iPad II to my Dropbox account and downloaded your Backup project (it looks fine). Created a new Test project, created a dependency with the Backup project and added the 2 lines:

    img=readImage("Dropbox:AAA")
    r=Restore(img)

    Now, when touching AAA, Dropbox appears in the drop down list but I do NOT se any of my saved files (images).

    Is there anything more that needs to be setup for it to work?

  • IgnatzIgnatz Mod
    Posts: 5,396

    I'll have a look, Staffan

  • Posts: 2

    For some reason, the problem vanished by setting up the Dropbox accounts properly :\">
    Sorry.

  • @Rudolphe - instead of pasting all the code as Main, hold down Add New Project until you get the option Paste into New Project. That will paste into all the tabs.

    Let me know if you have a problem.

    -- This solved it for me. Thanks.

    Great Project!

  • IgnatzIgnatz Mod
    Posts: 5,396

    @Chaverbe - if you look more closely, you'll find Rudolphe's message was posted 6 months ago
    :\">

    Never mind, at least you're trying to help!

Sign In or Register to comment.