Sunday, December 30, 2012

A shell recipe for backups with logs and history

I wrote a shell script for a cron job that grabs backups of some remote files. It has a few nice features:

  • Output from the backup commands is logged, with timestamps.
  • cron will send me email if one of the commands fails.
  • The history of each backup is saved in Git. Nothing sucks more than corrupting an important file and then syncing that corruption to your one and only backup.

Here's how it works.

#!/bin/bash -e

cd /home/keegan/backups
log="$(pwd)"/log

exec 3>&2 > >(ts >> "$log") 2>&1

You may have seen exec used to tail-call a command, but here we use it differently. When no command is given, exec applies file redirections to the current shell process.

We apply timestamps by redirecting output through ts (from moreutils), and append that to the log file. I would write exec | ts >> $log, except that pipe syntax is not supported with exec.

Instead we use process substitution. >(cmd) expands to the name of a file, whose contents will be sent to the specified command. This file name is a fine target for normal file output redirection with >. (It might name a temporary file created by the shell, or a special file under /dev/fd/.)

We also redirect standard error to the same place with 2>&1. But first we open the original standard error as file descriptor 3, using 3>&2.

function handle_error {
    echo 'Error occurred while running backup' >&3
    tail "$log" >&3
    exit 1
}
trap handle_error ERR

Since we specified bash -e in the first line of the script, Bash will exit as soon as any command fails. We use trap to register a function that gets called if this happens. The function writes some of the log file to the script's original standard output. cron will capture that and send mail to the system administrator.

Now we come to the actual backup commands.

cd foo
git pull

cd ../bar
rsync -v otherhost:bar/baz .
git commit --allow-empty -a -m '[AUTO] backup'
git repack -da

foo is a backup of a Git repo, so we just update a clone of that repo. If you want to be absolutely sure to preserve all commits, you can configure the backup repo to disable automatic garbage collection and keep infinite reflog.

bar is a local-only Git repo storing history of a file synced from another machine. Semantically, Git stores each version of a file as a separate blob object. If the files you're backing up are reasonably large, this can waste a lot of space quickly. But Git supports "packed" storage, where the objects in a repo are compressed together. By repacking the repo after every commit, we can save a ton of space.

62 comments:

  1. #!/bin/bash -e
    already buggy; use "#!/bin/sh" and "set -e".
    If you must use bash prefer "#!/usr/bin/env bash"

    function handle_error {
    buggy. The correct syntax is
    handle_error() {
    "function" is not a POSIX keyword

    "By repacking the repo after every commit, we can save a ton of space."
    Also buggy and slow. Git should do this automatically if required.

    ReplyDelete
    Replies
    1. POSIX says that a script starting with "#!" has implementation defined behavior. POSIX does not provide a way to force a script to be run with a POSIX shell. "#!/bin/sh" is likely to work, but doesn't on some systems. "set -e" is required to work by POSIX; so I agree that it should be preferred over using -e in the she-bang line.

      The env trick fails on esoteric systems. A different set than the ones that fail on "#!/bin/bash -e" sure, but it's roughly the same number of systems in the wild. It's a wash and "#!/bin/bash" is shorter, so whatever.

      Agreed on POSIX shell function syntax. But, the she-bang did request bash and bash takes the syntax in the script. I think it's a wash, but it would be nice if these non-bash-specific techniques used non-bash-specific syntax.

      Yeah, repacking the repo after every commit is unlikely to save much space at all. Git automatically repacks as needed during normal operations. If you do massive history re-writing with filter-branch or something like that, you might want to repack, but only if you are very sure of the results and have properly removed the backup refs and the pruned old refs from the reflogs. The repack in the article is more likely to waste resources than to save them.

      Delete
  2. I think that `git commit [..] -a [..]' is not enough and you should add `git add .'? Otherwise, I'm not sure if I understand the relation between "foo", "bar" and "baz".

    ReplyDelete
  3. Great blog! Do you have any tips for aspirring writers?
    I'm hoping to start my own site soon but I'm a little
    lost on everything. Would you propose starting with a frtee
    platform like Wordpress or go for a paid option? Thee are so many choicxes out therfe that I'm totally overwhelmed
    .. Any tips? Bless you!

    ReplyDelete
  4. I really like your blog.. very nice collrs & theme.
    Did yoou design this website yourself or didd you hife someone tto do it ffor you?
    Plz reply as I'm looking to construct my oown blog and would like tto find out where
    u got this from.kudos

    ReplyDelete
  5. RedBox TV is one of the most used television channel streaming app, you can download RedBox TV APK latest version and enjoy watching shows.
    https://apkmist.com/redbox-tv-apk/
    redbox tv
    redbox tv apk
    redbox tv app

    ReplyDelete
  6. a great article, With lots of information in it, These articles help users interested in the site, and continue to share more ... good luck!

    ReplyDelete
  7. مهمترین مرحله از طراحی اپ اندروید و iOS، چگونگی انتشار آنها در پلتفرم های معتبری مانند App Store و play Store می باشد.

    ReplyDelete
  8. Thanks for sharing this information. I really like your post very much. You have really shared an informative and interesting post with people DQFanSurvey

    ReplyDelete
  9. If you found any common error like installation issue, performance, Printing issue, Network connectivity issues, you can simply download Quickbooks Tool Hub. It's a collection of different Quickbooks tools in a single application.
    Quickbooks Repair Tool

    ReplyDelete
  10. WBCHSE which Conducts the Examination of Higher Secondary level or Pre-University level Examination in Month of March. Supplementary Examination Conducts in month of Jun. WBCHSE HS Model Paper 2021 WBCHSE Annually and Private and Regular Students Participate in HS Examination every year more than laks of students. WB HS Exam are conducted by West Bengal Council of Higher Secondary Education Only.

    ReplyDelete
  11. Appreciating Information. Thanks for sharing with us. Keep sharing again.
    AOL is a global mail service provider that offers a variety of features and advanced technology. If you are facing an issue while sending or receiving emails, documents, or attachments, and much more, then you need to AOL Mail Server Settings Android properly. To fix this issue, you can take hassle-free advice to AOL Email Supports Experts and resolve your issues.

    ReplyDelete
  12. A very awesome blog post. We are really grateful for your blog post. You will find a lot of approaches after visiting your post.
    Hi! We are water treatment company in uae Great points made up above! And
    Package Sewage Treatment Plants dubai thanks…
    I think this is one of the most important information for me. And i am glad reading your article. But should remark on few general things…

    ReplyDelete
  13. Way cool, some valid points! I appreciate you making this article available, the rest of the site is also high quality. Have a fun. Feel free to visit my website;
    야설


    ReplyDelete
  14. Very good information. Lucky me I recently found your website by accident
    I have book marked it for later! Feel free to visit my website; 일본야동


    ReplyDelete
  15. gclub-winner เกาะติดทันกระเเสเกมส์สล็อตออนไลน์ เว็บไซต์ที่ครบวงจรที่สุด เเนะนำเกมออนไลน์เล่นง่ายได้เงิน เกมเดิมพันออนไลน์อัพเดทใหม่ สล็อตออนไลน์น่าเล่น 2022 เกมใหม่ เกมฮิต สูตรสเเกนสล็อตที่เเม่นยำมากถึง 99.99% เเนะนำทริคในการทำเงินต่างๆ สำรหรับใครที่ชื่นชอบเกมเดิมพันออนไลน์ไม่ควรพลาด !!

    ReplyDelete
  16. I am personally using Whatsapp Plus for a long time and it’s beneficial for me. I use this app to hide blue ticks, use multiple WhatsApp accounts, delete the sent message.

    ReplyDelete
  17. Modo Bass VST Crack offers a professional and easy programming tool for musicians and DJs. It aids users in the production of sound and correction. It is the most renowned and renowned programming on the propelled of sound. Further, Modo Bass is the top physical instrument, which is also known as an instrument virtual.

    Modo Bass VST Crack
    EndNote Crack
    TubeMate Downloader Crack
    Sony Vegas Pro Crack

    ReplyDelete
  18. Thank you for sharing your wonderful ideas, Admin. To get the official crack, go to this link.
    Bat Professional Crack
    startcrack.co
    crackpropc.com
    crackersoftwares.com

    ReplyDelete
  19. In any case I’ll be subscribing to your feed and I hope you write again very soon! 바카라사이트

    ReplyDelete
  20. 스포츠토토 Fantastic beat ! I wish to apprentice at the same time as you amend your website, how can i subscribe for a blog site? The account helped me a appropriate deal. I had been tiny bit familiar of this your broadcast provided shiny clear concept

    ReplyDelete
  21. 바카라사이트 very cool post, i actually love this web site, carry on it

    ReplyDelete
  22. Aw, this was a very nice post. Taking a few minutes and actual effort to make
    a very good article… but what can I say… I put things off a whole lot and don’t
    manage to get nearly anything done. 스포츠중계

    ReplyDelete
  23. 토토사이트 The ability to work from anywhere with an internet connection provides me with greater flexibility while traveling, whether it be to visit family, travel to watch my favorite sports on TV. I’m looking for this kind of flexibilty.

    ReplyDelete
  24. I really enjoyed reading your programming blogs! Your writing style is engaging and informative. You have a great way of presenting complex ideas in an easy-to-understand manner. Your passion for the topics you write about shines through in your writing. Keep up the excellent work!

    University Homework Help is a leading online tutoring company that has been providing top-quality academic assistance to students for years. Their team of experienced educators specializes in providing online homework solutions, including Python programming homework solution. Their 24/7 availability ensures that students can get help at any time, no matter how complex the problem.

    They offer a flexible and personalized approach to learning, allowing students to work at their own pace and receive individualized attention. Their commitment to excellence and customer satisfaction is reflected in their high success rate and positive reviews. Whether you are struggling with programming concepts or need help with a specific assignment, University Homework Help can provide the support you need to excel in your academic pursuits.

    ReplyDelete

  25. This is an awesome post. Really very informative and creative contents.

    ReplyDelete

  26. Thanks for sharing valuable information.

    ReplyDelete
  27. ฟรีเครดิต กดรับเองหน้าเว็บไซต์ปัจจุบัน pg slot รับไปเลยเครดิตให้เล่นฟรี 100 บาท ให้ท่านสามารถทดสอบรับเครดิตฟรีกดรับเอง ก็ได้ของ PG SLOT เว็บไซต์ตรงของเรา pg-slot.game

    ReplyDelete
  28. การใช้ pgro หากพูดถึงปัญหาด้านการปลูกพืช PGro นั้น จะเกี่ยวข้องกับการจัดการที่ถูกต้องตามวิธีการและการดูแลรักษาที่เหมาะสม ดังนั้น พวกเราควรทำอย่างไรในการเล่น PG SLOT

    ReplyDelete
  29. slot ยอดเยี่ยมค่ายเกมออนไลน์ สล็อตบนโทรศัพท์เคลื่อนที่slotแล้วก็น่าเล่นเยอะที่สุด pg slot สมัครเล่น พีจี รับโบนัสแรกเข้า 100% แจกเครดิตฟรีหรือเลือกไม่รับโบนัสเล่นง่ายเล่นสนุกสนาน

    ReplyDelete
  30. Nemoslot คืออะไร เป็นเรื่องที่สนุกและน่าตื่นเต้นมาก เว็บตรงซึ่งเป็นเกมสล็อตออนไลน์ที่มีความนิยมอย่างมากในขณะนี้ หากคุณต้องการเข้าร่วมการเล่น PG SLOT

    ReplyDelete
  31. ทดลอง เล่น เกม สล็อต เว็บไซต์สล็อตที่อัพเดทเกมสดใหม่มาแรงทุกวี่ทุกวัน ทดสอบเล่นสล็อตฟรีทุกค่าย ทดสอบเล่นสล็อต pg slot ซื้อฟรีสปินสล็อตได้ทุกเกม มีเกมให้เล่นกว่า 1,000 เกม 

    ReplyDelete
  32. Hengjing168 เป็นเว็บไซต์ที่มีการให้บริการเกี่ยวกับการเดิมพันออนไลน์ โดยเน้นเป้าหมายในกลุ่มผู้เล่นที่ชื่นชอบการเล่นเกม PG SLOTออนไลน์

    ReplyDelete

  33. Please stay us up to date like this. Thanks for sharing.

    ReplyDelete
  34. Very informative blog, This blog is very easy to understand when compared to other blogs.

    ReplyDelete
  35. You should keep it up forever! Best of luck.

    ReplyDelete
  36. I really appreciate this post. I have been looking all over for this!

    ReplyDelete
  37. Thank goodness I found it on Bing.

    ReplyDelete
  38. You have made my day! Thanks again!

    ReplyDelete
  39. I appreciate you penning this article plus the rest of the site is also really good.

    ReplyDelete
  40. Very nice post. I definitely appreciate this website. Stick with it!

    ReplyDelete
  41. I absolutely believe that this web site needs a lot more attention.

    ReplyDelete
  42. I’ll probably be back again to read through more, thanks for the advice!

    ReplyDelete
  43. I like reading through a post that can make people think.

    ReplyDelete
  44. Also, thanks for permitting me to comment!

    ReplyDelete
  45. Thank you for sharing your info. I really appreciate your efforts and I will be waiting for your next post thank you once again.

    ReplyDelete
  46. Transform your online presence with top-notch Delhi Web Design services. From tailored layouts to seamless functionality, our Delhi web design experts will ensure your website exceeds all expectations. Get in touch today for a cutting-edge digital experience!

    ReplyDelete
  47. เกมออนไลน์ที่มีให้เลือกหลากหลายรูปแบบไม่ว่าจะเป็นรูปแบบไหนคุณสามารถเลือกได้ตามใจชอบพีจีสล็อตไทยเล่นผ่านหน้าเว็บของเราล็อกอินผ่านมือถือแล้วเล่นได้ทันทีไม่ว่าจะเป็น IOS หรือ Android เรารองรับหมดพร้อมให้บริการ 24 ชม.

    ReplyDelete
  48. Please keep us up to data like this. Thanks for sharing and Keep on writing

    ReplyDelete
  49. Everyone loves it Great website, continue the good work! Thanks

    ReplyDelete
  50. I'll bookmark it and come back to read more of your useful info. Thanks for the post.

    ReplyDelete
  51. I’m glad to read this article. The website style is perfect, I'll support your articles

    ReplyDelete