Master Thesis Two Months In

I'm writing my master thesis about the topic of longterm preservation of digital cultural heritage. Specifically about the part where you want to store large amounts (in the petabytes) of data cost efficiently.

The time span we have to keep these files around is forever and not just 30-60 years that tax related data need to be available. The top direct response when talking about this is:

I just save it on [Tape, Paper, etc.]. This is so much saver than disks.

While this may be true, people forget or are not aware that the data not only must be stored but also must be accessible at the same time. People want to go into an archive (either physical or online) and dig through the data.
Suddenly [Tape, Paper, etc.] is not an option anymore because of access time.

Another problem is that you have to be absolutely sure that your data is still intact and not corrupted by silent bit flips. This means not only storing it but visiting [Tape, Paper, etc.] from time to time to make sure that the medium still works.

Traditionally this has been solved by huge raid systems which are unfortunately very expensive and in most cases push the responsibility to a single machine. If the machine is down, the data is either not accessible or might become corrupt.

My plan to tackle this tasks is to provide an architecture that uses many copies of a file to cross check itself, uses commodity hardware to run and consumer grade disks to store the data. Actually, lots of  hardware. The more the better.

By moving the responsibility of data availability from the raid controllers and a single machine onto a network of machines we get a system that is very robust against individual hardware errors. If one machine goes down, the rest of the system recognized this and heals itself by making sure that a predefined number of copies of the data is always in the system.

I spend a lot of time coding the last months and this weekend marks another major milestone for me.
I am eating my own dog food and running an archive at home.

Hardware wise it consist of:

4x 2TB external hard disks with a USB 2.0 Port
4x Shevaplug computers (1.2GHz ARM CPU, 512MB RAM, 512MB internal flash disk, 1 GBit Ethernet, 1xUSB 2.0 Port)

I had to optimize the software running specifically in such a "low" powered environment but I wanted to keep the entry level for hardware really to a minimum. The Sheevaplugs and the disks are around 70€ per item.
Currently the archive is running with 400.000 small files (4k). For now it is access by a REST-API

Here is a picture of my current setup.

Slicing and dicing the 4. Lichter Filmtage movie selection

It's this time of the year again and I'm very proud that we once again are able to show you the movie selection of this years 4. Lichter Filmfest selection in a beautiful design. The festival takes place in Frankfurt am Main, Germany from the Thursday the 24.3.2011 until Sunday the 27.3.2011.

Florian Kissling and myself got together and developed some new user interface features.

A common request was to be able to reduce the view to only one day. Last year we showed you every movie the festival was showing on one page.
The complete movie selection is still on one page but we added a filter option at the top of the page which enables you to easily focus on just one day. The movies from the other days will fade to the back in a beautiful animation.

But there is more. Last year you had to guess the type of movie from the title. Not bad but it could be done better.
For this year we have movie categories. You want to watch only shorts films, no problem now you can scan through the main page without going to the detail page to find the genre you want.

Sounds like a lot of scanning on your part? Right, thats what came to our minds too. But since we have beautifully filtering now, lets extend this paradigm to genres and it looks absolutely brilliant.

Filter down by genres and see exactly what you want to see on the main page. That means you only have to leave the main page if you want to watch a trailer (yes, we feature a trailer for every movie) and read more of the description.

You like to be surprised by the movie and think that trailers spoil the movie experience? Just stay on the main page which now features all the info you need, like the time, the location and the movie genre.

Hope we again made it easier for you to find the right movie for you more quickly. So go, find the movie for you and have fun on the festival!

http://www.wasgehtheuteab.de/de/frankfurt-am-main/lichter-filmtage-2011

And if your already on the go, the page also works great on the iPhone, the iPod and the iPad.

Longterm Digital Preservation

Makes preperation for my master thesis a bit more fun. Find more of them on YouTube.

wasgehtheuteab Büro - Tag der offenen Tür

Vom 19. bis 21. November 2010 veranstaltet »AtelierFrankfurt« zum dritten Mal in Kooperation mit den städtischen Ateliers der Stadt Frankfurt und mit der Basis ein Wochenende der offen Ateliers. Über 200 Künstlerateliers können an diesen beiden Tagen in der Stadt besucht werden.

Das Büro von wasgehtheuteab.de/partyhq.de befindet sich auch in den Räumen des AtelierFrankfurts und deswegen laden wir euch herzlich ein bei uns vorbeizuschauen, Hallo zu sagen, Kritik und Lob zu äussern oder einfach nur ein bisschen zu plaudern.

Die Öffnungszeiten sind:
Freitag, 19. November 2010: 20:00 – 22:00 Uhr
Samstag, 20. November 2010: 14:00 – 21:00 Uhr
Sonntag, 21. November 2010: 14:00 – 18:00 Uhr

Was geht heute ab?Büro@Atelier Frankfurt
Hohenstaufenstraße 13-25
3. Stock/Raum 3.06
60327 Frankfurt am Main
Google Maps

Nippon Connection 2010 Program

And we did it again. This time wasgehtheuteab.de features the well established film festival Nippon Connection. This year marks there 10th anniversary and we are very proud to show you this years selection of over 150 movies from Japan. The festival takes place in Frankfurt/Main from the 14.4.2010 - 18.4.2010.

Have fun and click the image for the movies!

New Feature: Music genres for your event

Let's say your out of your hometown, you want to go out but you never heard of the dj's before and do not know what kind of music they play. How do you decide where to go? 

At least for Frankfurt/Germany. We want to solve this problem with the latest feature addition to wasgehtheuteab.de, the little, clutter free eventcalender.

We added the posibility to attach music genres to your event. When you upload a flyer you'll see the new field music and the suggestions that we already have in the system.

You can either click the genres below the text field and we will fill it out for you automaticaly or in case where we are missing the music you want to play, you can simply enter your genre seperated by a comma and we will add it to our system. Easy as one, two, three!

We added the genres in two places while you browse for flyers. 

The first is in the lower left corner when you go with your mouse over a flyer to instantly see more details about the event. This way you can make your choice without clicking a link or open a new page.

The other place is inside an individual flyer right below the date and before the location address. Maybe you clicked a link on another page and got transferred here, you'll see right away if the music is yours or not.

Seeing the genres that are played at the location right away, will help you make an even better choice for your valuable time out in town and leaves more time to just have fun at the party!

Soon floating around town