Wikipedia gearing up

Wikipedia is gearing up for an explosion in digital content with new servers and storage designed to handle larger photo and video uploads.

Video file sizes are quickly reaching the dozens and hundreds of megabytes, and the proliferation of high-megapixel cameras means even small photos can take up a few megabytes, says Brion Vibber, CTO at the Wikimedia Foundation, which operates Wikipedia. Until early 2008, the user-generated encyclopedia's primary media file server had just 2TB of total space, Vibber says.

“For a long time, we just did not have the capacity [to handle very large media files],” he says.

Wikipedia has since scaled up from 2TB to 24TB and now 48TB of storage for its primary medial file server, and recently raised file upload limits from 20MB to 100MB. The amount of storage actually being used is about 5TB but that will grow quickly, Vibber says.

For example, users are uploading public domain classical music, and some recordings can last a half-hour. Documentary films that are out of copyright are also being uploaded, and some users are struggling to keep files under the 100MB limit, according to Vibber.

Vibber's long-term goal is to let users upload feature-length, high-quality videos, but in addition to capacity limits he says there are challenges related to getting files in the appropriate format and the physical movement of large files.

“There's no limit, and there's no practical limit,” Vibber says. “The limits will get bigger and bigger to where it will be relatively easy for someone who has a legitimate need to upload a two-hour video of good quality.”

Wikipedia's new servers and storage were supplied by Sun, which provided donations and special pricing on Sun Fire x4500 and x4150 servers and StorageTek arrays. (Compare storage products.) Wikipedia, which handles more than 10 billion page views a month, also uses Sun's open source MySQL as its primary database software.

Wikimedia operates a primary data center in Tampa, Fla., and caching centers in Amsterdam and South Korea, and has a total of 450 or so commodity servers from a mix of vendors, Vibber says.

The site's technology goals this year and next include improving usability, making it easier to edit articles and upload videos and photos, and integration with media file sites like Flickr, he says.