Category: Worth Reading

[See remaining text…] 

One of our primary services,
CVS, suffered a series of interrelated, critical hardware failures
in recent weeks. We understand how frustrating this CVS outage
must be to you and your users; however, our top priority remains
preservation of the integrity of your data.

The series of CVS hardware failures prompted us to expedite the
deployment of planed improvements to our CVS infrastructure,
drawing upon much of the knowledge that we gained from our
Subversion deployment. Our improved CVS service architecture,
which we plan to deploy tomorrow afternoon (2006-05-12), will
offer greater performance and stability and will eliminate several
single points of failure.

The Site Status page (https://www.sf.net/docs/A04) will be
updated as soon as the new infrastructure is rolled out. In the
interim, please read the important information provided below
to learn about how these changes will affect your project.

Summary of changes, effective 2006-05-12:

1. Hostname for CVS service

Old: cvs.sourceforge.net

New: PROJECT_UNIX_NAME.cvs.sourceforge.net

This change will require new working copies to be checked out of all
repositories (so control files in the working copy will point to the
right place). We will be updating the instructions we supply, but
instructions that your team has written within documentation, etc. will
need to be updated.

cvs -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/gaim co gaim

would be changed to

cvs -d:pserver:anonymous@gaim.cvs.sourceforge.net:/cvsroot/gaim co gaim

2. ViewCVS

We are moving from ViewCVS to its successor, ViewVC. ViewVC is
currently in use for our Subversion service.

3. Sync delay

Old: CVS pserver, tarballs and ViewCVS provided against a separate
server which is a minimum of three hours behind developer CVS.

New: ViewVC will be provided against developer CVS (it will be current).
CVS pserver will be provided against a secondary server (not developer
server) with a maximum expected delay of two hours.

Follow-up work is planned (this infrastructure takes us 80% of the way)
to essentially eliminate the sync delay.

4. Read-only rsync service

As a new service offering, we are now providing read-only rsync access
against developer CVS. This allows projects to efficiently make
on-demand backups of their entire CVS repository.

All projects should be making regular backups of their CVS repository
contents using this service.

5. Nightly tarball service

Nightly tarball service is being dropped in lieu of read-only rsync
service. Projects which currently depend on nightly tarballs for
repository backups will need to begin using rsync to make a backup copy
of their repository contents.

We see this as a major functional improvement. For a number of reasons,
tarballs have fallen out of sync with the data in the repository at
times in the past few years. Tarballs required a substantial amount of
additional disk, and I/O to generate. The move to read-only rsync
allows backups to be produced on-demand, with an update frequency chosen
by the project.

6. Points of failure

In the past, developer CVS service for all projects was provided from a
single host. CVS pserver service was provided from individual backend
heads based on a split of the data.

Under our new design, developer CVS and most of our CVS-related services
are provided from one of ten CVS hosts (count subject to increase with
growth). Each host is independent, and makes a backup copy of the
repository data of another host (which is used to provide the pserver
CVS service).

Failure of a single host will impact only the availability of data on
that host. Since the data is split among a larger number of hosts, the
size of data impacted by an individual host outage is substantially
smaller, and the time required for us to restore service will be
substantially shorter.

This rapid architecture change has been made possible specifically using
the research we performed for our recent launch of Subversion service.
We’ve applied our best practices, produced a substantial amount of
internal documentation, and kept an eye toward maintainability.
This effort has allowed us to deploy this new architecture quickly
once hardware was received, and will permit us to quickly scale
this service horizontally as growth and demand requires.

Many other minor improvements have also been made to improve the service
offering and make it less trouble-prone. The most important of which are
listed above. For a full description of the new service offering, and
for information on how to use the services described above, please refer
to the site documentation for the CVS service after the service has been
launched: https://www.sf.net/docs/E04

Thank you,

The SourceForge.net Team

« Hide it